I work with bots all day: I can have anything from one to fifty working at the same time on different projects. I call the infrastructure that makes this possible La Machina.
I keep track of the tokens each project and each bot uses. As I studied those statistics, a question came to mind: if I let a conversation with a bot grow very long, does each message end up consuming much more?
I was writing short questions. But behind them, an enormous backpack could be travelling along.
What we carry in the backpack every time we talk to AI
A token is a small piece of information that AI processes: a long word such as “unconstitutionality” can be split into several pieces.
The context window is the capacity of the backpack: how much material the AI can keep to hand while preparing an answer.
It can hold your question, the instructions, the documents and parts of the previous conversation. The app decides what to include. If it keeps the entire history, each new question can also carry the weight of all the earlier ones.
Example: you have spent the morning planning a trip. You ask “what if we go by train?” and the backpack is still loaded with hotels, flights and discarded plans.
That is why a short answer can have an enormous input behind it. The AI needs the material available to continue; earlier content does not become free just because it is already on screen. OpenAI explains this in its conversation documentation.
A model with a context window of 1,000,000 tokens (as a rough ballpark estimate, about 700,000 words or 2,000 pages) has an enormous backpack. But we do not have to fill it: everything we put inside travels along again with each question and costs tokens. And tokens cost money.
How much the backpack weighs
Let us compare a conversation that keeps everything with one that replaces earlier content with summaries. The work continues, but the load accompanying each message changes.
| Example with 100 messages | Without summarising | With summarising |
|---|---|---|
| Total input tokens | 50.5 million | 10.6 million |
| Relative consumption | ≈5× | 1× |
| Cost of that input with Gemini 3.8 Flash, at USD 0.75 per million | ≈38 USD | ≈8 USD |
Illustrative example with fixed-size additions and summaries. The calculation adds up everything sent across all requests, even when repeated. The cost uses Google’s official rate, without a cache discount, responses or the work of summarising. It does not represent my bill or the price of a subscription; the full calculation is in the editorial notes.
The surprise lies in that repetition. Each message adds to the load for later questions. Keeping the conversation open can become expensive even if the length of what we write barely changes.
Quick comparison
Caching is a shortcut that allows work on repeated content to be reused and that part of the backpack to be charged at a lower rate.
These figures are for specific models used through an API, the connection other programs use to ask them to do work. The summary column refers to the chat app. Its available capacity may vary by plan.
| Service / reference model | Model’s published maximum capacity | Does it summarise automatically in the chat app? | Discount on cache reads through the API |
|---|---|---|---|
| ChatGPT / OpenAI — GPT-6.1 Sol | 1,050,000 tokens | Not published in the sources consulted | 95 % |
| Claude / Anthropic — Opus 5.5 | 1,000,000 tokens | Yes, with code execution enabled | 95 % |
| Gemini / Google — Gemini 3.8 Flash | 1,048,576 input tokens | Not published in the sources consulted | 90 % |
The discount applies only to content that can be reused, not to the entire bill. Preparing or storing it may involve other costs. “Not published” does not mean the app does not do it. Nor do these capacities mean all that space is available for conversation: each model’s response limits must be respected.
If you want to distinguish the subscription fee from usage, it is explained in the article on chat plans and the API. To understand the units, see the article on tokens and real costs.
Lightening the load without losing the work
AI can also lose track of what sits in the middle of an enormous backpack, as the study Lost in the Middle showed in the models it evaluated.
Summarising helps organise things, but it can leave out details. I would keep the originals and check that the summary captures the goal, the decisions made and what remains to be done. Then I would take it into a new conversation.
Example: “Summarise what we agreed, the conditions we must respect and the next step.” Review the result before continuing in another chat.
| What to do | Advice |
|---|---|
| You change the subject | Open a new chat. |
| The conversation has grown long | Ask for a summary, review it and start again. |
| You need to examine an enormous document | Use a large window and provide the necessary material. |
It is the same idea as matching AI capacity to the work and choosing the right model for each task.
I want La Machina to have what it needs in each backpack. And to avoid carrying useless weight every time I ask it a question.
Text by Roberto Ortega, prepared with AI editorial assistance. The Norwegian and English editions are editorial adaptations of the Spanish original.