More powerful is not always better: sizing AI when it becomes part of a process

While AI is a tool we consult occasionally, that rule costs very little. The problem arises when AI stops being a conversation and becomes part of a process: a task that runs hundreds or thousands of times a month, without anyone watching every execution.

At that point, choosing the model, and what we send it, stops being a personal preference. It becomes a decision about organisation, costs and, as we shall see, the law as well.

The “just in case” bias

In a law firm, nobody assigns a partner to open the mail. Not because the partner cannot do it, but because their time is expensive and the task does not require it. Someone filters the mail, distinguishing the urgent from the routine, and only matters requiring judgement reach the partner’s desk.

Allocating work according to its complexity is not penny-pinching; it is good organisation. With AI, we tend to forget this because we assume that a more capable model will deliver a more accurate answer in every case.

For narrowly defined tasks, that assumption is less sound than it appears. Deciding whether an email is an invoice, a client enquiry or advertising does not require deep reasoning. It requires clear instructions, good examples and a way to check the result.

A well-instructed small model can do it just as well, faster and for a fraction of the price. If the task did not need more capability, the large model adds little assurance; what it adds is waiting time and cost.

The useful question, therefore, is not which is the best model available, but how much capability this task needs to be done well.

What a recurring task costs

Consider an automation that checks email every five minutes. That is 288 checks a day and 8,640 in a thirty-day month, even though most will find nothing relevant has arrived.

Does it make sense to activate a model with maximum capability each time and give it the complete project history? Probably not. The first operation, finding out whether there are new messages, does not even need AI.

Email services already let us receive notifications or query only the changes since the last check, through mechanisms such as Gmail push notifications or Microsoft Graph incremental queries. This is conventional software: inexpensive, fast and predictable.

The first saving comes before choosing a model. It consists of not calling one when none is needed.

Context has a price too

There is a common misunderstanding about the context window. A model supporting a million tokens does not mean it consumes a million on every call: billing is based on what is actually sent to it.

The problem arises when the automation resends the instructions, tools, entire history and previous documents on every execution. A small task then drags along an enormous input.

Keeping the conversation on the provider’s server does not necessarily avoid this either. In the OpenAI API, chaining responses with previous_response_id preserves continuity, but previous tokens in the chain continue to count as input, according to its conversation-state documentation.

A hypothetical example shows the scale of the difference:

These are input volumes, not bills. The final cost depends on the provider, model, generated responses and cache.

Caching makes the parts of the context repeated identically at the beginning of each call cheaper. However, whether it can be used depends on conditions set by each provider, such as the initial content being exactly the same or not leaving too long between calls (OpenAI prompt-caching documentation).

And more context does not guarantee a better answer. Models do not make equal use of all the information they receive: information placed in the middle of long contexts tends to be used less effectively (Liu et al., “Lost in the Middle”). Sending everything “just in case” can make the task more expensive and, at the same time, less reliable.

A legal issue as well

For those of us working with client data, sizing the context is not merely a matter of efficiency. Article 5(1)(c) of the GDPR requires personal data to be adequate, relevant and limited to what is necessary for the purposes of processing. Article 25 requires this limitation to be incorporated by design and by default.

An automation that sends the entire history of a matter to an external provider every five minutes, when the new message and a few references would suffice, processes far more data than necessary.

For lawyers, moreover, every piece of data that leaves the firm unnecessarily expands the perimeter within which we must safeguard professional secrecy. A properly sized design is also the easiest to justify.

This is where a local model makes particular sense. If classification takes place on the firm’s own infrastructure, the data does not leave for an external provider at that stage. Only what genuinely needs a larger model reaches third parties.

How I would design it

For email, the process can be organised in stages, just as work is organised in a law firm. First, conventional software detects new messages. Then a small model, preferably local (so it does not send data outside), classifies them and extracts the essentials.

Only ambiguous or complex cases move to a more capable model. Sensitive decisions, such as accepting terms, committing money or responding to a client on certain matters, still go through a person.

The system should also keep a state record: which messages it has processed, what it has concluded and what remains pending. With that record and a few relevant documents, most tasks can continue without reconstructing the whole conversation. That is what a good case file does: nobody rereads the entire matter to file a new document.

None of this removes the need for testing. Choosing a small model is a hypothesis that must be tested against real cases. Cheap classification that requires every result to be reviewed ends up being expensive, while a more powerful model can justify its price if it prevents significant errors.

OpenAI’s own cost-optimisation guide proposes smaller models provided they maintain the necessary accuracy. The metric that interests me is not the price per call, but the cost of obtaining a correct, accepted result, including retries, corrections and human review time.

Knowing when less is enough

As projects and automations multiply, this discipline stops being optional. Some tasks deserve the best available model and all the context we can give it. Others can be solved with a lightweight model and a few lines, and some do not need AI at all.

Always using the most powerful model seems prudent, but it is the prudence of someone who has not measured. Maturity in using AI also means knowing when less is enough.