"Which is the best model?" is the question of someone who has not shipped one yet
The conversation always starts the same way. Someone opens a leaderboard, points at the top spot and asks whether we should build the agent on that one. It is a fair question and it is the wrong one, because it assumes an agent uses one model. Agents that survive production do not use one: they use several. And the decision that actually matters is not which, it is the split.
The reason is architectural and boring. An agent does not do one task: it does a chain. It reads an email and works out what it is about. It searches your documentation and pulls three passages. It drafts a reply one of your customers will read. It decides whether that goes out on its own or through a person. Four steps, four different demands: sorting an email into six categories has been a solved problem for years; drafting what your customer reads is where your face is on the line. Paying for the most capable model on all four is putting your best lawyer on photocopying duty.
This is not our opinion. OpenAI's guide to building agents says it plainly: models have different strengths and trade-offs around task complexity, latency and cost, and not every task needs the smartest model. Their example is exactly the one above: a simple retrieval or intent classification can be handled by a smaller, faster model, while deciding whether to approve a refund benefits from a more capable one. And they recommend considering a variety of models for different tasks in the workflow rather than one for everything. Source: A practical guide to building agents, OpenAI, accessed 12 September 2026.
This guide is about making that decision on purpose: the three axes that actually decide, the order you walk them in, how the split works, and what layer you need so that changing model is not a rebuild. What it is not: a "best LLMs of 2026" table — that expires before we publish it — nor the bill for running AI in production, which is a different conversation with different numbers.
The three axes that actually decide
Strip out the noise and the choice rests on three questions. None of the three is answered by reading a model card: all three are answered by measuring in your own house.
| Axis | The question it answers | How you measure it in your case |
|---|---|---|
| Accuracy on your task | How many of my real cases does it get right? | A set of 30-50 of your own cases with the correct answer written next to each |
| Latency | Does it hold up in the channel the agent lives in? | The 95th percentile of response time, never the average |
| Cost per resolved case | What does it cost to resolve one end to end? | Total cost of the whole flow divided by resolved cases, not price per million tokens |
Accuracy is the axis most people skip, because it is the one that takes work. "Accuracy" is not a general grade: it is the percentage of YOUR cases that come out right. Getting it means doing the boring part — thirty or fifty real cases, actual emails with their typos and their odd attachments, with the correct answer written next to each. Without that set you are not choosing a model: you are holding an opinion about models, which is a different thing and does not survive a meeting.
Latency is not an absolute number: it is a number against a channel. A chat on your website has a budget of seconds because there is a person watching the screen. A nightly job reconciling invoices has all night. The same model is fast in one and slow in the other, so "is it fast?" means nothing until you say where. Anthropic frames it as what it is — a trade: agentic systems often trade latency and cost for better task performance, and you should consider when that trade makes sense. Source: Building effective agents, Anthropic, 19 December 2024, accessed 12 September 2026.
Price per million tokens is a list price, not your bill. A cheap model that needs three attempts, leaves half the fields empty and ends up escalating to a person costs you more than an expensive one that gets it right first time. The correct unit is the resolved case: total cost of the flow — every call, every retry, including the ones from the step that failed — divided by the cases that came out right without anyone touching them.
The public leaderboard knows nothing about your work
With the three axes on the table it is clear why the leaderboard does not decide. A public ranking measures a standardised exam: quiz questions, toy coding problems, logic puzzles. Your task is not that. Your task is your customers' emails, your internal jargon, your badly scanned PDFs and your returns policy, and none of those four appear in any table. We made the case at the time: LLM benchmarks do not predict your result.
What a public ranking is good for is the opposite of what people do with it: it is for ruling out, not for choosing. It tells you which models are in the conversation and which fell behind two generations ago, and that saves you from evaluating fifteen candidates. From there, the shortlist — two, three — gets ordered by your own case set. That order rarely matches the leaderboard, and when it does, it is a coincidence you cannot bank on next time.
The split: cheap to classify, expensive to decide
The pattern has a name and it is documented. Anthropic calls it routing: a first step classifies the input and directs it to the specialised follow-up task it belongs to. Its value, they explain, is separation of concerns and the ability to write more specialised prompts, because without that split, optimising for one kind of input hurts performance on the others. And one of the examples they give is literally the cost split: routing easy, common questions to smaller, cost-efficient models, and hard or unusual ones to more capable models. Source: Building effective agents, Anthropic, 19 December 2024, accessed 12 September 2026.
Translated into the steps of a typical agent, the split usually lands like this. The right-hand column is a starting point, not a law:
| Agent step | What it really demands | Where it usually lands |
|---|---|---|
| Classify the input and route it | Consistency and speed over a closed set of categories | Small model |
| Extract fields from a document | Stable format; errors are caught by validation | Small model with an output contract |
| Search and summarise your documents | Faithfulness to the source; retrieval quality dominates | Mid-tier model |
| Decide on money, people or regulated data | Judgement and nuance; errors are paid in euros or reputation | Most capable model |
| Draft what a customer will read | Tone, precision and zero invention | Most capable model |
The only thing that can confirm that table is your own case set. There are classification jobs with twenty overlapping categories where the small model sinks, and drafting jobs so narrow — an acknowledgement with three variables — that the small one is overkill. That is why the split is decided by measuring, and revisited when the model catalogue changes or your volume does.
One nuance that saves grief: routing only pays off when the categories are genuinely distinct and classification can be handled accurately. That is the condition Anthropic attaches to this pattern, and it is the one that fails when someone drops a router in where none was needed. If the classifier gets it wrong you have not saved anything: you have put a new point of failure in front of everything else, and a silent one at that, because a misrouted input does not throw an error — it returns a confident answer of the wrong kind.
The right order: start expensive and come down until it shows
With the split clear, the question is how. OpenAI's guide proposes a recipe that runs against everyone's instinct: build the prototype with the most capable model for every task to establish a performance baseline, and from there try swapping in smaller models to see whether they still achieve acceptable results. Their principles, in this order: set up evals to establish the baseline, meet your accuracy target with the best models available, and only then optimise for cost and latency by replacing larger models with smaller ones where possible. Source: A practical guide to building agents, OpenAI, accessed 12 September 2026.
- Build the case set before you touch the agent. Thirty or fifty, real, with the correct answer next to each. This is the step everyone skips and the one that decides whether the rest is worth anything.
- Build the whole agent with the most capable model at every step. You are not optimising yet: you are finding out whether your accuracy target is reachable at all.
- Measure against the set. If you fall short here, the problem is not the model: it is the prompt, your data, the retrieval or the task — and dropping to a smaller model will only hide it.
- Once you hit the target, come down one step at a time and re-run the set. One change, one measurement. Two changes at once and you no longer know which one it was.
- Stop at the rung before the one that breaks. And write down which step uses which model and at what measured accuracy, because in three months nobody will remember.
The reason not to do it the other way round is diagnostic, not purist. If you start with the cheap model and the agent does not work, you have four suspects and no way to separate them: it could be the model, the prompt, your data, or a task that was never properly defined. Starting from the top, when something breaks on the way down you know exactly what broke it, because you only changed one thing.
The layer that lets you change models without touching the agent
Everything above has an expiry date. The model catalogue turns over every few months, prices move and providers deprecate versions. If today's decision is written inside the agent — the model name repeated in seven places in the code — in six months you will not be able to redo it, and you will end up with the problem we described in how to change model without breaking your automations: the flow does not break with an error, it breaks with a different format and a different tone, quietly.
The layer that prevents it is five pieces, and none of them is sophisticated:
- One single function that calls the model. Every call goes through it. If there are seven places in the code with a model name inside, you already have debt.
- The model identifier, in configuration. One file with the model for each step. Changing model has to be changing a line, not opening the agent.
- A validated output contract. Before the result touches any system, you check it has the agreed shape. That is what turns a silent format change into a visible error.
- The case set, runnable with one command. If trying a new model costs half a morning of manual work, nobody will try one.
- A log of which model handled which case. Without it you cannot compare before and after a change, or explain why last week it worked better.
This is not an optional optimisation you defer to phase two. It is the difference between choosing a model today and being able to choose again in six months. When that layer does not exist, the model decision is made once and inherited forever, which is exactly the wrong shape for a system living in a market that moves every quarter.
The split sheet: one page, with an owner and a date
The ending is the same as in an AI agent's instructions: if the decision is not written down, it does not exist. One page is enough, and it has to answer these five things.
- Which model each step uses, with one line of why and the accuracy measured against the set on the day it was decided.
- What was tried and rejected. The candidate that did not make it and the reason. Without this, somebody will test it again from scratch in four months.
- Who signs it. A named person. A sheet with no owner never gets reviewed.
- What triggers a review: a relevant new model appearing, your provider deprecating the one you use, volume changing by an order of magnitude, or measured accuracy dropping.
- Next review. A date, even if nothing has happened. The decision expires on its own and nobody announces it.
If your agent is already in production and this page does not exist, start with the steps that touch money: which model decides a refund, which model drafts what your customer reads, and at what measured accuracy. The rest can wait a week.
We build that layer before the agent: isolated call, model in configuration, output contract and a runnable case set, as part of the AI infrastructure for enterprise that the AI employees are then built on. It is the part nobody shows in a demo and the only one that decides whether, a year from now, you can change model in an afternoon or have to rebuild the system. The full logic is in building an AI agent that survives production.