Picking the Right Model: Cheapest Per Successful Outcome, Not Per Token

Three key takeaways

1. A small private eval will teach you more than any public benchmark. Five well-designed test cases beat SWEBench for your specific workload.
2. The right model is the one that is cheapest per successful outcome, not the one that is cheapest per token. A faster model that fails costs more than a slower one that works.
3. You have more dials than you think. Effort, thinking, prompt caching, and context engineering can shift the cost-quality curve, not just move along it.

There is a moment after every model release when teams gather to ask the same question. Should we switch? And then someone opens a benchmark report and the conversation goes sideways.

Lucas, on Anthropic's applied AI team, has spent enough time watching this play out to give it a name. He calls it the noise problem. When a new model ships, the model card lands, the prompting guide lands, the benchmark scores land, and then the takes land. AGI is here. Anthropic is cooked. Everyone has a hot take, and none of them tell you whether the new model is right for your business.

His talk at Code with Claude London 2026 is a methodical answer to that question. The short version: stop asking the internet which model is best. Build an eval. Run it across configurations. Read the results. The longer version is more interesting.

What public benchmarks actually tell you

Public benchmarks are not useless. SWEBench Verified gives you a directional read on coding capability. BrowseComp gives you a directional read on research capability. If a new model jumps significantly on these, that is real information.

But Lucas was clear about the limits. Your production workload is heterogeneous. A coding agent in the real world has to find a niche detail about an SDK on the web, then implement it. That is two benchmarks crossing each other, and neither one captures the whole task. You might also be working in a language that does not show up in SWEBench at all.

So benchmarks tell you the model is moving in a direction. They do not tell you the model is right for you. For that, you need your own measurement.

There is a deeper reason this matters. Public benchmarks are inherently averages over public tasks. Your production workload is a specific distribution that almost certainly does not match the benchmark distribution. A model that improves on SWEBench by ten points might be worse on your codebase if your codebase has unusual conventions, uses a language with low representation in training data, or relies on internal libraries the model has never seen. The averages cancel out the details that matter for your business. Your eval is what brings the details back.

What an eval actually looks like

The framing I liked best was the math exam comparison. You have a question, you have an answer, and the working between the question and the answer matters. For an agent, the steps are as important as the outcome. The agent has to reach the right answer, but it also has to take the right path.

For a customer service agent, that might mean an LLM judge checking the final response, plus another LLM judge checking that the agent queried the database correctly, plus a deterministic check that the agent called a specific tool with the right arguments. Multiple graders per task, each catching a different category of failure.

The hard yards are creating the dataset. Writing the questions, defining the correct outcomes, defining the correct steps. Lucas was direct on this point: in a world automating large chunks of engineering work, building a quality eval dataset is one of the highest leverage uses of human time. The eval is the thing that decides which automation actually works.

A few failure modes to watch for when building your first eval suite.

Noise mistaken for signal. Run every task multiple times. If results vary widely across runs, the task is probably underspecified or the success criteria are ambiguous. Variance is information, not background.

Infrastructure failures mistaken for model failures. Lucas told the story of digging into a benchmark transcript where the numbers looked off, and finding a wall of API failures and tool call timeouts. The model had not performed badly. The plumbing had failed. These need to be separated, because if you confuse them, you make architecture decisions based on noise.

Silent saturation. Your eval needs to look like your production traffic, including the long tail. Once you launch, collect real traces, find the failure modes, fold them back into the eval. This is the feedback loop that keeps the eval honest.

And one habit worth installing: read the transcripts. Lucas described an eval where Claude Code was performing strangely well on a coding benchmark. The transcripts showed Claude was searching the git history and pulling answers from previous trials. Headline metrics would have looked like a huge win. Transcript reading caught the actual behavior.

"The closer you can get to the raw data, very much the better."

That is the line worth taping above your monitor.

The reason transcripts matter is that aggregate metrics compress away the failure shapes. A pass rate of 78 percent tells you something, but it does not tell you whether the 22 percent that failed all failed for the same reason or for twenty different reasons. A cluster of failures sharing a single root cause is a fix waiting to happen. Twenty different reasons might mean the eval is too varied to drive a focused improvement. You only see the difference in the transcripts.

The pillars: quality, cost, latency

Once the eval exists, model selection collapses into three pillars. How well does the model perform on your task. How long does it take. How much does it cost.

Conceptually simple. Operationally complex, because Anthropic now ships models with knobs. Sonnet with max thinking. Opus with low thinking. Haiku with no thinking but with high effort. And that is just one provider. Add Anthropic versus competitors and the comparison surface gets large.

But here is the counterintuitive thing the talk surfaces, which I had not internalized: smaller models are not always faster.

The faster small model fallacy

Lucas told a story about an internal code fix pipeline. The team started with Haiku 5, no thinking, and scored 92 percent. They wanted 100 percent, so they turned thinking on, and got there. Out of curiosity, they reran with Sonnet and Opus. Both scored 100 percent. Both took less time than Haiku with thinking on.

That last detail is the one to sit with. The larger, more intelligent models finished faster on this task. Not because the per-token speed was higher, but because they did the work in fewer turns. Less wandering, less re-validation, more direct execution.

This is where the "smaller equals faster" heuristic breaks. If your task is one prompt to one response, then per-token speed dominates and Haiku probably wins. If your task is an agent that might take ten turns to complete something, then the model that gets there in three turns wins, even if each turn is slower.

A second example. When Opus 4.5 shipped, on certain tasks it produced higher accuracy than Sonnet with significantly fewer output tokens. Just running on vibes, you would have picked Sonnet because it is the smaller model. But the data showed Opus completing the task with less work.

The rule of thumb: cheapest per successful outcome, not cheapest per token. A model that fails 30 percent of the time at half the per-token cost is more expensive than a model that succeeds reliably at full cost, because the failures still cost tokens and they also cost retries, customer trust, and engineering time.

This reframing also changes how you allocate engineering effort. If you optimize for cheapest per token, you spend your time finding ways to shave tokens off prompts. If you optimize for cheapest per successful outcome, you spend your time eliminating failure modes. Those are different workstreams. The first one can be done in isolation. The second one requires the eval, the transcripts, and a clear understanding of what success means for your task. The teams I see making real progress are the ones who picked the second framing and stayed with it.

Effort and thinking, untangled

A useful clarification from the talk: effort and thinking are different dials, and you can combine them.

Thinking is a scratchpad. From Sonnet 4.6 onward, Anthropic models have adaptive thinking, which means the model itself decides how much to reason before acting. This is the system-two thinking mode, the slow deliberative mode that helps on hard problems.

Effort tells Claude how much to write across thinking, tool calls, and responses. You can have low thinking with high effort, or no thinking with high effort. They are not the same axis.

For a hard task you might want thinking on and high effort. For a chatty customer service flow you might want no thinking and low effort. For something in between you might want thinking on but low effort to keep latency manageable.

The practical point is that running an eval across these configurations gives you a fine-grained map of the cost-quality-latency surface. You stop choosing models and start choosing operating points.

The operating point framing also lets you change your mind cheaply. If business priorities shift toward latency, you can move along the curve without re-architecting. If you suddenly need more accuracy on a specific subtask, you can dial up effort on that path while keeping defaults elsewhere. The configuration becomes a tunable property of the system rather than a fixed property of the model choice. That flexibility is undervalued until the day you need it.

Shifting the curve, not just moving along it

So far, all of this is about navigating the existing tradeoff curve. Lucas's more interesting claim is that two techniques shift the curve entirely.

Prompt caching

When you use a cached prefix, you pay one-tenth the list price for input tokens. That is not a small efficiency. That is the difference between "we cannot afford Opus" and "we can use Opus everywhere."

The reframe Lucas offered: prompt caching gets you Opus quality at Sonnet cost, or Sonnet quality at Haiku cost. It moves the curve down. Anthropic uses it extensively in Claude Code and other internal products. The systems with the best economics tend to have prompt cache hit rates around 80 to 90 percent.

To get there, the rule is simple. Treat your messages array as append-only. Anything in the prefix should be immutable. The classic failure mode is sticking a datetime variable in the system prompt, which ticks every turn and breaks the cache. Pull dynamic content out of the prefix. Append, do not edit.

The Anthropic API returns cache hit metrics on every call. Measure, hill climb, iterate.

Context hygiene

The second curve-shifter is context engineering. Lucas's hot take was that teams spend too much time on multi-agent orchestration and not enough on the basic discipline of making tool responses lean and clean.

His example was a sports data tool returning Premier League scores. Original response was full JSON with verbose ISO date timestamps. The optimized version used markdown instead of JSON, simplified the dates, and added the day of the week so the model would not need to compute it. That single cleanup produced a 66 percent token reduction on the tool response. And the tool response shows up every turn the agent runs, so the savings compound.

A second example, from a web search use case. Multiple searches were returning overlapping articles. Adding deduplication before passing results to Claude produced a 77 percent input token reduction, 65 percent cost reduction, and a 9 percent accuracy improvement. Less data to reason over, better answers.

The principle: do not pipe raw API responses into your tools. Treat tool responses as something a human will read. Clean up the formatting. Remove duplicates. Drop fields the model does not need. Every cleanup compounds across every turn of every conversation.

There is a second-order effect worth naming. When you put effort into making tool responses readable, you also force yourself to understand what the model actually needs to do its job. That clarity makes the prompt simpler too, because the model is not having to compensate for noisy inputs by carrying extra instructions. The discipline cleans up the whole system, not just the per-turn token count.

The workshop result

Lucas closed with a workshop demo using TauBench, an airline customer service benchmark. They ran the eval across Haiku, Sonnet, and Opus, with thinking on and off, across multiple effort levels. The skill in question instruments the sweep automatically and plots the results.

A few of the findings were genuinely counterintuitive. Opus 4.7 with thinking on at high effort had the highest pass rate, which was expected, but it consumed fewer tokens than Sonnet did on the same task. Haiku with thinking on performed similarly to Sonnet with thinking on at high effort, which compresses the cost gap dramatically if Haiku is good enough for your use case. And Opus with high effort and thinking on ran faster than Sonnet at comparable thinking levels.

None of those results are guessable from outside. You have to run the sweep.

The pattern repeats across customer engagements I have seen. Teams expect their eval results to confirm their intuitions, then the results surprise them. The smaller model is more accurate on a specific subtask. The bigger model is cheaper on a different subtask. Thinking helps less than expected on one path and more than expected on another. The intuitions are not wrong because the engineers are bad at reasoning. They are wrong because the surface is large and the relationships are not monotonic. Measurement collapses the surface to a chart you can read in five minutes.

What this looks like in practice

If you take the talk seriously, the workflow is:

Build a private eval. Five to twenty test cases minimum, with multiple graders per task, covering steps as well as outcomes. Include cases where the model should refuse or escalate.

Run the eval across models and configurations. Haiku, Sonnet, Opus. Thinking on and off. Multiple effort levels. Plot pass rate against tokens, cost, and latency.

Read the transcripts. Especially when results look off. Separate infrastructure failures from model failures. Catch any data leakage or gaming.

Pick your operating point. Not your favorite model. Your operating point on the Pareto frontier. The model and configuration that gives you the cost or latency or accuracy you need.

Shift the curve. Prompt caching first. Context hygiene second. Then re-run the eval.

The teams that make this work do not have stronger opinions about models. They have stronger measurement. Everything else follows.

What changes once you have this

The most useful side effect of running this process is that hot takes stop being load-bearing in your model decisions. You read the launch post. You read the prompting guide. You feed the prompting guide to Claude and ask it to update your prompts. Then you run your eval. The eval tells you whether to switch.

It is a calmer way to work. New model lands, you spend an afternoon, you have an answer. The X commentary becomes background noise. The benchmark scores become directional context. The decision becomes a number on your dashboard.

That is the real argument for building the eval. Not just better model selection. A faster, quieter, more confident way of making the decision in the first place.

Marco Kotrotsos, specializing in practical AI implementation for organizations ready to close the gap between AI hype and AI value. With 30 years of IT experience now focused purely on AI deployment, he works hands-on with companies to turn AI potential into measurable business outcomes.

This article is published in Autocomplete, a Medium publication about real-world AI for practitioners and decision-makers.

My free Substack newsletter, also called Autocomplete, can be found here: https://acdigest.substack.com.

Source talk: Picking the Right Model by Lucas at Code with Claude London 2026. https://youtu.be/P0uMXS6emHA

← All articles