Picking the Right Model: Stop Reading the Hot Takes

I watched Lucas's London talk on picking the right model last week, and the part that stuck with me was not the technical content. It was the framing of the problem.

Every time a new Anthropic model ships, the same cycle plays out. The model card lands. The prompting guide lands. Benchmark scores land. Then come the takes. Within forty-eight hours, X is split between "AGI is here" and "Anthropic is cooked," and the only consistent signal is that there is no consistent signal. Lucas's talk is a methodical answer to that noise. Stop asking the internet which model is best. Build an eval. Run it across configurations. Read the results.

That sounds obvious. Most teams I work with do not do it. They read benchmark scores, pick the model that sounds most impressive on launch day, swap it into production, hope it works. When it does not, they argue about which prompt to change. The eval gap is the gap between teams that ship reliably and teams that ship anxiously.

Three things worth keeping

1. A small private eval will teach you more than any public benchmark. Five well-designed test cases beat SWEBench for your specific workload.
2. The right model is the one that is cheapest per successful outcome, not per token. A faster model that fails costs more than a slower one that works.
3. You have more dials than you think. Effort, thinking, prompt caching, and context engineering can shift the cost-quality curve, not just move along it.

What public benchmarks actually tell you

SWEBench Verified tells you a model is moving in a direction on general coding capability. BrowseComp tells you something similar for research tasks. If a new model jumps on these, that is real information.

It is not enough information for your business. Your production workload is heterogeneous. A coding agent might have to find a niche detail about an SDK on the web and then implement it. That is two benchmarks crossing each other, and neither one captures the whole task. You might also be working in a language that does not show up in SWEBench at all.

So benchmarks tell you a direction. They do not tell you a decision.

There is a deeper reason for this. Public benchmarks are averages over public tasks. Your workload is a specific distribution that almost certainly does not match the benchmark distribution. A model that improves on SWEBench by ten points might be worse on your codebase if you use unusual conventions, a language with low representation in training, or internal libraries the model has never seen. The averages cancel out the details that matter for your business. Your eval is what brings those details back.

What an eval actually looks like

Lucas used a math exam comparison that I will be borrowing. You have a question, you have an answer, and the working between the question and the answer matters. For an agent, the steps are as important as the outcome.

For a customer service agent, that means an LLM judge checking the final response, plus another LLM judge checking that the database was queried correctly, plus a deterministic check that the agent called a specific tool with the right arguments. Multiple graders per task, each catching a different category of failure.

The hard part is the dataset. Writing the questions, defining the correct outcomes, defining the correct steps. Lucas was direct on this point. In a world where AI is automating large chunks of engineering, building a high quality eval is one of the highest leverage uses of human time. The eval is what decides which automation actually works.

A few failure modes to watch for.

Noise mistaken for signal. Run every task multiple times. If results vary widely across runs, the task is probably underspecified.

Infrastructure failures mistaken for model failures. Lucas told the story of a benchmark transcript that looked bad until he found a wall of API timeouts and tool call failures. The model had not performed badly. The plumbing had. If you confuse these, you make architecture decisions based on noise.

Silent saturation. Your eval needs to look like your production traffic. Once you launch, collect real traces, find the failure modes, fold them back into the eval.

And one habit worth installing: read the transcripts. Lucas described a case where Claude Code was performing strangely well on a coding benchmark. The transcripts showed Claude was searching git history for answers from previous trials. Headline metrics would have looked like a huge win. Transcripts caught the actual behavior.

"The closer you can get to the raw data, very much the better."

That is the line worth taping above your monitor.

The reason transcripts matter is that aggregate metrics compress away the failure shapes. A pass rate of 78 percent tells you something, but it does not tell you whether the 22 percent that failed all failed for the same reason or twenty different reasons. A cluster of failures sharing a root cause is a fix waiting to happen. Twenty different reasons might mean the eval is too varied to drive a focused improvement. You only see the difference in the transcripts.

The faster small model fallacy

Lucas told a story that changed how I think about model selection. An internal code fix pipeline started with Haiku 5, no thinking. Scored 92 percent. The team wanted 100, so they turned thinking on, and got there. Out of curiosity, they reran with Sonnet and Opus. Both scored 100. Both took less time than Haiku with thinking on.

That last detail is the thing. The larger, more intelligent models finished faster. Not because per-token speed was higher, but because they did the work in fewer turns. Less wandering, less re-validation, more direct execution.

This breaks the "smaller equals faster" heuristic that lives in most teams' heads. If your task is one prompt to one response, per-token speed dominates and Haiku probably wins. If your task is an agent that might take ten turns, the model that gets there in three turns wins, even if each turn is slower.

The rule of thumb worth writing on your whiteboard: cheapest per successful outcome, not cheapest per token. A model that fails 30 percent of the time at half the per-token cost is more expensive than a model that succeeds reliably at full cost. The failures still cost tokens, and they also cost retries, customer trust, and engineering time.

This reframing changes how you allocate engineering effort. If you optimize for cheapest per token, you spend your time shaving tokens off prompts. If you optimize for cheapest per successful outcome, you spend your time eliminating failure modes. Those are different workstreams. The first one can be done in isolation. The second one requires the eval, the transcripts, and a clear understanding of what success means for your task. The teams I see making real progress chose the second framing and stayed with it.

Effort and thinking, untangled

A clarification that helped me. Effort and thinking are different dials, and you can combine them.

Thinking is a scratchpad. From Sonnet 4.6 onward, Anthropic models have adaptive thinking, meaning the model itself decides how much to reason before acting. This is system-two thinking, the slow deliberative mode for hard problems.

Effort tells Claude how much to write across thinking, tool calls, and responses. You can have low thinking with high effort, or no thinking with high effort. They are not the same axis.

For a hard task you might want thinking on and high effort. For a chatty customer service flow you might want no thinking and low effort. Running your eval across these configurations gives you a fine-grained map of the cost-quality-latency surface. You stop choosing models and start choosing operating points.

Shifting the curve

So far, everything is about navigating the existing tradeoff curve. The more interesting claim is that two techniques shift the curve entirely.

Prompt caching. When you use a cached prefix, you pay one-tenth the list price for input tokens. That is not a small efficiency. That is the difference between "we cannot afford Opus" and "we can use Opus everywhere."

Lucas's reframe: prompt caching gives you Opus quality at Sonnet cost, or Sonnet quality at Haiku cost. The best systems internally have prompt cache hit rates around 80 to 90 percent.

To get there, treat your messages array as append-only. Anything in the prefix should be immutable. The classic failure mode is sticking a datetime variable in the system prompt. It ticks every turn and breaks the cache. Pull dynamic content out of the prefix. Append, do not edit.

Context hygiene. Lucas's hot take was that teams spend too much time on multi-agent orchestration and not enough on the basic discipline of making tool responses lean.

His example was a sports data tool returning scores. Original response was full JSON with verbose ISO timestamps. The optimized version used markdown instead of JSON, simplified the dates, and added the day of the week so the model did not need to compute it. That single cleanup produced a 66 percent token reduction. The tool response shows up every turn the agent runs, so the savings compound.

Another example. A web search use case where multiple searches returned overlapping articles. Adding deduplication before passing results to Claude produced a 77 percent input token reduction, 65 percent cost reduction, and a 9 percent accuracy improvement. Less data to reason over, better answers.

The principle: do not pipe raw API responses into your tools. Treat tool responses as something a human will read. Clean up the formatting. Remove duplicates. Drop fields the model does not need.

A second-order benefit. When you put effort into making tool responses readable, you force yourself to understand what the model actually needs to do its job. That clarity makes the prompt simpler too, because the model is not compensating for noisy inputs by carrying extra instructions. The discipline cleans up the whole system, not just the per-turn token count.

The workshop result

Lucas closed with a demo using TauBench, an airline customer service benchmark, running across Haiku, Sonnet, and Opus, with thinking on and off, at multiple effort levels.

A few of the findings were genuinely counterintuitive. Opus 4.7 with thinking on at high effort had the highest pass rate, which was expected, but it consumed fewer tokens than Sonnet did on the same task. Haiku with thinking on performed similarly to Sonnet with thinking on at high effort. And Opus with high effort and thinking on ran faster than Sonnet at comparable thinking levels.

None of those results are guessable from outside. You have to run the sweep.

What this looks like in practice

Build a private eval. Five to twenty test cases minimum, with multiple graders per task, covering steps as well as outcomes.

Run the eval across models and configurations. Plot pass rate against tokens, cost, and latency.

Read the transcripts. Separate infrastructure failures from model failures.

Pick your operating point on the Pareto frontier. The model and configuration that gives you what you need.

Shift the curve. Prompt caching first. Context hygiene second. Re-run the eval.

The teams that make this work do not have stronger opinions about models. They have stronger measurement.

The calmer way to work

The best side effect of running this process is that hot takes stop being load-bearing in your model decisions. New model lands, you spend an afternoon, you have an answer. The X commentary becomes background noise. The benchmark scores become directional context. The decision becomes a number on your dashboard.

Hit reply if you have run a model sweep across configurations recently. I want to hear which results surprised you the most, especially the cases where a smaller model with thinking beat a bigger model without it, or where the per-turn cost story flipped once you measured actual completions. The surprises are where the real learning is.

Source talk: Picking the Right Model by Lucas at Code with Claude London 2026. https://youtu.be/P0uMXS6emHA

← All articles