The Prompting Playbook: Stop Telling Models to Be Good
I watched Margot Vanlaer's London talk last week and ended up scribbling notes in three different colors. The talk is called The Prompting Playbook, and it is the kind of session that quietly demolishes a habit you did not know you had. In my case, the habit was writing prompts that read like a pep talk.
If you have been shipping LLM features for more than six months, the scenario she opens with will be familiar. A prompt has been in production for a while. Multiple people have edited it. There is no clear owner. Policy is mixed with tone, tone is mixed with calculation rules, and there are three patches in there that you wrote to compensate for something a previous model used to do badly. You migrate to a new model, your test cases start failing, and you cannot tell whether the prompt is the problem or the model is the problem.
This is also where Vanlaer's playbook starts. The answer is not to revert. The answer is to bring evaluations with you when you move forward.
Three things worth keeping
- Treat prompts like code. Evaluations first, structure second, and patches you wrote for old models are usually why a new model misbehaves.
- Instructions do not add capability. If the model needs to do math, give it a tool. If it has to choose between two outcomes, tell it both sides.
- New use cases rarely need a smarter model. They often need a smaller model with a cleaner prompt, or three small prompts working together.
The first rule is that you need an eval
Her opening was unambiguous. You need evaluations to know whether a prompt change correlates with improvement. Without that, you are running on vibes.
What I like about how she framed this is the failure mode logic. When a new model performs worse on your task, there are exactly two possible causes. Either the model is capable but behaves differently, in which case prompting can fix it. Or the model is genuinely less capable for your task, in which case no prompting will fix it. You cannot tell those apart by reading outputs. You need to measure.
Her starter eval had five test cases. That sounds small, but each case covered a distinct category. One control case that should always pass. Edge cases where the model has failed before, so the regression cannot slip back in. And capability boundary cases where the model should refuse, escalate, or hand off to a human.
If your test suite does not include all three categories, you do not have an eval. You have a happy path test.
Read your prompt like a stranger
The first move after building the eval is general hygiene on the prompt itself. Not feature work. Just structure.
In her example, a fictional telco customer service bot, the prompt had collected the kind of debris every long-lived prompt accumulates. A line telling the bot it was a human. A block of text copied from a marketing page, complete with stray references to hero images and cookies. Instructions all dumped into one paragraph with no separation between role, policy, tone, and calculation rules.
She added XML tags. Role here, guidelines there, policy in its own block, tone in another. No new information. Just structure. The eval improved.
The line that stuck with me was this: if you read a prompt and cannot tell guidelines from policy from data, the model probably cannot either. Models do not have a magical parser for unstructured text that humans lack. If a human reader needs ten seconds to find the policy section, the model is also working harder than it should.
The other thing structure does is give you a cleaner control surface for iteration. Once your prompt is broken into sections, you can change one thing at a time and see what it does to the eval. With unstructured prompts, every edit touches two or three concerns at once, and your eval results become hard to interpret. You stop being able to reason about cause and effect. Structure first, iteration second. That is the order.
A small habit that helps. When you inherit a prompt, before you do anything else, run it through a markdown viewer or color the XML tags in your IDE. If the visual structure does not match the logical structure, you have already found a problem.
The three failures and what they taught
After hygiene, three failures remained. Each one points at a different lesson.
The model that withholds information. A customer on a legacy plan asked how much hotspot data they had. The customer data, fed into the prompt, said five gigabytes. The model gave the standard plan answer of four gigabytes and pointed them to a URL.
The model had the right number. It chose not to use it.
The cause was a defensive instruction written for an earlier model: "Never give a customer the wrong plan details. Instead, point them to the URL." The new model was now over-applying the rule, refusing to give any answer that might be wrong, including the correct one.
This is the failure mode nobody runs an eval for, because it looks like caution. The bot is technically not lying. It is just not helping.
The fix was to rewrite the rule with rationale. Customer information is the source of truth, use it. The eval passed.
The general lesson, which Vanlaer stated directly: "We worry a lot about hallucinations or the invention of facts and numbers, but actually the opposite can also happen. The model can withhold information that it actually has access to."
The auditing exercise this suggests is straightforward. For every defensive instruction in your production prompt, ask what model it was written for, and what failure it was meant to prevent. If you cannot answer either question, that instruction is a candidate for review. Some of them will turn out to still be doing useful work. Some will be the reason your new model behaves oddly. The only way to tell is to remove it, run the eval, and read the diff.
The math problem. A customer asked what their bill would look like if they switched plans mid-month. The model reasoned through it, did mental math in its response, and produced a vague answer no customer could act on.
The prompt said: "Critical: always calculate any prorated amounts correctly."
This is the moment in the talk where I underlined a quote twice: "Telling the model to do a good job isn't particularly helpful when we don't give the model the capability to actually do a good job."
The fix was a tool. Define a calculate_proration function, expose it through the API, tell the prompt to use it for any calculation. Eval passed.
Instructions do not add capability. They redirect existing capability. If the model is unreliable at mental math, no adjective in the prompt will save you. Tools, structured outputs, retrieval, code execution, those add capability. Adjectives do not.
This is the rule that gets violated most often, in my experience. I have seen prompts with three or four "critical" instructions stacked in a row, each one trying to shore up a different weak spot. The team writing them knew on some level that this was not the right shape, but the alternative felt heavier. Defining a tool, writing the schema, implementing the function, threading it through the API. That is more code. But the code is reliable, and the adjectives are not. Once you have done the tool route once, you stop reaching for adjectives.
The one-sided cost. A billing error case wanted the bot to escalate to a human. The bot kept trying to diagnose the issue itself, explaining possible causes to the customer.
The prompt had this line: "Avoid escalating or transferring to a care specialist unless absolutely necessary as it costs approximately $8."
The model was doing exactly what it was told. It had a cost with no offsetting benefit, and it was minimizing the cost.
The fix gave both sides of the tradeoff. Escalating costs $8, but failing to escalate a billing error can cost a refund and customer trust. Now the model is making a real tradeoff instead of optimizing one variable to zero.
Older models could get away with flat rules. Newer models reason about competing concerns. One-sided guidance that worked in 2023 is a failure mode in 2026.
The shift is subtle but important. With older models, the prompt was often doing the reasoning on behalf of the model, baking conclusions into instructions because the model could not be trusted to reach them on its own. With newer models, the prompt should provide the inputs to reasoning, not the conclusions. Trust the model to weigh tradeoffs, give it the variables it needs to weigh them, and verify with the eval. As a side effect, prompts get shorter and more maintainable.
When you build new instead of debugging
The second half shifted to a greenfield example. A retail staff scheduler with eight employees, a week of shifts, and a list of hard constraints. Because constraints are deterministic, a Python function can grade outputs by counting violations.
She ran five experiments to find the right combination of model and architecture.
Sonnet 4.6 with a simple prompt: all five test cases failed. Opus 4.7 with the same prompt: still failing, but the violation count dropped. Opus 4.7 with adaptive thinking: passed reliably, but with triple the tokens and triple the latency. Sonnet 4.6 with a better prompt that included "check your work": two out of five passing, with the rest failing on truncation rather than wrong answers.
The winner was none of those. It was an agentic loop. A generator prompt drafts a schedule. An evaluator prompt reads the draft and reports specific rule violations. A repair prompt takes the violations and makes targeted fixes. Three small prompts running in sequence beat the bigger model on accuracy, on tokens, and on latency.
The bonus, which I had not thought about, is that soft constraints can be added at runtime through the evaluator. "Harry does not like working with Sally, separate them where possible." That kind of rule lives in natural language inside the evaluator, not in the Python constraint checker.
The point of running all five experiments was not to declare a universal winner. It was to make the tradeoff space visible. You cannot guess the right architecture from outside.
The patterns worth memorizing
Pulling the talk together into a working checklist:
Before any prompt change, run the eval. If you do not have one, build it.
Run general hygiene before targeting failure modes. Add structure with XML tags. Remove copy-paste residue.
Audit defensive patches. Old patches for old models often suppress useful behavior in new ones.
Replace instructions with capabilities. Math problems need tools. JSON consistency needs structured outputs.
State both sides of every tradeoff. Newer models reason. Give them the inputs.
For new agents, test the architecture. Compare one big prompt to an agentic loop. The cheap reliable answer is often counterintuitive.
What to do tomorrow
Open the longest prompt in your production system. Read it as if you were a new hire who has to maintain it. Note every instruction you do not immediately understand the reason for. Those are your patches. Some are still working. Some are the reason your model behaves oddly.
Then write five test cases. One control, three edge cases, one capability boundary. Run your prompt through it. Now you have a baseline.
Everything else, the structure cleanup, the tool integration, the both-sides-of-the-tradeoff rewrites, is iteration against the baseline. Slow at first, then very fast once the loop is set up.
Hit reply if you have a prompt that has been in production long enough to accumulate this kind of debris. I am curious what the oldest patch in your system looks like, and what it was originally protecting against. The archaeology in these prompts is the most underrated part of LLM ops.
The teams that ship reliable AI features are not the ones with the cleverest prompts. They are the ones who treat the prompt as a maintained artifact and the eval as the contract.
Source talk: The Prompting Playbook by Margot Vanlaer at Code with Claude London 2026. https://youtu.be/G2B0YWuJUgI