The Analytics Harness Is the Real Product

The talk frames an agentic analytics harness as the missing layer between a question in Slack and a chart someone trusts. My read on that framing is simple. The harness is the product. The model is the engine. Most teams I work with still treat that backwards, and it is the single biggest reason their analytics agents stall after the first demo.

Three key takeaways

• An analytics agent is not a prompt with a database connection. It is a runtime, with context staging, query planning, and verification as first-class components, all of which need owners and SLOs.
• Context engineering beats prompt engineering for any data domain. Schema metadata, business definitions, allowed joins, and prior queries belong in a versioned context layer, not in a system prompt.
• Verification is where trust compounds or collapses. Build the checking loop before you build features, because one wrong number with confidence undoes ten correct ones.

The harness is software, not glue code

When I talk to data teams that have shipped an internal Claude bot for analytics, the first question I ask is who owns it. The answer is almost always a side project from one engineer on the data team, run on a hobby Slack app, with no tests and no on-call. That is fine for a prototype. It is not fine for anything that ends up in front of the CFO.

A serious analytics harness has the same shape as any other internal platform. It has a repo, versioned releases, a deploy pipeline, an eval suite, and a person whose job it is to keep it working. The harness is also the place where the team's opinions about the warehouse get encoded: which tables are blessed, which joins are forbidden, which metrics have a canonical definition, and which questions the agent should refuse to answer because they are ambiguous.

This reframing matters because it changes how the work is staffed. A prompt is a one-person job. A harness is a small team. If you cannot get a small team funded for this work, you do not have an analytics agent project, you have a demo.

Context staging is the actual hard part

The model can write SQL. That part is solved. What it cannot do, and what no improvement to the base model will solve, is know which of the 4,000 tables in your warehouse actually answers the question. Schemas drift. Definitions of revenue differ between finance and product. The same column name means different things in two adjacent tables because someone copied a pipeline three years ago and never reconciled it.

The harness has to stage that knowledge before the model sees the question. In practice that means a context layer with three things in it. First, the metadata: tables, columns, types, freshness, ownership. Second, the semantics: what each blessed metric means, written by humans, reviewed quarterly. Third, the history: which queries worked, which got revised, which were rejected, and why.

This context layer is its own engineering problem. It needs to be refreshed when the warehouse changes. It needs to be queryable by the agent in a structured way, not pasted in wholesale. It needs to know when it does not have an answer, because a hallucinated join is worse than a "I do not have enough context to answer this." Teams that get this layer right ship agents that feel like a competent analyst. Teams that skip it ship agents that produce confident garbage.

Verification with teeth, or no verification at all

The fastest way to lose a stakeholder is to give them a wrong number with a chart attached. Once that happens, even the right numbers feel suspicious for months. So verification is not a feature you add later. It is the spine of the system.

What does real verification look like for analytics? A few patterns I have seen work. Row count checks against expected magnitudes, so the agent refuses to return a single-digit revenue number when the business does billions a quarter. Re-running the same question with deliberate variation, like changing the date range by a day, to check that the answer moves the way it should. Comparing against a small set of canonical queries that the team has hand-verified. Citation back to the underlying tables so a human reviewer can replay the work.

The interesting design choice is what the agent does when verification fails. The lazy answer is to return the number with a warning. The right answer is to refuse and explain what went wrong. Refusal is a feature. An analytics agent that says "I tried three approaches and the numbers disagreed, so I am not going to guess" is far more valuable than one that always answers.

Cost and the fan-out problem

A naive agent will answer one question by running forty queries. Some of those are necessary exploration. Most are not. At enterprise scale, that fan-out becomes a real budget line, and your warehouse vendor will notice before your finance team does.

The harness has to make decisions the model cannot make on its own. Which queries to cache. Which to memoize. When to reuse a result from earlier in the session. When to refuse a question because the cost of answering it correctly is higher than the value of the answer. These are platform decisions, encoded in the runtime, not prompt decisions.

I have seen this play out in two ways. Teams that build cost controls early ship agents that scale across the company. Teams that ignore cost get one quarter into production and then get told to shut it down because the bill spiked. The cost layer is part of the harness. Treat it that way from day one.

Built with Claude Code, not just powered by it

The detail that gets buried in talks like this is that the harness itself was built using Claude Code. That changes how the engineering work scales. A small team can ship a serious internal platform because much of the keyboard work is delegated. The team's job becomes design, review, and integration, with the agent handling implementation.

This is the part of the analytics-harness story that generalizes beyond data work. Any team building internal AI tooling now has the option to build it with the same kind of agent they are deploying. The development loop and the product loop reinforce each other. The team learns what works in production by using it in development, and the patterns that survive both contexts are the ones that ship.

If you are running an analytics team, this is the place to push hardest. Most data engineers I work with are still writing pipelines by hand and treating Claude Code as a toy. The teams that move fastest have flipped that. The agent writes the harness. The humans review, design, and stage context.

What this means for your stack

If you are building or scoping an analytics agent project, three concrete moves come out of this.

First, fund the harness, not the prompt. Get a small team, a real repo, a deploy pipeline, and an on-call rotation. Treat it like any other internal platform.

Second, invest in the context layer before you invest in the model layer. The model will get better on its own. The context layer will not. The teams that have the cleanest semantic layer over their warehouse will own this category in their company.

Third, build the verification loop first. Pick three or four canonical questions, hand-verify the answers, and use those as your eval suite. Add features only after the eval passes consistently. This feels slow at the start. It is the only thing that scales.

Where this is heading

Analytics is a clean test bed because the answers are checkable. The harness pattern, runtime plus context plus verification, ports directly to operations, finance, and product analytics. It also ports into domains where the answers are softer, like legal and compliance, with the caveat that the verification layer gets harder. That generalization is where the next year of agentic engineering will be spent.

The companies that internalize this now will have a quiet advantage. Their agents will be trusted because they were designed to be checkable. Their teams will scale because the harness is software, not folklore. And their stakeholders will keep asking the agent questions, because the answers keep being right.

That is the bar. Right answers, repeatedly, with the cost of being wrong made small by design. Everything else is decoration.

Marco Kotrotsos, specializing in practical AI implementation for organizations ready to close the gap between AI hype and AI value. With 30 years of IT experience now focused purely on AI deployment, he works hands-on with companies to turn AI potential into measurable business outcomes.

This article is published in Autocomplete, a Medium publication about real-world AI for practitioners and decision-makers.

My free Substack newsletter, also called Autocomplete, can be found here: https://acdigest.substack.com.

Source talk: Building the best agentic analytics harness at Code with Claude London 2026. https://youtu.be/K4-flzsPraE

← All articles