AI Data Agents Need Metric Contracts, Not Raw SQL

ChatGPT Work now ships a Data agent that connects to your warehouse, writes its own SQL, and renders a dashboard before your coffee cools. Ask it for churn, active customers, or overdue invoices and it hands back a chart — axis labels, legend, the works. No analyst required. What it just did, quietly, is automate the argument that usually happens in the Tuesday leadership meeting: what does “active customer” actually mean here?

That argument used to be a feature, not a bug. Someone would push back on a number, ask how it was calculated, and the room would either agree on a definition or escalate to finance. Slow, but it caught errors. An agent that skips straight to a polished chart skips that friction too — and the friction was doing real work.

The SQL Problem Is Solved. The Definition Problem Isn’t.

Text-to-SQL is no longer the hard part. Modern models handle joins, window functions, and date arithmetic without much coaching, and context windows are wide enough to hold a full schema plus sample rows. The syntax is fine. What hasn’t moved is the problem sitting underneath it: an agent with table access still has to guess what “revenue,” “churn,” and “overdue” mean in your specific business, and it will guess with exactly the same confidence it would use for a correct answer.

That guess is the actual risk surface. Give an agent raw database credentials and a natural-language prompt, and you’ve handed it discretion over which columns represent truth. Nobody signed off on that discretion — it exists only because some column in the schema happened to look plausible.

A Churn Chart That Was Wrong and Looked Right

We ran an internal pilot earlier this year with a straightforward brief: calculate monthly user churn from our own subscription data and chart it. The agent found the user table, found a legacy subscription log left over from an earlier billing migration, joined them, and produced a clean, descending line — exactly the shape a churn chart is supposed to have.

It was wrong. The join used account_created_at as the reference timestamp instead of subscription_cancelled_at, because the legacy table’s cancellation column had a different name than the one in the current schema, and the agent fell back to the timestamp that was present and well-populated in both tables. The query ran clean. The chart rendered clean. The number was fabricated in the sense that actually matters: it measured account age decay, not churn.

That’s the failure mode worth losing sleep over. A broken query throws a stack trace, and someone fixes it before it ever reaches a slide. A wrong-but-plausible chart clears every technical check and walks straight into a deck. Nobody in that meeting is going to eyeball a smooth downward-sloping line and suspect the join condition — it looks like every other churn chart they’ve ever seen. That’s what makes it dangerous. It’s not obviously broken. It’s quietly wrong. And the cost of catching it late is exactly the kind of correction nobody wants to make in front of a board.

What Commission Payouts Taught Me About “Recognized Revenue”

I ran into a version of this problem back in 2014, on an ERP integration built around commission payouts. Sales wanted commission calculated the moment a deal closed. Finance wanted it calculated against recognized revenue, which under our accounting treatment meant revenue tied to delivery milestones, not signature dates. Both sides had a defensible definition. Both sides had a spreadsheet that proved they were right.

We didn’t fix that with a better SQL view — a better view just encodes one side’s opinion more elegantly, with cleaner syntax. We fixed it by getting finance and sales into a room, writing down the exact definition of recognized revenue for commission purposes on paper, and having the CFO sign it. Only then did we translate that document into code. The definition changed twice over the following two years, as new product lines with different delivery terms came online, and every one of those changes required a fresh signature before it shipped. That signature wasn’t ceremony. It was the thing that made the number defensible when someone challenged it in a payout dispute.

An LLM agent staring at a schema has none of that context. It has column names and data types — nothing more. Expecting it to reconstruct “recognized revenue” or “active customer” from that alone, and then trusting the output enough to put it in front of a board, is asking a language model to do the CFO’s job without the CFO’s context. That’s not a model capability gap. It’s a governance gap we’re pretending is a technical one.

The Semantic Layer Is the Firewall

The fix is to stop letting the agent see raw tables at all. Instead of prompting against the schema, the agent should prompt against a semantic model: a curated layer that defines each metric once, locking in the exact tables, joins, and filters that produce it. When someone asks for Q3 revenue, the agent doesn’t write a SELECT from scratch. It calls the revenue node in the semantic model and asks for it grouped by quarter.

This shrinks the agent’s action space on purpose. It can aggregate, filter, and visualize a defined metric. What it can’t do is decide, mid-query, that account_created_at looks like a reasonable stand-in for a missing cancellation date. That decision got made once, by a human, when the metric was defined — and it stays frozen until someone deliberately changes it. Semantic layer tools like dbt’s metrics layer or Cube exist precisely to put this boundary in place; which specific tool you pick matters far less than the principle underneath it — the agent’s SQL generation happens inside a fence, not outside one.

The trade-off is real, though. Analysts lose the ability to ask a genuinely novel question on the fly — if a metric isn’t modeled yet, the agent can’t invent it, and someone has to define and ship it first. Building and maintaining that layer is ongoing engineering work, not a one-time setup you check off and forget. That’s a fair price for not shipping fabricated numbers. But it’s a cost, not a free lunch.

Metrics as Signed Commits, Not Tribal Knowledge

The practical move is to treat metric definitions the way we already treat infrastructure: as version-controlled code with an approval gate. Each metric — revenue, churn, overdue invoice, active customer — lives as a definition in a repository. Changing it means opening a commit. Merging it requires sign-off from whoever owns that number in the business: finance for revenue, product for active-user definitions, credit control for overdue status.

That gives you exactly what the 2014 commission document gave us: a paper trail. If a number gets challenged six months later, you don’t debug a prompt — you look at the commit history and see who approved the current definition and when it last changed. The AI agent’s job shrinks down to executing against whatever definition is currently active. Which, frankly, is the right amount of authority for a model to have.

No Contract, No Chart

SQL generation looked like the bottleneck holding back AI-driven analytics. It wasn’t. Trust was — and trust doesn’t come from better syntax. It comes from provenance. An AI data agent without a versioned, human-signed semantic layer underneath it isn’t an analytics tool. It’s a fast way to produce a wrong number that looks exactly like a right one.

The rule going forward should be simple enough to enforce in a design review: if a metric doesn’t have a signed contract behind it, the agent doesn’t get to chart it. Everything else is a prompt engineering problem wearing a governance problem’s clothes.