GenBI in Practice

The 74-Point Accuracy Gap: Why AI Analytics Fails Without Human Curation (And How to Fix It)

Sean Liu — Wed, 23 Sep 2026 01:59:16 GMT

Anthropic recently published how they run self-service data analytics internally. One number stood out: without structured skills docs, their AI agent scored 21% accuracy. With them, it hit 95%+.

That 74-percentage-point gap wasn't closed by a better model, a fancier framework, or a new tool. It was closed by people writing markdown files.

This article breaks down what Anthropic tested, what didn't work, and a generalizable process any data team can follow to build the artifacts that actually move the needle.

What Anthropic Tested — And What Failed

Before we talk about what works, it's worth understanding what Anthropic tried that didn't work. These findings challenge common assumptions about AI analytics.

Failed: Auto-generated semantic layer definitions

Anthropic tried having an LLM auto-generate metric definitions from raw tables and query logs. The result was net-negative on their evaluation set — worse than having no semantic layer at all.

Why? The auto-generated definitions produced plausible-looking metrics that actually encoded the very ambiguities the team was trying to eliminate. When "revenue" has three valid interpretations in your data warehouse, an LLM will pick one without understanding why the others exist.

Takeaway: Use AI to generate documentation (descriptions, summaries). Have a human own the definition (what the metric means, how it's calculated, which table is canonical).

Failed: Raw query corpus access

This is the surprising one. Anthropic gave their agent direct grep access to their entire corpus of dashboard SQL, transformation code, and analyst notebooks — thousands of files. They verified in transcripts that the agent actually read the relevant queries before answering.

Accuracy moved by less than one percentage point.

They checked the obvious confounds: Was the correct answer in the corpus for questions it got wrong? About 80% of the time, yes. Did "answer present" predict "now gets it right"? No — the flip rate was flat.

The information was there. The agent saw it. It still didn't use it correctly.

The bottleneck wasn't access to prior work. It was structure — mapping a question to the right entity. Unstructured retrieval couldn't connect a new question to the right precedent.

Failed: Additional rounds of doc refinement past a certain point

They hit three consecutive net-negative iterations where docs were getting longer, not better. More words didn't mean more accuracy.

Failed: Cheaper model for adversarial review

Swapping the adversarial reviewer to a cheaper model to cut latency lost most of the accuracy gains without meaningful speedup.

What Actually Worked

Three artifacts accounted for nearly all of the accuracy improvement:

Artifact	What it does	Impact
Skills (structured markdown instructions)	Routes the agent to the right sources in the right order	21% → 95%+ accuracy
Human-curated semantic layer	Single source of truth for metric definitions	Mandatory first step for every query
Curated reference docs	Domain-specific tables, gotchas, and patterns distilled from the query corpus	Solved retrieval failure

A fourth technique — an adversarial review sub-agent — added 6% accuracy on top, at the cost of 32% more tokens and 72% higher latency.

The Process: Building These Artifacts From What You Already Have

The knowledge required to build these artifacts isn't new. It already exists in your organization — scattered across dashboards, Slack threads, analyst notebooks, and people's heads. The process is extraction and structuring, not creation from scratch.

Phase 1: Mine Your Existing Resources (1–2 weeks per domain)

Step 1 — Identify your top 20 business questions per domain

Your company already asks these questions repeatedly. Find them in:

Dashboard titles and chart labels. Every chart is essentially a natural language question with validated SQL behind it. "Monthly Revenue by Region" is both the question and proof that someone already wrote correct SQL to answer it.
Slack channels where people request data from your team. These contain the real phrasing, real ambiguity, and real business jargon your users actually use. Search for patterns like "can someone pull..." or "what's the latest..."
Query logs. Every warehouse (Snowflake, BigQuery, Databricks) records query history. Filter to analytical queries (not ETL jobs), cluster by business domain, and identify the most frequently run patterns.
Recurring meeting agendas. "Review weekly active users" and "check pipeline by region" are business questions hiding in plain sight.

Don't try to cover everything. Anthropic found diminishing returns past a few dozen evals per topic. Start with the questions that get asked every week.

Step 2 — For each question, trace back to the canonical answer

Open the dashboard or notebook that answers it. Find the SQL. For each question-SQL pair, document:

Which tables and columns were used
Which filters were applied (and why)
Which definition of the metric was used
What a senior analyst would warn you about

This is where you discover ambiguity. You'll find that "revenue" is calculated differently in three dashboards. That's not a bug — it's the most valuable finding. Resolve it now with your business stakeholders and pick one canonical definition.

Step 3 — Document the gotchas

Ask your most senior analyst: "What do new hires get wrong in their first month?" That list is your gotchas doc:

"This field double-counts if you don't filter by X"
"Status = 4 means refunded, not cancelled"
"This table has a 2-day lag — always use MAX(date), not 'yesterday'"
"We changed product names in Q3 2025 — old values are frozen in the data"

These are the things that make an answer go from "valid SQL" to "correct business answer."

Phase 2: Structure the Artifacts

Now convert what you found into the three artifacts that Anthropic showed actually work.

Artifact 1: Reference Docs (One Per Domain)

This is the most impactful artifact. It replaces unstructured tribal knowledge with a document an AI agent can reliably navigate.

Anthropic shared the skeleton they use internally. Here it is, adapted with annotations:

# [Domain] Tables

## Quick Reference
### Business Context
[What this domain means in plain words. Not technical —
write it for someone who just joined the company.]

### Entity Grain
[What one row represents. This single line prevents
the most common join errors.]

### Standard Hygiene Filter
[The filter every query in this domain applies.
Example: "WHERE is_test_account = FALSE AND event_date >= '2024-01-01'"]

## Dimensions
[How the key dimensions are encoded, and how the same
concept is named differently across tables.
Example: "region" in table A = "geo" in table B = "market" in table C]

## Key Tables

### [table_name]
- **Grain**: [what one row represents]
- **Scope/exclusions**: [what's included, what's deliberately left out]
- **Usage**: [when to use it, when NOT to use it, join keys, required filters]

### [table_name_2]
[... one section per governed table ...]

## Gotchas
[The wrong-answer modes a senior analyst would warn you about.
These are the things that make the difference between
"valid SQL" and "correct business answer."]

## Best Practices / Common Query Patterns
[Default choices, standard cuts, worked patterns where
the exact query form is the hard part.]

## Cross-References
[Neighboring domain docs that own adjacent questions.
"For customer lifetime value, see finance/ltv.md"]

Key principle: Write these docs for retrieval by an AI agent, not for human reading. That means explicit routing triggers ("IF the question is about experiment lift... DO NOT use for raw event counts"), not narrative explanations.

Artifact 2: Skills (Routing Instructions)

Skills are what turned Anthropic's accuracy from 21% to 95%. They're the procedural knowledge: which sources to consult in what order.

Anthropic's approach uses two paired skills per domain:

Knowledge skill — a thin router that narrows the search space:

---
name: [domain]-knowledge
description: "IF the user asks about [domain topics] — THEN invoke this skill."
---

# [Domain] Knowledge

## Semantic Layer (REQUIRED first step)
The governed semantic layer is the mandatory default path for
every data question. Raw SQL is the fallback, used only after
the semantic layer is shown not to cover the ask.

## Required workflow
1. **Search** the semantic layer for matching metrics/dimensions
2. **Check segments** — named canonical population filters
   (hand-rolled WHERE clauses for these are the dominant wrong-answer mode)
3. **Compile and run** if found
4. **Fallback** to reference docs ONLY if no semantic layer coverage

> Don't bail early. Do NOT fall back to raw SQL because:
> - "Needs a custom date filter" → [covered by time-dimension specs]
> - "Needs a join" → [the metric layer already encapsulates joins]

## Date Conventions
- "Last week/month" → last *complete* calendar week/month, not trailing-7/30
- Timezone default: [your timezone]
- Freshness lag: [which tables settle late]

## Reference Docs Navigation
### [Sub-domain A] → `references/[sub_domain_a].md`
- **Use for**: [kinds of questions]
- **Key tables**: [...]

### [Sub-domain B] → `references/[sub_domain_b].md`
- **Use for**: [...]

## Gotchas
- Use `[field_x_v2]` NOT `[field_x]`
- [Two tables report the same metric at different grains — which to use]
- [Which of two plausible sources is canonical]

Runbook skill — encodes the analysis process:

---
name: [domain]-runbook
description: "Execution process for [domain] analysis questions."
---

# [Domain] Analysis Runbook

## Step 1: Clarify the question
- Time period? Segment? What business decision does this inform?
- Disambiguate terms (see Entity Disambiguation in knowledge skill)

## Step 2: Check for existing dashboards
- [Link to domain dashboard catalog]

## Step 3: Find sources
- Invoke [domain]-knowledge skill
- Follow its semantic layer → reference doc → raw table priority

## Step 4: Execute and validate
- Run the query
- Apply adversarial review: challenge all assumptions
- Check data freshness

## Step 5: Report with provenance
Every answer ends with:
> **Source:** [semantic layer | governed table | raw exploration]
> **Freshness:** [max date in data]
> **Owner:** [owning team]

## Reusable Analysis Patterns
### Retention curve
[Standard approach for this domain]

### Funnel analysis
[Standard approach]

### Rate decomposition
[Standard approach]

Artifact 3: Semantic Layer (Human-Curated Metrics)

Start small: just the top 10–15 metrics per domain. For each one, you need:

The exact calculation
The grain
Required filters (always applied)
Dimensions it can be sliced by
Who owns it

The tool doesn't matter — dbt MetricFlow, Cube, warehouse-native semantic views, or even a well-structured YAML file. What matters is that a human defined it and a business stakeholder signed off.

Do not auto-generate these with an LLM. Anthropic tested it. It was net-negative.

Phase 3: Validate and Maintain

Build your evaluation set

Your top 20 questions per domain from Phase 1, paired with verified answers, become your offline eval set. Format:

{
  "domain": "revenue",
  "question": "What was total revenue by region last month?",
  "gold_sql": "SELECT region, SUM(amount) FROM fct_orders WHERE ...",
  "expected_behavior": "Should use semantic layer metric 'total_revenue'",
  "snapshot_date": "2026-09-01",
  "difficulty": "simple",
  "owner": "finance-analytics"
}

Anthropic's guidance:

Target ~100% offline accuracy before launching a domain
Pin every eval to a snapshot date — an eval against live data goes stale when the underlying number moves
Store results like telemetry, with skill version, model ID, and per-assertion pass/fail
Gate launches per domain — a domain owner can't announce the agent to stakeholders until their eval set clears threshold

Maintain like a first-class product

Anthropic watched accuracy drift from ~95% to ~65% within a month when they stopped maintaining docs. Their solution:

Colocate everything in one repo. Skill markdown, reference docs, and transformation models live together. The PR that changes a data model must also update the docs.
CI hook flags any reporting-model change that doesn't touch a skill file
90% of data-model PRs now include a skill change in the same diff
Harvest corrections. A scheduled agent scans Slack for correction language, drafts a fix to the relevant reference doc, and opens a PR

The Uncomfortable Truth

The most valuable skill in AI-powered analytics isn't prompt engineering, model selection, or framework architecture. It's the unglamorous work of writing and maintaining structured documentation about what your data actually means.

This can't be automated. The knowledge is tribal — it lives in people's heads, in Slack threads, in the footnotes of dashboards nobody reads. The job is extracting that knowledge into a structured format an AI agent can follow.

Anthropic proved the math: 21% without human curation, 95%+ with it. The gap isn't closed by better models. It's closed by analytics engineers doing what they've always done — understanding the business — and writing it down in a way machines can use.

Sources

How Anthropic enables self-service data analytics with Claude — Anthropic, June 2026
BEAVER: An Enterprise Benchmark for Text-to-SQL — The first text-to-SQL benchmark derived from real enterprise data warehouses
Text-to-SQL for Enterprise Data Analytics — LinkedIn's internal SQL Bot paper, reporting 53% accuracy on enterprise data
From analytics engineer to context engineer — dbt Labs, on the role shift

My semantic layer answers this question confidently. The answer is ambiguous.

Sean Liu — Thu, 10 Sep 2026 03:50:18 GMT

I'm building an evaluation harness for AI-generated SQL. While simplifying my semantic layer, I accidentally removed a guardrail I didn't know I had — and discovered a class of question that conventional semantic layers don't handle well.

The question

"What was average order value by product category in 2023?"

This sounds straightforward. It isn't. The problem is that 28.62% of orders in this dataset span multiple product categories. An order with a jacket ($100) and socks ($15) belongs to both Outerwear and Accessories.

So "AOV by category" requires a decision about how to attribute multi-category orders, and there are at least four defensible methods:

Attribute the full order to every category it touches. The Outerwear row shows $115 (the whole order), and so does the Accessories row. Revenue double-counts across categories. AOV is high everywhere.

Attribute only the lines belonging to the category. Outerwear shows $100, Accessories shows $15. Revenue doesn't double-count, but the denominator is "orders touching this category," which still includes the multi-category order in both rows. You're dividing line-level revenue by order-level counts — a mixed-grain ratio.

Split the order proportionally. Outerwear gets 100/115 of the order, Accessories gets 15/115. Clean in theory, almost never how anyone actually computes it.

Exclude multi-category orders. Only single-category orders contribute. Clean, but you've just dropped 28.62% of your data.

Four methods, four numbers, all defensible. The correct behavior for any analytics system is to ask which one the user means.

What my semantic layer does instead

I have a MetricFlow semantic layer defined over fct_order_items at line-item grain. The metric is:

- name: average_order_value
  type: ratio
  type_params:
    numerator: net_revenue
    denominator: order_count

Where order_count is COUNT(DISTINCT order_id) at line grain, and category is a dimension on the same table.

When I run:

mf query --metrics average_order_value --group-by category

It returns numbers. Outerwear $110, Socks $13, and so on. No error, no warning, no indication that a choice was made.

What it actually computed: for each category, sum the net revenue of lines in that category, divided by the count of distinct orders that touched that category. That's method 2 from the list above — line-level revenue divided by order-level counts. The denominator counts the multi-category order in both Outerwear and Accessories.

The calculation is mathematically valid. The problem is its interpretation.

It measures something like:

category-line revenue per order that contains this category

That could be a useful metric. It is simply not the only defensible interpretation of "average order value by category."

Nothing in the output signals that an attribution or definitional choice has been made.

Was this caused by bad data modeling?

This is where the distinction between data modeling and semantic modeling matters.

fct_order_items at line-item grain is not inherently a bad model. In fact, it is a natural place to represent product, category, quantity, and line-level revenue.

The problem is that the requested metric and dimension naturally live at different conceptual grains:

AOV is fundamentally an order-level concept.
Category is a line-item-level concept.
An order can contain multiple categories.

A different data model could make one particular definition explicit.

For example, we could create a category-order fact such as:

order_id
category
attributed_revenue

Then category AOV could be defined over that model.

But that doesn't make the ambiguity disappear. We would still have to decide whether attributed_revenue means full-order attribution, line revenue, proportional allocation, or something else.

Better data modeling can encode an attribution decision.

It cannot determine which attribution decision the user intended.

That's why I don't think the right conclusion is "the semantic layer should have been modeled differently." The deeper problem is that the question itself is underspecified.

How I found this

I originally had two semantic models — one at order grain, one at line-item grain. order_count lived on the order model, category lived on the line model. When I asked for AOV by category, MetricFlow refused: no valid join path from an order-grain measure to a line-grain dimension.

That refusal was accidental. I hadn't designed a guardrail — MetricFlow's join resolution just couldn't bridge the two grains for that combination. But it was correct behavior: the system couldn't compute the ambiguous thing, so it didn't.

Then I simplified to a single semantic model at line grain. Everything lives on one table, no cross-grain joins, simpler to maintain. The simplification was an improvement in every other way. But it removed the accidental guardrail, because order_count (now COUNT(DISTINCT order_id)) and category live on the same table. MetricFlow has no join to resolve, so it answers without hesitation.

I ran it, got numbers, and almost moved on. The numbers looked normal. That's the dangerous part.

The semantic-layer gap

This isn't a MetricFlow bug. The metric definition is valid, and the generated calculation is valid for the chosen interpretation.

What the experiment exposes is a gap in conventional semantic-layer abstractions.

Semantic layers are good at describing how a metric is computed: aggregation, filters, measures, dimensions, and relationships between entities.

But that doesn't necessarily tell the system whether a metric is unambiguous for a particular dimension.

For example, a semantic layer could theoretically know that:

average_order_value + month
    → unambiguous

average_order_value + category
    → requires an attribution policy

The challenge is that conventional metric definitions generally don't capture this kind of metric × dimension validity or attribution requirement.

That means a query can pass every technical check:

the metric exists;
the dimension exists;
the relationship is valid;
the SQL is valid;
the aggregation is valid.

And the result can still be semantically ambiguous.

That's a more interesting failure mode for GenBI than broken SQL.

Broken SQL usually announces itself.

A clean query that returns a plausible number for an underspecified business question does not.

What I'm doing about it

I'm not trying to "fix" the metric in the semantic layer. The metric is defined correctly. The computation is correct for one valid interpretation.

The problem is that the question has multiple valid readings and the system picks one without asking.

Instead, I'm putting this question into the evaluation harness I'm building. It's one of 20 "ambiguous tier" questions where the correct system behavior is to ask for clarification, not to answer. The harness will test multiple systems — including MetricFlow, a RAG-based agent, and raw text-to-SQL — on the same question, and measure which ones ask and which ones guess.

I also added a warning to the metric definition:

description: >
  Net revenue per order. WARNING: grouping by category produces
  "average category-line revenue per order touching that category,"
  not true AOV within a category, because 28.62% of orders span
  multiple categories.

That warning ends up in the schema context that an LLM reads.

Whether it changes the model's behavior is something the eval will measure.

And that's an important distinction: if the warning makes the model ask for clarification, that's useful evidence that metadata can help. If the model still confidently generates the query, that tells me the problem needs a different mechanism than metric descriptions alone.

Caveats

One dataset, one semantic layer tool.

The 28.62% multi-category rate is specific to this snapshot — a different business might have 5% or 80%, which changes how material the issue is.

The point isn't the specific number.

It's that transactional data frequently contains relationships where a metric defined at one grain becomes ambiguous when grouped by a dimension at another grain.

Whether a semantic layer should encode these ambiguities, whether an agent should detect them dynamically, or whether they belong in an evaluation or governance layer is still an open design question.

That's what I'm testing next.

Part of a project building an evaluation harness for AI-generated SQL.

Repo: github.com/Sean-Liu-GitHub/genbi-ecommerce

Key files:

The definitions document isn't documentation. It's an accuracy intervention

Sean Liu — Thu, 03 Sep 2026 02:46:43 GMT

In a previous post I showed that column descriptions improve the accuracy of AI-generated SQL but don't make it deterministic. This is the follow-up: what happens when you go further and give the model a structured business definitions document?

Short answer: it fixed every error I tested, and — more interesting — it changed the kind of answer the model gave.

Setup

I built a dbt project over BigQuery's thelook_ecommerce dataset. The marts layer has two main tables: fct_order_items (line-item grain) and fct_orders (order grain, pre-aggregated). Both are clean, well-named, documented in the schema yaml.

Before building anything AI-related, I wrote a definitions document — ten business decisions, each traceable to a profiling finding. Things like: what does "revenue" mean (gross or net)? Which statuses count as "sold"? Which timestamp anchors a time period? Which join path reaches the product table?

Then I asked Claude the same three questions, twice: once with just the schema, once with the definitions document pasted as context.

Question 1: "How many items did we sell in 2023?"

Without definitions:

SELECT COUNT(*) AS items_sold
FROM fct_order_items
WHERE EXTRACT(YEAR FROM ordered_at) = 2023
  AND line_status != 'Returned'

It filtered out returns — reasonable. But it kept Cancelled items. In this dataset, a cancelled item isn't a sale. The model made a judgment call about which statuses count as "sold" and got it partially right.

With definitions:

SELECT COUNT(*) AS items_sold
FROM fct_order_items
WHERE EXTRACT(YEAR FROM ordered_at) = 2023
  AND line_status NOT IN ('Returned', 'Cancelled')

Correct. And it cited the reasoning: "the definitions specify that Processing lines count as sales because the customer committed to buy."

The difference isn't just the filter. Without definitions, the model explained what it did. With definitions, it explained why, referencing a convention rather than improvising one.

Question 2: "What was our revenue in 2023?"

This is the question with two defensible answers.

Without definitions:

SELECT SUM(gross_revenue) AS revenue_2023
FROM fct_orders
WHERE EXTRACT(YEAR FROM ordered_at) = 2023
  AND order_status != 'Cancelled'

Three problems:

It picked gross_revenue — which includes returned items. In this dataset, returns are 10% of revenue. That's a material difference.
It added an ad-hoc order_status != 'Cancelled' filter on the header, but gross_revenue was already computed from all lines regardless of status. The filter and the column contradict each other.
It never mentioned that the table has a net_revenue column, or that the choice between gross and net existed.

With definitions:

SELECT SUM(net_revenue) AS revenue_2023
FROM fct_orders
WHERE EXTRACT(YEAR FROM ordered_at) = 2023

Correct. No ad-hoc filter needed — net_revenue already excludes Returned and Cancelled at the line level. And it explained why the previous approach was wrong: "adding AND order_status != 'Cancelled' would actually under-count by dropping entire orders that have a Cancelled header status."

This is the result that surprised me most. Without definitions, the model didn't just pick the wrong column — it built an inconsistent query that mixed a gross measure with a net filter. With definitions, it picked the right column and caught the mistake it would have made.

Question 3: "What was our gross margin by category in 2023?"

Without definitions:

SELECT category, SUM(line_margin) AS gross_margin, ...
FROM fct_order_items
WHERE EXTRACT(YEAR FROM ordered_at) = 2023
GROUP BY category

No status filter. Margin included returned and cancelled items. The number is defensible in isolation — you could argue margin should include all items — but it's inconsistent with how the same model answered question 1, where it filtered out returns.

That inconsistency is the real problem. Across three questions, the model without definitions made three different decisions about what "sold" means. A user asking these questions in sequence would get numbers that don't reconcile with each other.

With definitions:

SELECT category, SUM(line_margin) AS gross_margin, ...
FROM fct_order_items
WHERE EXTRACT(YEAR FROM ordered_at) = 2023
  AND line_status NOT IN ('Returned', 'Cancelled')
GROUP BY category

Correct filter, consistent with questions 1 and 2.

The pattern

Without definitions, the model was:

Partially right on status filtering (caught returns, missed cancellations)
Inconsistent across questions (filtered status for counts, not for margin)
Silent about ambiguity (never mentioned that "revenue" has two meanings)
Incoherent within a single query (gross column + net filter on Q2)

With definitions, the model was:

Correct on every status filter
Consistent across all three questions
Explicit about conventions ("per the business definitions...")
Self-correcting ("this differs from my earlier draft because...")

The definitions document didn't just improve accuracy. It gave the model a coherent framework to reason from, and the reasoning was visible in the response.

What a definitions document actually is

This is the point I want to make clearly, because it changes how you think about the work.

Most teams treat business definitions as documentation — something you write for onboarding or compliance, stored in Confluence, read once, and never updated. The definition of "active user" is written somewhere, and nobody checks whether the dashboard agrees with it.

What this experiment suggests is that a definitions document is an accuracy intervention — it directly changes the correctness of AI-generated queries, measurably, on every question that touches a defined term. That makes it engineering work, not paperwork. And it makes maintaining it operational, not optional.

The ten definitions I wrote took about two days, including the profiling that produced the evidence behind each one. Every one resolved an ambiguity that would otherwise be resolved differently on every query, by every user, with no way to detect the inconsistency.

Caveats

Three questions, one model (Claude), tested 2026-09-03. I didn't run N repetitions on this one because I used the chat interface rather than the API — so I can't report variance. The findings are directional, not statistical. The full eval harness with systematic runs across multiple models is coming.

This also tested the ideal case: the definitions were pasted directly into context. In a real deployment, the definitions would need to be retrieved, selected, and injected automatically — which is a harder problem. What this shows is the ceiling: if the model has the definitions, it uses them well.

Part of a project building an evaluation harness for AI-generated SQL. The definitions document, the profiling methodology, and the dbt project are open-source.

Repo: github.com/Sean-Liu-GitHub/genbi-ecommerce

Key files:

Better column descriptions made the AI right more often. Not reliably.

Sean Liu — Fri, 28 Aug 2026 06:27:19 GMT

I assumed most "the AI can't query our warehouse" problems are really documentation problems. Before building an evaluation harness to test that properly, I ran a small experiment.

Half right, it turns out. The half that's wrong is the interesting bit.

Setup

I snapshotted BigQuery's public thelook_ecommerce dataset into my own project on 2026-08-27. Synthetic ecommerce data — orders, line items, products, inventory, users. Clean by the standards of a real warehouse, which matters: whatever breaks here breaks worse in production.

One question, asked repeatedly through Gemini in BigQuery:

How many items did we sell in 2023?

10 runs per configuration, changing exactly one thing between configurations. No prompt engineering, no hints. Just the question a business user would type.

I want to be precise about what kind of failure I was looking for. A model that writes broken SQL is easy to catch — the query fails, you see it immediately. The harder failure is when the SQL is valid, the number is plausible, and the model has silently made a business decision on your behalf. That's what this experiment is about.

The order_items table is the one that matters. Here's its schema in BigQuery — note the status column:

The trap

order_items.status holds values including Complete, Shipped, Processing, Cancelled and Returned. Nothing in the schema says which of those count as a sale.

29.2% of lines in this dataset are returned. So "how many items did we sell" has at least two defensible answers, and they differ by 29.2%.

This isn't a bug. It's ordinary warehouse ambiguity — the kind that lives in every schema I've worked on, usually resolved by an analyst who happens to know.

Run 1: no description

Hit rate: 0/10

It never filtered on status. Every run counted returned and cancelled items as sales.

-- how many items did we sell in 2023
SELECT
  COUNT(DISTINCT order_items.id) AS count_of_items_sold
FROM
  `genbi-ecommerce`.`raw_thelook`.`order_items` AS order_items
WHERE
  TIMESTAMP_TRUNC(order_items.delivered_at, YEAR) = TIMESTAMP '2023-01-01 00:00:00 UTC';

The SQL was valid. The number was plausible. The question genuinely doesn't specify whether returns count. The model made a business decision and presented the result as a fact — and that's a worse failure than broken SQL, because broken SQL announces itself and this doesn't.

Run 2: a vague description

I added a description for the status column on the order_items table:

ALTER TABLE `project.raw_thelook.order_items`
ALTER COLUMN status SET OPTIONS(
  description="Sales metrics only include complete items by default."
);

Hit rate: 6/10

Better, and unstable. Sometimes it filtered correctly. Sometimes it ignored the description entirely. Same question, same schema, same description — different answers across runs.

-- a run where it filtered correctly
SELECT
  COUNT(order_items.id) AS count_of_items_sold
FROM
  `genbi-ecommerce`.`raw_thelook`.`order_items` AS order_items
WHERE
  order_items.status = 'Complete'
  AND TIMESTAMP_TRUNC(order_items.created_at, YEAR) = TIMESTAMP '2023-01-01 00:00:00';

In the other 4 runs, it reverted to counting all rows — sometimes even querying a different table entirely. Looking at the description again, it was bad. "Complete items" asks the model to infer two things: that I meant status = 'Complete', and that this excludes returns. Either inference can go the wrong way on any given run.

Run 3: a precise description

I made the description more precise:

ALTER COLUMN status SET OPTIONS(
  description="Line-item fulfilment status. Values: Complete, Shipped, Processing,
  Cancelled, Returned. Sales and revenue metrics count only lines with
  status = 'Complete'; exclude Cancelled and Returned."
);

Hit rate: 9/10

-- how many items did we sell in 2023
SELECT
  COUNT(DISTINCT order_items.id) AS count_of_items_sold
FROM
  `genbi-ecommerce`.`raw_thelook`.`order_items` AS order_items
WHERE
  order_items.status = 'Complete'
  AND EXTRACT(YEAR FROM order_items.delivered_at) = 2023;

Listing the actual values and using imperative phrasing helped measurably.

This is the finding I didn't expect. Description quality matters, not just description presence. If you're about to spend a sprint filling in your data catalog, that distinction is worth knowing before you start — a catalog full of one-line summaries may buy you much less than the effort suggests.

The part that still bothers me

Even at the best configuration, it wasn't 10/10.

A description is advisory. It shifts a probability. It doesn't remove an option. Nothing in the mechanism makes the correct filter mandatory, so some fraction of the time the model weighs it differently and skips it.

The status filter isn't the only thing that shifted between runs. The model also varied which timestamp column it used — delivered_at in some runs, created_at in others. That's another business-logic choice being made silently, and another axis of non-determinism that a description on a single column can't fix.

And the variance is a bigger problem than the failure rate. A system that's consistently wrong is fixable — you find the bug, you fix it. A system that's right 9 times out of 10 can't be trusted even when it's right, because there's no way to tell which run you're looking at. Someone spot-checks it once, sees a correct answer, and draws exactly the wrong conclusion about reliability. This is why I'm running every configuration 10 times rather than once.

Two things I noticed but haven't measured

order_items carries both a product_id and an inventory_item_id, so there are two join paths to the same product. Across these runs the model sometimes took one and sometimes the other. I replaced the raw tables with a single curated view where the joins are already made, and that variance went away — there was no second path to choose. But I haven't counted it properly, and I haven't verified whether the two paths return different numbers.

Separately, the timestamp column variance I mentioned above needs its own controlled test. It's a business rule that should be defined somewhere the model can find it, and it's the kind of thing a semantic layer is designed to solve.

Both are separate experiments. I'll post them when I've run them.

What I think this means

If you're evaluating GenBI tools, the interesting question isn't "can it write SQL." It's "what happens when the question is under-specified," which is most real questions.

Documentation is worth doing, and doing precisely. It also isn't sufficient on its own. My working hypothesis — which is what the rest of this project exists to test — is that the last gap closes by not letting the model make certain choices at all: routing defined-metric questions to compiled SQL, and having the system ask rather than guess when a question is genuinely ambiguous.

I'll find out whether that's true and post the numbers either way.

Caveats

One question. One dataset. One trap. One model, tested 2026-08-28, 10 runs per configuration. Gemini in BigQuery is doing something genuinely hard here — inferring business meaning from a schema that doesn't contain it — and these tools change month to month. Treat this as a snapshot, not a verdict.

Part of a project building an evaluation harness for AI-generated SQL on a dbt semantic layer. Repo and full methodology to follow. Repo: github.com/Sean-Liu-GitHub/genbi-ecommerce

What Took BI Engineers Hours, WrenAI Did in 15 Minutes — A Hands-On Test with Manufacturing Data

Sean Liu — Thu, 13 Aug 2026 03:32:52 GMT

As a data engineer with years of experience building pipelines and dashboards, I've always felt the traditional BI workflow could be faster: connecting data sources, writing SQL, designing visualizations, tweaking layouts, iterating with stakeholders.

So when I heard about GenBI (Generative Business Intelligence) — the idea that AI can generate dashboards from natural language — I was skeptical. I decided to put it to the test with a real dataset and a real use case.

The Dataset: Manufacturing Downtime

I used the Manufacturing Downtime dataset from Maven Analytics, a well-known practice dataset that simulates a soda bottling production line. It contains four tables:

Line Productivity — a fact table with details for each batch produced, including operator, product, and start/end times
Products — a dimension table with product details like flavor and size
Line Downtime — a fact table recording downtime in minutes by factor for each batch
Downtime Factors — a dimension table categorizing each downtime factor (e.g., machine failure, operator error, inventory shortage)

The business questions I wanted to answer were straightforward but typical of what a manufacturing operations team would ask:

What are the leading factors causing downtime?
Which operators are experiencing the most downtime?
Are certain operators struggling with specific types of issues?

The Traditional Approach

In a traditional BI workflow, answering these questions means exploring the schema, figuring out that the downtime table needs to be unpivoted first, writing SQL joins across all four tables, building aggregations, choosing chart types, and arranging everything into a coherent dashboard. For a seasoned BI engineer, this typically takes 2–4 hours. For someone less experienced, it could take a full day.

But the bigger problem isn't speed — it's that the knowledge isn't transferable. The metrics live in SQL queries, in DAX formulas, or in a dbt semantic layer. The business logic lives in the engineer's head. Some good data teams write documentation, but it still takes time for a new analyst to internalize it. And every time a stakeholder requests a new dashboard, the whole process starts over.

This is where a semantic layer changes the equation — not just by speeding things up, but by making the business context explicit, reusable, and queryable by anyone.

The Wren AI Approach: From Excel to Dashboard in 15 Minutes

Here's where things got interesting. Wren AI is an open-source GenBI engine that lets you connect a data source, define a semantic model, and then ask questions in natural language. It generates SQL, runs it, and produces visualizations.

Setting Up the Project

The dataset is a single Excel file with a CSV data dictionary — that's it.

I asked Claude Code to set up the Wren AI project by pointing it at the quickstart guide:

Read the quickstart guide in this url https://docs.getwren.ai/oss/get\_started/quickstart. Guide me how to set up a Wren AI project for the Excel file in this folder.

Claude Code handled the entire setup: created a Python environment, installed Wren AI and DuckDB, loaded all four Excel sheets into a DuckDB database, installed the Wren CLI skill, created a database profile, and scaffolded the project structure.

The most notable thing: Claude automatically detected that the Line Downtime table was in wide/pivoted format and unpivoted it into a normalized (batch, factor, downtime_minutes) structure — a data modeling decision that would normally require human judgment.

Setting Up the Semantic Layer (5–10 minutes)

The semantic layer is where you tell the AI what "downtime" means, how tables relate, and what metrics matter. Think of it as teaching the AI your business context.

Using the /wren skill, Claude Code generated a metadata YAML file for each table and a relationships file defining how they connect:

For example, the downtime factors table:

And the relationships between all four models:

Asking Questions

Once the semantic model was in place, I simply asked natural language questions.

"What are the top downtime factors by total minutes?"

Machine adjustment is the biggest downtime factor at 332 total minutes, followed by Machine failure (254) and Inventory shortage (225). Notably, the top contributor is an operator error, but the next two are not — so downtime splits roughly evenly between operator-caused and non-operator causes.

View the generated SQL and results