Better column descriptions made the AI right more often. Not reliably.

I assumed most "the AI can't query our warehouse" problems are really documentation problems. Before building an evaluation harness to test that properly, I ran a small experiment.

Half right, it turns out. The half that's wrong is the interesting bit.

Setup

I snapshotted BigQuery's public thelook_ecommerce dataset into my own project on 2026-08-27. Synthetic ecommerce data — orders, line items, products, inventory, users. Clean by the standards of a real warehouse, which matters: whatever breaks here breaks worse in production.

One question, asked repeatedly through Gemini in BigQuery:

How many items did we sell in 2023?

10 runs per configuration, changing exactly one thing between configurations. No prompt engineering, no hints. Just the question a business user would type.

I want to be precise about what kind of failure I was looking for. A model that writes broken SQL is easy to catch — the query fails, you see it immediately. The harder failure is when the SQL is valid, the number is plausible, and the model has silently made a business decision on your behalf. That's what this experiment is about.

The order_items table is the one that matters. Here's its schema in BigQuery — note the status column:

The trap

order_items.status holds values including Complete, Shipped, Processing, Cancelled and Returned. Nothing in the schema says which of those count as a sale.

29.2% of lines in this dataset are returned. So "how many items did we sell" has at least two defensible answers, and they differ by 29.2%.

This isn't a bug. It's ordinary warehouse ambiguity — the kind that lives in every schema I've worked on, usually resolved by an analyst who happens to know.

Run 1: no description

Hit rate: 0/10

It never filtered on status. Every run counted returned and cancelled items as sales.

-- how many items did we sell in 2023
SELECT
  COUNT(DISTINCT order_items.id) AS count_of_items_sold
FROM
  `genbi-ecommerce`.`raw_thelook`.`order_items` AS order_items
WHERE
  TIMESTAMP_TRUNC(order_items.delivered_at, YEAR) = TIMESTAMP '2023-01-01 00:00:00 UTC';

The SQL was valid. The number was plausible. The question genuinely doesn't specify whether returns count. The model made a business decision and presented the result as a fact — and that's a worse failure than broken SQL, because broken SQL announces itself and this doesn't.

Run 2: a vague description

I added a description for the status column on the order_items table:

ALTER TABLE `project.raw_thelook.order_items`
ALTER COLUMN status SET OPTIONS(
  description="Sales metrics only include complete items by default."
);

Hit rate: 6/10

Better, and unstable. Sometimes it filtered correctly. Sometimes it ignored the description entirely. Same question, same schema, same description — different answers across runs.

-- a run where it filtered correctly
SELECT
  COUNT(order_items.id) AS count_of_items_sold
FROM
  `genbi-ecommerce`.`raw_thelook`.`order_items` AS order_items
WHERE
  order_items.status = 'Complete'
  AND TIMESTAMP_TRUNC(order_items.created_at, YEAR) = TIMESTAMP '2023-01-01 00:00:00';

In the other 4 runs, it reverted to counting all rows — sometimes even querying a different table entirely. Looking at the description again, it was bad. "Complete items" asks the model to infer two things: that I meant status = 'Complete', and that this excludes returns. Either inference can go the wrong way on any given run.

Run 3: a precise description

I made the description more precise:

ALTER COLUMN status SET OPTIONS(
  description="Line-item fulfilment status. Values: Complete, Shipped, Processing,
  Cancelled, Returned. Sales and revenue metrics count only lines with
  status = 'Complete'; exclude Cancelled and Returned."
);

Hit rate: 9/10

-- how many items did we sell in 2023
SELECT
  COUNT(DISTINCT order_items.id) AS count_of_items_sold
FROM
  `genbi-ecommerce`.`raw_thelook`.`order_items` AS order_items
WHERE
  order_items.status = 'Complete'
  AND EXTRACT(YEAR FROM order_items.delivered_at) = 2023;

Listing the actual values and using imperative phrasing helped measurably.

This is the finding I didn't expect. Description quality matters, not just description presence. If you're about to spend a sprint filling in your data catalog, that distinction is worth knowing before you start — a catalog full of one-line summaries may buy you much less than the effort suggests.

The part that still bothers me

Even at the best configuration, it wasn't 10/10.

A description is advisory. It shifts a probability. It doesn't remove an option. Nothing in the mechanism makes the correct filter mandatory, so some fraction of the time the model weighs it differently and skips it.

The status filter isn't the only thing that shifted between runs. The model also varied which timestamp column it used — delivered_at in some runs, created_at in others. That's another business-logic choice being made silently, and another axis of non-determinism that a description on a single column can't fix.

And the variance is a bigger problem than the failure rate. A system that's consistently wrong is fixable — you find the bug, you fix it. A system that's right 9 times out of 10 can't be trusted even when it's right, because there's no way to tell which run you're looking at. Someone spot-checks it once, sees a correct answer, and draws exactly the wrong conclusion about reliability. This is why I'm running every configuration 10 times rather than once.

Two things I noticed but haven't measured

order_items carries both a product_id and an inventory_item_id, so there are two join paths to the same product. Across these runs the model sometimes took one and sometimes the other. I replaced the raw tables with a single curated view where the joins are already made, and that variance went away — there was no second path to choose. But I haven't counted it properly, and I haven't verified whether the two paths return different numbers.

Separately, the timestamp column variance I mentioned above needs its own controlled test. It's a business rule that should be defined somewhere the model can find it, and it's the kind of thing a semantic layer is designed to solve.

Both are separate experiments. I'll post them when I've run them.

What I think this means

If you're evaluating GenBI tools, the interesting question isn't "can it write SQL." It's "what happens when the question is under-specified," which is most real questions.

Documentation is worth doing, and doing precisely. It also isn't sufficient on its own. My working hypothesis — which is what the rest of this project exists to test — is that the last gap closes by not letting the model make certain choices at all: routing defined-metric questions to compiled SQL, and having the system ask rather than guess when a question is genuinely ambiguous.

I'll find out whether that's true and post the numbers either way.

Caveats

One question. One dataset. One trap. One model, tested 2026-08-28, 10 runs per configuration. Gemini in BigQuery is doing something genuinely hard here — inferring business meaning from a schema that doesn't contain it — and these tools change month to month. Treat this as a snapshot, not a verdict.

Part of a project building an evaluation harness for AI-generated SQL on a dbt semantic layer. Repo and full methodology to follow. Repo: github.com/Sean-Liu-GitHub/genbi-ecommerce