# The 74-Point Accuracy Gap: Why AI Analytics Fails Without Human Curation (And How to Fix It) Anthropic recently published how they run self-service data analytics internally. One number stood out: **without structured skills docs, their AI agent scored 21% accuracy. With them, it hit 95%+.** That 74-percentage-point gap wasn't closed by a better model, a fancier framework, or a new tool. It was closed by people writing markdown files. This article breaks down what Anthropic tested, what didn't work, and a generalizable process any data team can follow to build the artifacts that actually move the needle. --- ## What Anthropic Tested — And What Failed Before we talk about what works, it's worth understanding what Anthropic tried that **didn't** work. These findings challenge common assumptions about AI analytics. ### Failed: Auto-generated semantic layer definitions Anthropic tried having an LLM auto-generate metric definitions from raw tables and query logs. The result was **net-negative** on their evaluation set — worse than having no semantic layer at all. Why? The auto-generated definitions produced plausible-looking metrics that actually encoded the very ambiguities the team was trying to eliminate. When "revenue" has three valid interpretations in your data warehouse, an LLM will pick one without understanding why the others exist. **Takeaway:** Use AI to generate *documentation* (descriptions, summaries). Have a human own the *definition* (what the metric means, how it's calculated, which table is canonical). ### Failed: Raw query corpus access This is the surprising one. Anthropic gave their agent direct `grep` access to their entire corpus of dashboard SQL, transformation code, and analyst notebooks — thousands of files. They verified in transcripts that the agent actually read the relevant queries before answering. Accuracy moved by **less than one percentage point**. They checked the obvious confounds: Was the correct answer in the corpus for questions it got wrong? About 80% of the time, yes. Did "answer present" predict "now gets it right"? No — the flip rate was flat. **The information was there. The agent saw it. It still didn't use it correctly.** The bottleneck wasn't *access* to prior work. It was *structure* — mapping a question to the right entity. Unstructured retrieval couldn't connect a new question to the right precedent. ### Failed: Additional rounds of doc refinement past a certain point They hit three consecutive net-negative iterations where docs were getting longer, not better. More words didn't mean more accuracy. ### Failed: Cheaper model for adversarial review Swapping the adversarial reviewer to a cheaper model to cut latency lost most of the accuracy gains without meaningful speedup. --- ## What Actually Worked Three artifacts accounted for nearly all of the accuracy improvement: | Artifact | What it does | Impact | |---|---|---| | **Skills** (structured markdown instructions) | Routes the agent to the right sources in the right order | 21% → 95%+ accuracy | | **Human-curated semantic layer** | Single source of truth for metric definitions | Mandatory first step for every query | | **Curated reference docs** | Domain-specific tables, gotchas, and patterns distilled from the query corpus | Solved retrieval failure | A fourth technique — an **adversarial review sub-agent** — added 6% accuracy on top, at the cost of 32% more tokens and 72% higher latency. --- ## The Process: Building These Artifacts From What You Already Have The knowledge required to build these artifacts isn't new. It already exists in your organization — scattered across dashboards, Slack threads, analyst notebooks, and people's heads. The process is extraction and structuring, not creation from scratch. ### Phase 1: Mine Your Existing Resources (1–2 weeks per domain) #### Step 1 — Identify your top 20 business questions per domain Your company already asks these questions repeatedly. Find them in: - **Dashboard titles and chart labels.** Every chart is essentially a natural language question with validated SQL behind it. "Monthly Revenue by Region" is both the question and proof that someone already wrote correct SQL to answer it. - **Slack channels** where people request data from your team. These contain the real phrasing, real ambiguity, and real business jargon your users actually use. Search for patterns like "can someone pull..." or "what's the latest..." - **Query logs.** Every warehouse (Snowflake, BigQuery, Databricks) records query history. Filter to analytical queries (not ETL jobs), cluster by business domain, and identify the most frequently run patterns. - **Recurring meeting agendas.** "Review weekly active users" and "check pipeline by region" are business questions hiding in plain sight. Don't try to cover everything. Anthropic found diminishing returns past a few dozen evals per topic. Start with the questions that get asked every week. #### Step 2 — For each question, trace back to the canonical answer Open the dashboard or notebook that answers it. Find the SQL. For each question-SQL pair, document: - Which tables and columns were used - Which filters were applied (and why) - Which definition of the metric was used - What a senior analyst would warn you about **This is where you discover ambiguity.** You'll find that "revenue" is calculated differently in three dashboards. That's not a bug — it's the most valuable finding. Resolve it now with your business stakeholders and pick one canonical definition. #### Step 3 — Document the gotchas Ask your most senior analyst: *"What do new hires get wrong in their first month?"* That list is your gotchas doc: - "This field double-counts if you don't filter by X" - "Status = 4 means refunded, not cancelled" - "This table has a 2-day lag — always use MAX(date), not 'yesterday'" - "We changed product names in Q3 2025 — old values are frozen in the data" These are the things that make an answer go from "valid SQL" to "correct business answer." --- ### Phase 2: Structure the Artifacts Now convert what you found into the three artifacts that Anthropic showed actually work. #### Artifact 1: Reference Docs (One Per Domain) This is the most impactful artifact. It replaces unstructured tribal knowledge with a document an AI agent can reliably navigate. Anthropic shared the skeleton they use internally. Here it is, adapted with annotations: ```markdown # [Domain] Tables ## Quick Reference ### Business Context [What this domain means in plain words. Not technical — write it for someone who just joined the company.] ### Entity Grain [What one row represents. This single line prevents the most common join errors.] ### Standard Hygiene Filter [The filter every query in this domain applies. Example: "WHERE is_test_account = FALSE AND event_date >= '2024-01-01'"] ## Dimensions [How the key dimensions are encoded, and how the same concept is named differently across tables. Example: "region" in table A = "geo" in table B = "market" in table C] ## Key Tables ### [table_name] - **Grain**: [what one row represents] - **Scope/exclusions**: [what's included, what's deliberately left out] - **Usage**: [when to use it, when NOT to use it, join keys, required filters] ### [table_name_2] [... one section per governed table ...] ## Gotchas [The wrong-answer modes a senior analyst would warn you about. These are the things that make the difference between "valid SQL" and "correct business answer."] ## Best Practices / Common Query Patterns [Default choices, standard cuts, worked patterns where the exact query form is the hard part.] ## Cross-References [Neighboring domain docs that own adjacent questions. "For customer lifetime value, see finance/ltv.md"] ``` **Key principle:** Write these docs for retrieval by an AI agent, not for human reading. That means explicit routing triggers ("IF the question is about experiment lift... DO NOT use for raw event counts"), not narrative explanations. #### Artifact 2: Skills (Routing Instructions) Skills are what turned Anthropic's accuracy from 21% to 95%. They're the procedural knowledge: which sources to consult in what order. Anthropic's approach uses two paired skills per domain: **Knowledge skill** — a thin router that narrows the search space: ```markdown --- name: [domain]-knowledge description: "IF the user asks about [domain topics] — THEN invoke this skill." --- # [Domain] Knowledge ## Semantic Layer (REQUIRED first step) The governed semantic layer is the mandatory default path for every data question. Raw SQL is the fallback, used only after the semantic layer is shown not to cover the ask. ## Required workflow 1. **Search** the semantic layer for matching metrics/dimensions 2. **Check segments** — named canonical population filters (hand-rolled WHERE clauses for these are the dominant wrong-answer mode) 3. **Compile and run** if found 4. **Fallback** to reference docs ONLY if no semantic layer coverage > Don't bail early. Do NOT fall back to raw SQL because: > - "Needs a custom date filter" → [covered by time-dimension specs] > - "Needs a join" → [the metric layer already encapsulates joins] ## Date Conventions - "Last week/month" → last *complete* calendar week/month, not trailing-7/30 - Timezone default: [your timezone] - Freshness lag: [which tables settle late] ## Reference Docs Navigation ### [Sub-domain A] → `references/[sub_domain_a].md` - **Use for**: [kinds of questions] - **Key tables**: [...] ### [Sub-domain B] → `references/[sub_domain_b].md` - **Use for**: [...] ## Gotchas - Use `[field_x_v2]` NOT `[field_x]` - [Two tables report the same metric at different grains — which to use] - [Which of two plausible sources is canonical] ``` **Runbook skill** — encodes the analysis process: ```markdown --- name: [domain]-runbook description: "Execution process for [domain] analysis questions." --- # [Domain] Analysis Runbook ## Step 1: Clarify the question - Time period? Segment? What business decision does this inform? - Disambiguate terms (see Entity Disambiguation in knowledge skill) ## Step 2: Check for existing dashboards - [Link to domain dashboard catalog] ## Step 3: Find sources - Invoke [domain]-knowledge skill - Follow its semantic layer → reference doc → raw table priority ## Step 4: Execute and validate - Run the query - Apply adversarial review: challenge all assumptions - Check data freshness ## Step 5: Report with provenance Every answer ends with: > **Source:** [semantic layer | governed table | raw exploration] > **Freshness:** [max date in data] > **Owner:** [owning team] ## Reusable Analysis Patterns ### Retention curve [Standard approach for this domain] ### Funnel analysis [Standard approach] ### Rate decomposition [Standard approach] ``` #### Artifact 3: Semantic Layer (Human-Curated Metrics) Start small: just the top 10–15 metrics per domain. For each one, you need: - The exact calculation - The grain - Required filters (always applied) - Dimensions it can be sliced by - Who owns it The tool doesn't matter — dbt MetricFlow, Cube, warehouse-native semantic views, or even a well-structured YAML file. What matters is that a human defined it and a business stakeholder signed off. **Do not auto-generate these with an LLM.** Anthropic tested it. It was net-negative. --- ### Phase 3: Validate and Maintain #### Build your evaluation set Your top 20 questions per domain from Phase 1, paired with verified answers, become your offline eval set. Format: ```json { "domain": "revenue", "question": "What was total revenue by region last month?", "gold_sql": "SELECT region, SUM(amount) FROM fct_orders WHERE ...", "expected_behavior": "Should use semantic layer metric 'total_revenue'", "snapshot_date": "2026-09-01", "difficulty": "simple", "owner": "finance-analytics" } ``` Anthropic's guidance: - **Target ~100% offline accuracy** before launching a domain - **Pin every eval to a snapshot date** — an eval against live data goes stale when the underlying number moves - **Store results like telemetry**, with skill version, model ID, and per-assertion pass/fail - **Gate launches per domain** — a domain owner can't announce the agent to stakeholders until their eval set clears threshold #### Maintain like a first-class product Anthropic watched accuracy drift from ~95% to ~65% within a month when they stopped maintaining docs. Their solution: - **Colocate everything in one repo.** Skill markdown, reference docs, and transformation models live together. The PR that changes a data model must also update the docs. - **CI hook** flags any reporting-model change that doesn't touch a skill file - **90% of data-model PRs** now include a skill change in the same diff - **Harvest corrections.** A scheduled agent scans Slack for correction language, drafts a fix to the relevant reference doc, and opens a PR --- ## The Uncomfortable Truth The most valuable skill in AI-powered analytics isn't prompt engineering, model selection, or framework architecture. It's the unglamorous work of writing and maintaining structured documentation about what your data actually means. This can't be automated. The knowledge is tribal — it lives in people's heads, in Slack threads, in the footnotes of dashboards nobody reads. The job is extracting that knowledge into a structured format an AI agent can follow. Anthropic proved the math: 21% without human curation, 95%+ with it. The gap isn't closed by better models. It's closed by analytics engineers doing what they've always done — understanding the business — and writing it down in a way machines can use. --- ## Sources - [How Anthropic enables self-service data analytics with Claude](https://claude.com/blog/how-anthropic-enables-self-service-data-analytics-with-claude) — Anthropic, June 2026 - [BEAVER: An Enterprise Benchmark for Text-to-SQL](https://arxiv.org/html/2409.02038v3) — The first text-to-SQL benchmark derived from real enterprise data warehouses - [Text-to-SQL for Enterprise Data Analytics](https://arxiv.org/html/2507.14372v1) — LinkedIn's internal SQL Bot paper, reporting 53% accuracy on enterprise data - [From analytics engineer to context engineer](https://www.getdbt.com/blog/from-analytics-engineer-to-context-engineer) — dbt Labs, on the role shift