Raw SQL vs. a governed knowledge graph: 77 business questions tested
We asked the same AI model 77 everyday business questions about our own company, three times each. Even with direct SQL access to the raw data, it matched our own definitions about 60% of the time — against 86% through the Company Twin.
When teams point an AI assistant at their company data, the real risk is rarely a made-up number. It is a plausible, defensible number that quietly disagrees with the board pack.
To measure how often that happens in practice, we ran a controlled test on our own company data. We asked the same AI model 77 everyday business questions — 54 asked by our own team inside the product and 23 written for the test suite — three times each (231 runs per arm). In one arm, the model had direct read-only SQL access to our raw synced tables. In the other, it queried our Company Twin over MCP. A blind judge graded every run against reference answers, then audited the results twice on extra-high effort. Every figure below reproduced cleanly.
Key takeaways
Even with direct SQL access to the raw synced tables — more than a typical assistant connector gets — the model matched our own definitions only 58–61% of the time (45–47 of 77). Through the Company Twin, the same model matched 86% (66 of 77).
Raw SQL was rarely blatantly wrong (66–67 of 77 were defensible under some reading). It quietly picked its own definitions: counting on-hold deals as open, including archived records, or folding a half-finished month into a rolling year.
Through the Twin, the model gave the same answer across all three runs more often (66–73 vs 59), finished about a quarter faster, and needed about 40% fewer steps (median 8 tool calls vs 14).
The Twin arm read more input tokens at the median than raw SQL (186–195k vs 163k), though it wrote fewer output tokens (1.4k vs 2.2k). A follow-up MCP change sending the brief once instead of twice trims input tokens by about 5% with no drop in accuracy.
Caveats upfront: this is one organisation's data (our own), and the reference answers come from the Twin's own agreed definitions. For any customer, matching board reports holds only once finance and operations sign off those definitions.
How we ran the test
Both arms used the exact same reasoning model (GPT-6.2 Sol, running in Codex) against the exact same underlying Mercury Labs company records. Each of the 77 questions ran three times one-shot, with no human in the loop to answer follow-up questions.
Raw SQL arm: read-only SQL straight over the synced source tables from our CRM, project tracking, accounting and operations systems. That is a strong baseline — far better access than pointing a chat assistant at files or individual SaaS APIs.
Company Twin arm: the same model connected to our Company Twin through its MCP endpoint, where entities are already linked across systems and business definitions are encoded in a model-free graph engine.
Blind judging and audit: a separate model (GPT-6.2 Astra) graded all 462 runs blind against our reference answers, then audited the scoring twice at extra-high effort.
Direct SQL vs. Company Twin (77 questions × 3 runs each)
Measure
Direct read-only SQL
Company Twin (MCP)
Matches our own definitions (right in 2 of 3 runs)
45–47 / 77 (58–61%)
66 / 77 (86%)
Defensible under any stated, reasonable reading
66–67 / 77
68–69 / 77
Same answer across all 3 runs
59 / 77
66–73 / 77
Relative end-to-end speed
Baseline
About a quarter faster
Median tool calls per question
14 steps
8 steps (~40% fewer)
Median tokens (input / output)
163k in / 2.2k out
186–195k in / 1.4k out
Wrong answers vs refusals (out of 231 runs)
15 wrong, 1 refused
10–11 wrong, 3–4 refused
One detail on speed: most of the wall-clock time in both arms was the reasoning model thinking inside Codex. The Twin's own graph calls took roughly 0.2 seconds. Because the model no longer had to explore schemas, guess joins and reconcile edge cases across 14 queries, the full run finished about a quarter faster in roughly 40% fewer steps.
The real finding: quiet definition drift, not hallucination
Look at the first two rows of the table together. Under a generous reading — "is this answer defensible if you accept whatever definition the model chose?" — direct SQL scored 66–67 out of 77. The model almost never invented figures out of thin air.
Yet when graded against the definitions our own business actually runs on, direct SQL dropped to 45–47 out of 77 (about 60%). Without announcing what it was doing, the SQL arm repeatedly picked reasonable-sounding rules that disagreed with our reports:
Counting on-hold deals as part of the active open pipeline.
Including archived project or client records in live counts.
Folding the current unfinished part-month into a rolling annual figure.
Mixing up which system was authoritative when two tools held overlapping copies.
In a board meeting or client review, that is worse than an obvious error. A wild hallucination gets caught in seconds. A tidy number that is 12% off because it included three archived records quietly erodes trust in every AI answer after it.
What the Twin's 11 misses actually were
Even on the Company Twin arm, 11 of the 77 questions did not match the strict reference on at least two of three runs. Auditing every miss turned out to be more instructive than the headline score:
Four were errors in our own reference answers, not in the graph engine.
One was a deliberate refusal: there were no records for that year, and the Twin refused to turn "nothing recorded" into zero.
Two returned both valid sources, labelled with their dates (for instance, live operational records alongside a channel copy), where the strict reference accepted only one.
The remaining four were genuine definition edges — such as whether an invoice due today counts as overdue, or whether an internal organisation counts as a customer.
If you re-score to accept answers that return both valid sources with clear labels, the Twin comes out around 72 of 77 (94%). We have not audited that re-score, and raw SQL gains on some of the same questions too, so we quote 86%.
Caveats you should ask about before trusting any benchmark
Three things keep this result honest. First, this is one organisation's data: our own. Second, the reference answers come from the graph model's own queries, which we wrote — so the strict score partly rewards the Twin for following our own conventions. That consistency is the point of governed definitions, and for a customer "matches your own reports" holds only once their finance and operations leads sign off those definitions. Third, wrong answers still outnumbered refusals in both arms (15 vs 1 on SQL; 10–11 vs 3–4 on the Twin), so explicit refusal boundaries still matter.
On tokens, ambiguity and who asks for clarification
Two honest operational lessons came straight out of the traces.
First, input tokens were not lower on the Twin. At the median, the Twin arm read more input tokens than raw SQL (186–195k vs 163k) because the MCP brief and structured definitions travel in context, even though it wrote fewer output tokens (1.4k vs 2.2k). A small MCP change that sends the brief once per session instead of twice trims input tokens by about 5% with zero loss in accuracy, but the Twin wins on consistency, repeatability and step count — not on reading fewer input tokens.
Second, ambiguity belongs to the conversation, not the graph. Where a business has already settled a rule (like excluding archived records), the Twin enforces it before anyone asks. Where two readings are both genuinely valid, the Twin's job is to return both with source and freshness labels so the outside assistant — Claude, ChatGPT or Microsoft 365 Copilot — can state the reading it used and offer the other, or ask a quick clarifying question when the two numbers diverge sharply. Because this test ran one-shot with nobody to answer back, neither arm could ask.
What we are testing next
This run measured the model directly in Codex over MCP. We have not yet run the live Microsoft 365 Copilot comparison, so we are not claiming a live Copilot score yet. Once our latest MCP release is live in production, we will run a live Copilot test that includes deliberately ambiguous questions — scored right when Copilot either asks a sensible clarifying question or states its reading clearly and offers the alternative.
How to test AI accuracy on your own company data
04 stages
Stage01
Take 20–30 questions your team actually asks every week, not synthetic demos.
Watchpoint
Testing only tidy single-table questions hides the cross-system join and definition bugs.
Stage02
Run every question three times and check whether the number moves.
Watchpoint
A single lucky run looks impressive in a demo and fails quietly in Monday's finance review.
Stage03
Grade against your signed-off board or finance definitions, not 'looks plausible'.
Watchpoint
Counting on-hold deals or unfinished months produces confident answers that break trust.
Stage04
Check what happens when data is missing or restricted by role.
Watchpoint
Models strongly prefer guessing zero over saying 'not supported'.