GenBI: generative Business Intelligence without wrong numbers

Why does « plugging an LLM into the database » produce hallucinations?

An LLM is a probabilistic language model: it predicts the most plausible token, not the most accurate one. When you ask it to produce a number, it has two ways of doing so — and both are problematic without guardrails.

First case: you inject rows of data into its context and ask it to « do the maths ». The model has no reliable calculator: it aggregates mentally, rounds, drops rows, adds up incorrectly. Over a sum of a few dozen values, the silent error rate is high — and nobody notices, because the answer is delivered with total confidence. That is number hallucination: a wrong figure presented as truth.

Second case: the LLM generates free-form SQL. Here the calculation is done by the database (correct by construction), but the danger moves up a level: the model itself chooses the tables, the joins and the definition of the metric. Yet « revenue » has no universal definition: net or gross? Including or excluding VAT? Refunds deducted? With or without intra-group sales? The LLM decides at the whim of the context, and every question can yield a different definition. You end up with figures that never reconcile from one conversation to the next.

A story from the field. On one engagement, two departments presented a « monthly revenue » figure to the steering committee, both drawn from the same conversational assistant wired up as free-form text-to-SQL: €4.2M for one, €3.9M for the other. Depending on how the question was phrased, the model had sometimes included credit notes, sometimes not. Half a day lost reconciling two « accurate » yet inconsistent numbers — exactly the kind of scene a semantic layer eliminates at the root.

Is text-to-SQL necessarily dangerous?

No — governed text-to-SQL sits at the heart of good GenBI. What is dangerous is free-form, unconstrained text-to-SQL. The nuance: the LLM does not generate SQL against the database’s physical schema; it calls, via function calling / tool use, an API of pre-validated metrics. The final SQL is composed by the semantic layer, not by the model. You keep the flexibility of natural language without giving up the rigour of shared definitions.

Is the semantic layer enough? The blind spot you need to know

Let’s be honest: the semantic layer eliminates calculation and definition hallucination, but not the subtlest risk of all — intent mis-mapping. The LLM can return a perfectly exact number… that answers the wrong question. Three classic cases: it picks the wrong metric (gross revenue instead of net), the wrong dimension (billing region instead of delivery region), or the wrong time filter (calendar quarter instead of fiscal quarter). The answer is « true » in the sense that it is correctly calculated, but it is beside the point — and therefore misleading.

This is the blind spot most demos quietly skip. We address it with two mechanisms. First, clarifying: when the intent is ambiguous, the assistant does not guess — it rephrases (« do you want net or gross revenue, over the fiscal quarter? ») before executing. Second, continuous evaluation: a reference set of questions, each with the expected metric/dimension/filter, replayed at every change to the model or the layer, to measure the correct-mapping rate — not just the accuracy of the number. Presenting governance as a total solution would be dishonest; presenting it as an architecture that makes the risk measurable and correctable is the operational reality.

And compliance: where do the questions go, what does the LLM see?

This is the DNA of our sister company Datanaos, and a point too often neglected in GenBI. The structural good news: in a governed architecture, the LLM never sees the data rows. It handles metric and dimension names, not amounts or personal data — minimisation is native, which sharply reduces exposure under the GDPR. Two questions still need an explicit answer: where users’ questions are sent (an EU-hosted or self-hosted model on AWS/Docker may be required in sensitive contexts) and usage traceability under the EU AI Act. Prompt governance and LLM cost tracking (logs, quotas, per-profile caps) belong to the trust layer just as much as the evals do: without them, a GenBI drifts silently, in quality and in budget alike.

The pitfalls to avoid on a GenBI project

• Letting the LLM calculate. Any number produced by the model rather than by the database is suspect. The rule: translation yes, calculation never.
• Skipping the semantic layer. Without shared definitions, your figures diverge between conversations and between departments. It is the number-one cause of lost trust.
• Ignoring the mis-mapping risk. An exact but off-target number is more dangerous than a visible error. Clarifying and mapping evals are non-negotiable.
• Starting without a data map. A semantic layer built on poorly understood data inherits all of its ambiguities. The data foundation comes first.
• Forgetting access control and compliance. If scope is not enforced in the layer, natural language becomes a back door — and the question « where do the prompts go? » must be settled before production.
• Confusing demo with production. An impressive demo on three questions says nothing about robustness across hundreds of real, concurrent queries.

These are the same pitfalls that sink AI POCs in general. For the full framework on scaling up, read our checklist on taking an AI POC to production, and for the architecture of the agents and orchestrations feeding these flows, our guide to n8n, MCP and LLM agents.

Want to know whether your data foundation is ready for GenBI? Our AI assessment & scoping — a short, fixed-price engagement, no commitment — identifies the priority metrics and the data-mapping effort still ahead. Also discover our end-to-end GenBI offer.