The numbers the AI is not allowed to invent

Key findings (TL;DR)

• Language models are good at writing and bad at arithmetic. So in m51.ai they do not do arithmetic: every number in a report comes from one of 184 deterministic KPI formulas in a dedicated engine, not from the model.
• The model sees numbers, but only approved ones. An allowlist rejects every figure in a draft that cannot be traced to an approved fact — with dedicated safe classes for years, dates and phone numbers, so the gate does not fire on things that are not performance claims.
• Detection is deterministic; phrasing is the model's job. 16 playbooks find the problems using rules; the language model acts only as a bounded judge over what the rules already found.
• 124 automated regression checks run without a single LLM call against data packages with known answers, inside a hard 175-second budget, holding the baseline at zero critical and zero major deviations.
• One layer is an independent oracle: it recomputes the headline numbers straight from raw API responses and compares them with what the pipeline produced. It is the only check that does not trust our own arithmetic.
• Across 22 nights in July an automated review has read our own chat answers and scored them against a six-dimension rubric. 103 turns have been judged. We publish what it found, including what was wrong.

Why numbers are the hardest thing AI does

A marketing team can live with an AI-generated sentence being a little clumsy. It cannot live with the ROAS in the report reading 4.2 when it is actually 3.1. The first is an editing job. The second is a decision made on false grounds, and in the worst case a budget moved the wrong way.

The problem is that language models are built to produce plausible text, not correct numbers. A model that has seen thousands of reports knows exactly what a ROAS sentence looks like and which numbers usually sit inside it. It has no internal mechanism separating «the number I actually read in the data» from «the number that fits this sentence». From the inside, both feel equally right.

That is why «we use a better model» is not an answer to hallucinated numbers. Better models hallucinate less often, but they still hallucinate, and they do it more convincingly. The answer has to be architectural: the model must be placed where it cannot invent a number, because it was never the thing that produced it.

We have built six layers around that principle. Here is each of them, and what it actually catches.

Layer 1: The model does not do arithmetic

Every KPI in m51.ai is computed by KPIRegistry, a dedicated engine in the backend that currently holds 184 registered KPI formulas across ten domain modules: Google Ads, SEO, GA4, CRO, GEO, paid social, LinkedIn, Performance Max, portfolio management and the report KPIs.

A formula is ordinary code. ROAS is conversion value divided by cost, and nothing else. It runs before the language model is asked anything at all. When the agent later writes the analysis, it receives the KPI value already computed, together with the period it covers, the unit, and the raw field path the value was read from.

This means no language model performs a division anywhere in our production pipeline. It describes a calculation somebody else already did. The distinction sounds academic, but it is the whole difference: a description can be poorly worded, yet it cannot get the wrong answer to a calculation that was finished before it was asked.

Who does what

Layer 2: Numbers without a source do not get out

Layer 1 guarantees that the numbers which exist are correct. It does not prevent a model from writing a number that was never in the data at all. Outgoing content therefore has a numeric gate of its own.

The newsletter module carries the strictest implementation, and it is a good example precisely because it writes customer-facing copy that goes out by email without anyone necessarily proofreading it first. Generation happens in two steps — first a plan, then a draft — and both steps pass the same gate.

The model receives an explicit list of approved facts drawn from the KPI layer. Each fact has an id. If the model proposes a KPI block, it must reference an id present in that list. If the list is empty, it is not allowed to produce the KPI block at all.

The text is then tokenised, and every number in the draft is held up against the allowlist. This is where it gets interesting, because a naive numeric gate is useless in practice: it fires on years, dates, times and phone numbers, and within three days the editor has learned to ignore it.

The gate therefore has its own safe classes. Years between 1900 and 2099 are judged in context — «in 2026» or «since 2024» is a time reference, not a performance claim. Phone numbers are recognised by both format and surrounding words. Dates and times pass. URLs are masked rather than causing the whole field to be dropped, so a number inside a tracking link does not fell an otherwise correct sentence.

And the most important exception: numbers the customer wrote in the brief themselves. If you write «we celebrated 20 years in May» in your brief, the system should not refuse to repeat it. A separate allowlist is therefore extracted from the brief and applies to both the plan and the draft step, token-exact.

Headline surfaces — subject line, preheader and headings — are stricter than body copy. They inherit only bare numbers from the brief, not numbers carrying a modifier. «20 %» may appear in the body if the customer wrote it; it may not appear in the subject line on the same grounds. A subject line is the most quoted sentence in an email, and deserves the tightest gate.

Layer 3: The rules find the problem, the model explains it

The most common way to build an AI analysis engine is to hand the model all the data and ask it to find what is interesting. That produces impressive demos and unreliable production, because «find what is interesting» is exactly the task where a model has the most freedom to construct a pattern that is not there.

We do the opposite. m51.ai runs 16 playbooks in production — budget pacing, search term waste, Performance Max health, SEO content decay, striking distance, conversion tracking, landing page performance and Meta creative health, among others. Each is a deterministic rule written in a predicate language. The rule runs against the KPI values and either fires or does not.

Only once a rule has fired does the language model enter — and then in a bounded judge role. It assesses whether the finding is worth proposing to this particular customer, and writes the rationale. It can reject a finding. It cannot add one.

The consequence is an engine that is propose-only and auditable end to end. Any proposal can be traced back to the rule that triggered it, which version of the rule ran, and the values it fired on.

Layer 4: An oracle that does not trust us

Every layer above shares one weakness: it is our own code checking our own code. If KPIRegistry carries an error in a formula, every downstream layer will reproduce that error with perfect loyalty.

Hence the recompute oracle. The check takes the raw API responses from an actual run — not the data package the pipeline built, but what the platforms genuinely returned — and recomputes the headline numbers using an independent implementation. It then compares that against what the pipeline produced.

The oracle covers five headline KPIs for the weekly and monthly report, plus 26 KPIs across the five audit workflows for Google Ads, SEO, paid social, CRO and GEO. If the two computations diverge beyond tolerance, it is flagged.

It is the only check in the suite that uses not a single number the pipeline itself produced, and therefore the only one capable of catching an error in the ground truth.

Layer 5: 124 checks in under three minutes

Above this sits a deterministic regression suite. It runs 124 checks without a single LLM call, against pinned data packages with known answers, inside a hard 175-second budget — low enough to sit as a gate in CI on every change.

83 of the checks are substrate-independent contract and invariant checks. 34 require pinned fixtures with reference values. 7 require a fresh snapshot from live APIs.

The checks enforce things that must never be true. A percentage change cannot exist without a previous value to change from. A period cannot end before it begins. A description cannot be word-for-word identical to its own rationale. A monetary amount cannot be compared across currencies without a rate having been applied. Such checks are boring to write, and they catch exactly the class of defect that otherwise reaches the customer.

The baseline stands at zero critical and zero major deviations. That is the strictest state the file can hold, and it gates: if a change breaks it, the change does not merge.

Layer 6: We audit our own answers every night

The first five layers cover reports and generated content — surfaces where we control the format. The sixth covers chat, where the user asks freely and Nova answers freely. There is no schema to validate against.

The solution is a nightly automated review. Each night the day's actual chat turns are pulled from production, filtered to a chosen domain, and judged against a versioned rubric with six dimensions: grounding in data, relevance, helpfulness, tone, domain correctness and tool use. Every finding must be reproducible — noting that an answer «felt weak» does not count.

As of 30 July 2026 the routine has run 22 nights across a 24-day span, and 103 turns have been judged and stored with reasoning. Average grounding sits at 7.85 out of 10 across the 84 turns where the dimension was measurable. 20 of the 103 turns received at least one defect flag.

That number has to be read correctly. The sample is not random traffic — it is targeted at whichever domain we currently suspect most, and the judgement is deliberately adversarial: the reviewer is instructed to hunt for defects, not to assess whether the answer was good enough. 20 out of 103 is therefore not an error rate for the platform. It is the yield from looking where you are most likely to find something.

What it actually found

Two examples from the past week, both fixed.

On 28 July a Google Ads answer fabricated a data point in a chart. The prose was exactly right. The chart showed 82 GA4 sessions for a day where the figure was 10 — and 82 was the Google Ads click count for that same day. The model had copied the neighbouring series. Worse than the error itself: the point visually overturned the very conclusion the answer was built on, which was about the gap between clicks and sessions. The root cause was that chart blocks were validated nowhere — not in the backend, not the frontend, not the QA layer. The only defence was one sentence in the system prompt. It is now a deterministic check.

On 30 July an answer paired a correct pacing percentage with the wrong denominator: «136 % pacing (7,029 kr of ~5,518 kr)». 7,029 of 5,518 is 127 %, not 136 %. The percentage had been read correctly, but the pair of numbers the reader was handed was not reproducible. It occurred in two of four turns citing pacing, while two sibling turns on identical data got it right — drift, in other words, not a systematically wrong formula.

The root cause was worth more than the fix. The tool returned two different budget denominators per campaign — expected spend month-to-date and estimated monthly budget — without stating which one the percentage had been computed from. The model chose for itself, and did not always choose the same way. The fix was not to tell the model to be more careful, but to make the payload unambiguous: an explicit read contract inside the response itself, on both ad platforms.

That is the pattern we see again and again. When a model «hallucinates», the cause is more often an ambiguous contract than an inattentive model. That is good news, because ambiguous contracts can be fixed permanently — inattention cannot.

One honest limitation

We also run an online judge that samples turns continuously in production. On the night of 30 July it sampled 3 of 24 turns, or 12.5 percent, and hit none of the three turns where the deeper nightly review found factual errors. Same observation the night before. It is not the judge's quality that is the constraint, it is the sampling rate. We write it here because this is the kind of number that otherwise never leaves a company.

What this does not solve

The architecture above prevents a number from appearing out of nothing. It does not prevent a perfectly correct number from being used to support a poor recommendation. Nor does it prevent an agent from reading the right value off the wrong metric — as when the SEO agent presented Keyword Planner's competition index as SEO difficulty. That finding came from the same nightly routine, but it was about domain understanding, not arithmetic.

It also does not replace expert review. Everything the agents produce as proposals passes through an approval flow where a human decides what moves forward. That is a deliberate design choice, not a temporary safety valve we plan to remove.

What it means for your team

• The numbers in the report are the same numbers shown in Google Ads and GA4, because they were computed by code with a known formula — not restated by a model.
• Every proposal can be traced back to the rule that triggered it, and to the values that rule fired on.
• Content sent to recipients cannot carry a performance claim without an approved fact behind it.
• We find our own defects before you do, and we write down what we found.