Evaluation
The dataset, the five outcomes, the noise floor, the 13 checks a run must pass, and the conditions that void one. Then the one result we publish today.
Methodology version v2.4. Dataset: AfterQuery/FinanceQA (Apache-2.0), test split, 148 questions.
The finding
The dataset marks 46 of its questions unanswerable — the figure is not in the filing, so the only correct response is to say so. We put those questions to a commercial frontier language model with no retrieval, no sources and no verifier, under the same grader, 3 independent runs.
138 / 138
opportunities to decline — 46 unanswerable questions, 3 independent runs — every one received an invented figure. Refusals: 0.
The runs are independent, and that was measured rather than assumed: the model's answer text differs on 147 of 148 rows between runs, and 12 rows move between outcomes. It is not that the model repeated itself. It is that not once in 138 opportunities did it decline.
This is the whole argument for a verification layer, and it is why our own refusals are scored as failures against us rather than quietly excluded. A system with no way to say “the record does not show this” will always produce something.
Model: a commercial frontier language model; identity in the run artifacts and the research write-up. Closed book. Source artifacts: baseline-run-1 (sha256 044fa905421e) · baseline-run-2 (sha256 59c9b090225b) · baseline-run-3 (sha256 a70662618c2b)
Publication standard
Accuracy rates publish here when every arm of the comparison meets the acceptance bar: three clean runs and a real uncertainty band.
On answerable questions, raw frontier models currently score higher than Avarieux; the full table publishes when it meets that bar, and run artifacts are reproducible on request.
Scoring
A single accuracy percentage hides the distinction that matters most: the difference between being wrong and inventing something. These are scored separately, and one of them is treated as disqualifying.
Correct and verified
The answer matches the reference AND passed source verification. A correct figure that nothing verified does not count here — that rule is what makes this an evaluation of the pipeline rather than of a model's memory.
Refused something answerable
The question could be answered from the record and the system declined. Counted as a failure against us, deliberately: a system that refuses everything would otherwise score perfectly.
Wrong
An answer was given and it did not match the reference.
Correctly declined
The question could not be answered from the record, and the system said so instead of producing a figure.
Invented a figure
A number was produced for a question the record does not support. The outcome this company exists to make impossible, and the one measured below.
Noise floor
Before publishing any comparison we measured how much the same system varies against itself: 3 identical runs per tier, measured 2026-08-10. The headline wrong-answer rate moved by 8.4 points on one tier and 4.7 on the other, with nothing changed between runs.
Three rules follow from it. A quotable number is a three-run mean with its band. A difference smaller than the band is reported as within noise, never as a small improvement. And a fix states how many rows it expects to move before it is built.
Acceptance
A run that produces numbers is not the same as a run that happened. Every run is checked against all of them; any failure voids the run.
Voids
Blank answers
A row where the model returned nothing measures the harness, not the system. Runs carrying blanks are void rather than merely noisy.
A run that finished impossibly fast
Wall-clock time below a floor per row means the work did not happen. A fast green run is the most dangerous kind.
Answers cut off by the output budget
A truncated answer is not a wrong answer; it is an unasked question, and grading it as wrong would flatter every system that refuses early.
A control arm that used tools
The closed-book baseline is defined by having no retrieval. One tool call contaminates it, so for that arm the check inverts and any tool use is a hard failure.
The evaluation principles these run under — random sampling, point-in-time data, stated limits — and the things we explicitly do not claim are on the methodology page.