Hallucination of the Week: the AI math errors that 'are you sure?' cannot catch

We covered the headline finding of Confidently Wrong in our roundup of what research says about AI hallucinations on financial numbers. The paper by Richard Zhe Wang of St. John Fisher University was revised on October 13, 2026, and is accepted at the 7th ACM International Conference on AI in Finance (ICAIF '26), in Milan on November 14 to 17. This week we look at the two parts that matter most if you use AI to pull numbers from filings: what kind of mistakes slip through, and how many a human reviewer can realistically catch.

The setup in one paragraph

Three open models, Qwen3-8B, Llama-3.1-8B and Gemma-2-9B, answered numerical questions built from real company filings in two benchmarks, FinQA and TAT-QA. Each question was asked eight more times. If all eight resampled answers matched, the answer counted as confident. Among those confident answers, 15% to 23% were still wrong on FinQA nContext: n=3 models, about 6,100 FinQA questions each · paper v2, Oct 13, 2026 · Wang, arXiv:2607.11414 · margin: not disclosed.

The paper then compared two kinds of alarm. Cheap output-level signals: token probabilities, and asking the model whether its own answer is true (called P(True)). And a probe, a small classifier that reads the model's internal activations while it writes the answer.

The confident mistakes are mostly math

The author typed every wrong FinQA answer mechanically against the benchmark's gold reasoning. A computation error uses all the right numbers from the filing but applies the wrong operation or slips on the arithmetic. A selection error leaves out a needed number, for example by reading a different line item or period. A sign error gets the magnitude right and the direction wrong.

AUROC measures how well a signal separates wrong answers from right ones: 0.5 is a coin flip, 1.0 is perfect. The pattern is clear. On computation and sign errors, the output-level signals were close to useless, at 0.47 to 0.61, while the probe reached 0.70 to 0.81. In plain terms, when a model confidently botches the arithmetic or flips a sign, asking it 'are you sure?' does very little. Abstentions, answers like 'the filing does not provide this figure', were easy for every method to spot.

How many errors a review team catches

Real teams cannot check every AI answer. So the paper fixed a review budget and asked which ranking method sends the most wrong answers to the reviewer. For Qwen3-8B on FinQA, reviewing 20% of answers caught: nContext: n=6,105 FinQA questions (Qwen3-8B) · 20% review budget · Wang, arXiv:2607.11414, Table 5 · 95% CI on probe gain vs P(True): 3.4 to 11.6 points

• 45% of all errors when answers were ranked by the probe
• 38% with a supervised combination of the output-level signals
• 37% with the model's own true-or-false self-check
• 20% with random review

For Gemma-2-9B, the gap over the self-check was 12.0 points (95% interval 7.6 to 16.6). For Llama-3.1-8B, whose output signals are already more informative, the probe only matched the output combination. On confident-wrong answers alone, output-level routing caught 7% to 25% at a 20% budget, while the two probe variants caught 26% to 40% nContext: n=329 to 1,137 confident-wrong answers per model · 20% review budget · Wang, arXiv:2607.11414, Table 5 · margin: not disclosed.

Among Qwen3-8B's confident answers, flagging the 5% most suspicious gave 79% precision at 17% recall, and a 30% budget gave 45% precision at 60% recall nContext: n=2,503 confident FinQA answers (Qwen3-8B, correct answers subsampled and reweighted) · paper v2 · Wang, arXiv:2607.11414, Figure 1 · margin: not disclosed.

The fine print

• Self-hosted models only. The probe needs access to the model's internals. With ChatGPT, Claude or Gemini through an app or API, only the weaker output-level signals are available.
• Small models. All three are in the 8 to 9 billion parameter range; frontier models were not tested.
• Task-specific. A probe trained on FinQA lost 0.07 to 0.16 AUROC on TAT-QA, so it must be retrained for each task.
• A higher floor than it looks. A placebo probe on a randomly initialized model already scored about 0.63, so gains should be read against that, not against 0.5.
• Numbers only. The study covers numerical answers that can be graded mechanically, not free-text analysis. The author estimates residual label noise of 4% to 6% in the grading nContext: n=120 re-graded pairs from an 800-question pilot · paper v2 · Wang, arXiv:2607.11414 · estimate, no interval disclosed.

Code and per-example scores are public on the author's GitHub.