Saturn gave 18 AI models 121 money questions. 57% of answers failed

In September 2026, Saturn, a London firm that sells AI tools to financial advice firms, published a report called "Artificial Authority." Its claim is blunt: popular chatbots get money questions wrong more often than right. We read Saturn's report page and two trade-press write-ups. The full report sits behind a sign-up form and we have not read it, so every figure below is as Saturn reported it through those outlets.

How the test was run

Saturn tested 18 AI models, including free and paid versions of ChatGPT, Gemini, Claude and Copilot, plus Grok according to one report. It asked 121 questions on pensions, tax, debt, savings, mortgages and student loans, and repeated each question five times to check consistency. That produced more than 10,000 graded responses.

The grading rule is strict. An answer passed only if every required figure, rule, warning and deadline was present. So "wrong" in the headline includes "incomplete." One retired model was left out of the headline numbers, which leaves 17 current models.

The headline numbers

Across the 17 current models, average accuracy was 43%, which Saturn states as mistakes 57% of the time.² On hard questions, defined as multi-part scenarios with interacting tax rules and precise figures, accuracy was 12% and mistakes 88%. On the easiest questions, general financial literacy with no calculation, accuracy averaged 54%.

Free models failed 63% of the time against 49% for paid models. The best overall performer, Claude Opus 5 (reasoning), passed 61% of the time, which still means almost four in ten answers failed. On hard questions even that top model failed 67% of the time.²

Two ChatGPT rows appear because the two outlets named different variants, and we cannot tell from coverage whether they are the same configuration.

How the answers went wrong

InvestmentNews lists Saturn's error categories as shares of all failures. Giving an incomplete answer in a way not captured elsewhere was 37.1% of failures, leaving out a required figure, limit or deadline was 18.4%, fabricated rules were 6.1%, and out-of-date regulatory guidance was 2.3%.⁴ Those four do not add up to all failures, so the coverage gives a partial list.

Financial Reporter describes concrete examples from the report. One pension tax mistake could have meant a £17,500 HMRC charge. In debt scenarios, models told people to pay the highest-interest debt first rather than priority bills such as rent. One model invented a rule that a graduate could pause student loan repayments by moving abroad. A Gemini model said a mortgage payment holiday would not affect a credit score. Saturn also says wrong answers arrived in the same fluent tone as right ones.

Read with care

Saturn sells software to advice firms and is calling on the UK regulator, the FCA, to regulate AI financial advice. Financial Reporter notes the research reflects Saturn's own methodology and scoring criteria and that the firm has a commercial interest in that debate. That does not make the numbers wrong. It means they are a vendor's benchmark, not an independent audit.

Other limits are worth keeping in view. We have not seen the 121 questions or the grading rubric, so we cannot check how a borderline answer was scored. The all-pass rule makes a missing deadline count the same as a wrong figure. Many questions use UK rules, from HMRC to council tax, so the numbers do not transfer directly to US tax or retirement accounts. And the models change fast: these are versions tested in 2026 as named by the coverage.

What to take from it

The durable lesson is about the shape of the errors. Omissions and invented rules are hard to spot because the answer reads well. That matches what we found in our look at which AI finance errors slip through and in the Evergreen survey on how people act on chatbot answers.

A fair way to use a chatbot on a money question is to treat the answer as a draft to be checked against the primary source. We plan to run a small, documented version of this test ourselves and publish the prompts, models and raw outputs.