Four frontier AIs forecast all 104 World Cup matches. None beat the betting market

Most tests of AI forecasting have a hidden flaw: the questions are about the past, and the model may have read the answer during training. A preprint posted to arXiv on July 20, 2026 avoids that problem in a simple way. It used an event that had not happened yet when the models were trained: the 2026 FIFA World Cup.

The result is useful well beyond soccer. It is one of the cleanest published comparisons of frontier AI forecasters against a real market price, and it separates two things traders often blur: predicting an outcome and making money on it.

The setup

The authors, Jiacheng Ding and Cong Guo (University of Memphis) and Jason Xu (QuantaInsight), ran four models through the same search, act and reflect loop for each of the 104 matches: Claude Opus 4.8, ChatGPT (GPT-5.5 with high reasoning), Gemini 3.1 Pro and Grok in Expert Mode.

• Before each match, the agent searched the web, gave a probability for a win, a draw or a loss, and placed a virtual $100 bet.
• After the match, it saw only the final score and wrote a reflection on what it got wrong.
• A fifth competitor was the pre-match betting market itself, taken as published 1X2 odds.

The released dataset holds 416 forecasts and 414 reflections, with the verbatim reasoning, the odds and the code.

The scoreboard

The Brier score measures how close probability forecasts are to what happened. Lower is better. The market had the best Brier score of the five forecasters.

The four agents chose the same most likely outcome in 92% of the 104 matches (91.7% in the group stage and 93.8% in the knockout rounds; no interval disclosed). Their accuracies sit inside a band of about three points. Read on predictions alone, the authors note, this is an efficient-market result: the AIs did not know anything the odds did not already price. nContext: n=104 matches per forecaster · 2026 FIFA World Cup · Ding, Guo and Xu, arXiv:2607.17765 · margin: not disclosed

Where the four AIs split: money

Each agent chose its own picks and stakes, and every bet was settled at the real odds on the 90-minute result. Here the agents behaved very differently.

The most telling column is not in the table above. The paper splits each agent's bets into those that agreed with the market favorite and those that went against it. Betting against the market lost money for Claude (-24% on contrarian bets, which were 58% of its 73 bets), ChatGPT (-14%, 36% of 55 bets) and Gemini (-12%, 14% of 103 bets); intervals not disclosed. The table shows Grok's contrarian bets at +33%, but they were only 5% of its 104 bets, about five wagers, and the abstract still states that fading the market was unprofitable for all four agents. We flag that inconsistency as published; with so few bets, the Grok figure carries little weight either way. nContext: n=73, 55, 103 and 104 bets (Claude, ChatGPT, Gemini, Grok) · 2026 World Cup · arXiv:2607.17765, Table 3 · margin: not disclosed

In other words, the agent with the best betting record was the one that mostly bet with the favorite.

Do the AIs know when they were wrong?

The reflection step is a test of self-knowledge. On the matches where an agent's top pick was wrong, the self-reported error rate, meaning how often the agent's own reflection called its pick an error, ranged from 36% to 86% depending on the model (computed on each agent's wrong picks over the 104 matches; no interval disclosed). The share of forecasts that even mentioned the market odds ranged from 12% to 100%. nContext: n=each agent's wrong picks among 104 matches · 2026 World Cup · arXiv:2607.17765 · margin: not disclosed

That spread matters for anyone using an AI agent that "learns from its mistakes." Some models admitted errors most of the time. Others mostly did not.

The caveats the authors list

The paper is candid about its limits. It is a single tournament of 104 matches, and the authors stress patterns over precise rankings. The agents were queried through consumer apps with tools turned on, so retrieval was not fully controlled, and providers can update models mid-tournament. The odds are opening lines, mostly from one bookmaker, not the closing consensus, which is usually the harder price to beat. And the betting was virtual.

The authors also address misuse directly: treating any agent's edge as a betting strategy is "unsupported by our data, which show no agent beats the market."