7 AI models vs Wall Street on 456 earnings reports: the best scored 79.2%. A spreadsheet trick scored 75.8%.
On October 8, 2026, the AI research firm Samaya published a test of seven frontier models predicting Q2 2026 earnings for 456 large companies. The headline says AI beat human experts for the first time. The chart data behind the post shows how narrow that win is, and what it depends on.
Samaya AI, a company that sells AI research tools to banks and asset managers, published a blog post on October 8, 2026 titled 'Frontier AI models outperform human experts on earnings prediction for the first time'. It is a serious piece of work, with a point-in-time data gate, seven models, a stronger-than-usual baseline and its chart data published alongside the text. It is also a company blog post, not a peer-reviewed paper, and Samaya says it offers an alpha earnings prediction agent to its clients. We read the post, its appendix and the chart data behind it.
How the test worked
Seven models (GPT-6 Astra, Claude Fable 5.1, Claude Opus 5.5, GPT-5.6 Sol, Claude Sonnet 5, Gemini 3.8 Flash and Kimi K3) ran inside Samaya's research harness. Each one had to predict four numbers for a company's quarter: revenue, gross margin, operating income and adjusted EPS. The forecasts were made five trading sessions before each report, using only documents published up to that day. Web access was removed and replaced with news sources behind the same time gate.
The companies all reported from July 14, 2026 onward, after every model's knowledge cutoff, had a market value above $5 billion and more than 8 brokers publishing estimates. That left 456 companies across all sectors.
The baseline that matters
The usual benchmark is the analyst consensus, the median broker estimate. Samaya found it too easy to beat, because analysts tend to lower their estimates before results, so companies beat the consensus more often than not. So the authors built a harder baseline: take the consensus and add each company's median surprise over the past eight quarters, only when that surprise is positive. No AI involved. It is the kind of correction a careful analyst could do in a spreadsheet.
The results
On the share of forecasts that landed on the right side of the consensus, all four metrics pooled, GPT-6 Astra scored 79.2% and Claude Fable 5.1 scored 77.3%, while the spreadsheet-corrected consensus scored 75.8%3. Claude Sonnet 5 scored 63.0%4, below the trick. The chart data gives a one-sided p-value of 0.007 for Astra beating the corrected consensus and 0.124 for Fable, so only Astra's edge clears the usual bar for statistical significance.
On revenue, the average miss shrinks as models get stronger. Measured as a share of reported revenue, the raw consensus missed by 3.50 on average, the corrected consensus by 2.97, Fable 5.1 by 2.63 and Astra by 2.405. That is a real improvement. It is also an improvement of about half a point of revenue over a correction anyone can compute.
| Forecaster | Hit rate | Beats corrected consensus? |
|---|---|---|
| GPT-6 Astra | 79.2% | Yes, p=0.007 |
| Claude Fable 5.1 | 77.3% | Not significant, p=0.124 |
| Corrected consensus (no AI) | 75.8% | Baseline |
| Claude Sonnet 5 | 63.0% | No |
Take the data away and the edge disappears
The most useful part of the post is the ablation. With no documents at all, relying on memory only, Astra's surprise correlation, a measure of whether bigger predicted surprises match bigger actual ones, fell to 0.01, essentially zero. With data from 11 weeks before the report it reached 0.22, and with the latest data it reached 0.34. The model is not predicting earnings from what it knows. It is reading fresh documents fast and adjusting the consensus.
Two more details point the same way. Samaya added expert-written instructions to its harness, and with them the models researched 1.6 to 2.7 times more in time and context. And when Astra and Fable were each run five times on a 100-company cohort, results varied noticeably from run to run, while averaging the runs did not significantly improve them.
Our read
The headline is technically supported for one model on one quarter. The more practical finding is that the system matters as much as the model: point-in-time data, a retrieval stack and expert instructions. That echoes other tests we have covered, where the tools around an AI analyst moved scores as much as the model choice. It also echoes the 2024 'GPT-4 beats analysts' paper, which its own authors later withdrew, a reminder to wait for replication before treating any 'first time' claim as settled.
Finally, a forecast is not a trade. Samaya measured whether the direction and size of surprises were called correctly. It did not measure returns, costs or what the market had already priced in.
What it shows / What it doesn't
It shows
- In one quarter of US large-cap results, GPT-6 Astra inside Samaya's harness called the direction of surprises more often than a bias-corrected consensus, 79.2% vs 75.8%7.
- Stronger models made smaller revenue errors than both the raw and the corrected consensus.
- Fresh documents drive the result: with no data, the best model's surprise correlation was about zero.
It doesn't show
- It does not show any trading profit: no returns, costs or positions were measured.
- It covers one quarter and one vendor's harness, without independent replication or peer review.
- It does not show that a model alone beats analysts: the harness, the data gate and expert instructions are part of the result.
Why it matters
Claims that AI now beats Wall Street analysts are spreading fast. This test is better built than most, and it still shows a thin edge over a simple, free correction, earned by a system that feeds the model fresh data rather than by the model alone. Read the baseline before the headline.
Data notes
Sample size · period · source · margin of error for each figure marked with a number.
- n=456 companies, 4 metrics each · Q2 2026 reports from Jul 14 · Samaya AI chart data · one-sided bootstrap p=0.007 vs corrected consensus
- n=456 companies, 4 metrics each · Q2 2026 reports from Jul 14 · Samaya AI chart data · margin: not disclosed
- n=456 companies, 4 metrics each · Q2 2026 reports from Jul 14 · Samaya AI chart data (hit_center_data) · one-sided paired company-cluster bootstrap
- n=456 companies, 4 metrics each · Q2 2026 reports from Jul 14 · Samaya AI chart data (hit_center_data) · margin: not disclosed
- n=456 companies · Q2 2026 reports from Jul 14 · Samaya AI chart data (hero_data), error as % of revenue · margin: not disclosed
- n=456 companies, 4 metrics each · Q2 2026 reports from Jul 14 · Samaya AI chart data (hit_center_data) · p-values from one-sided paired company-cluster bootstrap
- n=456 companies, 4 metrics each · Q2 2026 reports from Jul 14 · Samaya AI chart data · one-sided bootstrap p=0.007
Primary sources
- VerifiedFrontier AI models outperform human experts on earnings prediction for the first time (Samaya AI research blog, Oct 8, 2026)Method, models, universe, ablations and appendix read Oct 17, 2026.
- VerifiedChart data published with the post (Samaya AI)Hit rates, p-values, revenue errors and ablation figures read from the embedded data Oct 17, 2026.
- LinkedThe use of management forecasts to dampen analysts' expectations (Baik and Jiang (2006), Journal of Accounting and Public Policy 25(5))Cited by Samaya for analysts lowering estimates before results.
Your turn: what should we test next?
Questions about this test or study, or an AI tool you want us to put through the same checks? Reply to our post on X @WallStreetDebug or email [email protected]. The best ideas become next week's tests.
Keep reading
In our directories
How we used AI: AI tools helped research and draft this article. Every figure was then checked against the primary sources listed above before publication.
Not financial advice. For information and education only. AI model outputs and research results can be wrong and do not predict future returns. Nothing here is a recommendation to buy or sell any asset. See our disclaimer and methodology.