BizFinBench v2: on real investor questions, GPT-5 scores 61.5 against 84.8 for human experts
BizFinBench v2 was built from questions real users asked a financial platform about Chinese and U.S. stocks. Accepted at ICML 2026, it ranks 21 AI models against human experts, and adds a live trading test whose results raise as many questions as they answer.
Most finance benchmarks are written by researchers. BizFinBench v2 takes a different route: its questions come from authentic user queries on a financial platform, covering Chinese and U.S. equity markets. The team is from HiThink Research and the Shanghai University of Finance and Economics. The paper was first posted to arXiv in January 2026, revised on July 13, 2026, and accepted at ICML 2026, one of the main machine learning conferences.
What is in the test
The benchmark has 28,860 questions across eight offline tasks and two live tasks, grouped into four scenarios:
- Business information provenance: tracing why a stock moved, multi-turn conversations, describing financial data.
- Financial logic reasoning: quantitative calculations, event logic and counterfactual questions.
- Stakeholder perception: reading user sentiment and analyzing financial reports (Chinese market only).
- Real-time market tasks: predicting daily closing prices for 100 popular Chinese and U.S. stocks between November 5 and December 24, 2025, and a live portfolio allocation test.
Scoring is rule-based, with no AI judge, which the authors present as fully reproducible. For the price prediction task, the model must give a range and is marked correct only if the actual close falls inside a 1% tolerance. Context: n: not applicable (scoring rule) · price window Nov 5 to Dec 24, 2025 · arXiv:2601.06401 · margin: not applicable
The scoreboard
| Rank | Model | Average | Stock price prediction |
|---|---|---|---|
| - | Human financial experts | 84.8 | 32.3 |
| 1 | GPT-5 | 61.5 | 32.1 |
| 2 | Gemini-3 | 61.3 | 34.9 |
| 3 | Doubao-Seed-1.6 | 59.4 | 31.1 |
| 4 | Qwen3-235B-A22B-Thinking | 53.3 | 36.9 |
| 5 | Kimi-k2 | 41.8 | 30.5 |
| 6 | DeepSeek-R1 | 41.5 | 32.3 |
| 7 | Qwen3-32B | 41.3 | 5.1 |
| 8 | Claude-Sonnet-4 | 39.5 | 29.1 |
| 9 | Grok-4 | 37.2 | 30.3 |
| - | Best finance-specialized model (Dianjin-R1) | 35.7 | 9.9 |
The authors frame the gap bluntly: GPT-5's 61.5% accuracy still fails to meet the practical business requirement, which they set at the experts' 84.8% (accuracy averaged over nine tasks; no interval disclosed). Context: n=28,860 questions (604 to 4,497 per task) · paper v2, Jul 13, 2026 · Guo et al., arXiv:2601.06401, Table 2 · margin: not disclosed
What stands out
Everyone struggles to predict prices, including the experts. On the stock price prediction task, human experts scored 32.3, and the best model, Qwen3-235B, scored 36.9. Within a 1% band, next-day closes are close to a coin toss for humans and machines alike. That is the most honest number in the table. Context: n=4,049 price questions on 100 stocks · Nov 5 to Dec 24, 2025 · arXiv:2601.06401, Table 2 · margin: not disclosed
Finance-specialized models trail general ones. The best open-source financial model, Dianjin-R1, averaged 35.7, below several general models. Fin-R1 scored 11.5. It is the same lesson as BloombergGPT versus GPT-4: domain branding is not a substitute for general reasoning.
Step-by-step reasoning did not help. Under chain-of-thought prompting, averages fell for the top models: GPT-5 dropped to 54.6 and Claude-Sonnet-4 to 13.7. The price prediction scores collapsed in that setting, to 6.0 for GPT-5. The paper reports this without a full explanation.
Some models fail specific tasks badly. Grok-4 scored 9.3 on quantitative computation and 4.7 on counterfactual inference, against 89.2 and 83.9 for GPT-5. A single average hides those holes.
The live trading test: read with care
Six commercial models managed simulated funds from November 19, 2025 to March 30, 2026, making one decision per hour during trading days, with fees, latency and slippage modeled.
| Strategy | Total return | Sharpe | Max drawdown |
|---|---|---|---|
| DeepSeek-R1 | +47.34% | 1.25 | -53% |
| Claude-Sonnet-4 | -2.11% | -0.05 | -11% |
| Qwen3-Max | -3.79% | -0.10 | -13% |
| Gemini-3 | -6.61% | -0.20 | -13% |
| Grok-4 | -10.15% | -0.32 | -15% |
| GPT-5 | -13.80% | -0.45 | -16% |
| Equal weight, rebalanced quarterly | +32.98% | 2.19 | -9.20% |
| SPY (ending assets 10,632) | about +6.3% | - | - |
The authors conclude that DeepSeek-R1 showed superior investment efficacy. The table says something more cautious: DeepSeek-R1's gain came with a 53% drawdown, five of six models lost money, and a simple equal-weight portfolio of the same universe beat every model on risk-adjusted terms. And the project's own GitHub page describes DeepSeek-R1's online result differently, as a 13.46% total return with an 8% maximum drawdown, presumably from a different window. We could not reconcile the two figures from the published materials. Context: n=1 run per model · Nov 19, 2025 to Mar 30, 2026 · Guo et al., arXiv:2601.06401, Table 4; 13.46% from the project README · margin: not disclosed
Limits
The questions come from one platform, HiThink's, and skew toward Chinese-market use cases. The expert baseline is small: the paper describes two outside financial experts invited to take part. And the live test is one run per model over about four months.
What it shows / What it doesn't
It shows
- How 21 AI models perform on financial questions drawn from real user queries, against a human expert baseline.
- That the best general models sit around 61 out of 100, well below experts at 84.8, and that finance-specialized models trail general ones.
- That short-horizon price prediction is hard for everyone: experts and models alike score near or below 37.
It doesn't show
- That any model trades well. In the live test, five of six models lost money, and a simple equal-weight portfolio beat them all on risk-adjusted terms.
- A robust expert baseline. Two outside experts took part, per the paper.
- Results for U.S.-only users. Some tasks, such as report analysis, cover the Chinese market only.
- A consistent live trading record. The paper and the project page report different DeepSeek-R1 results.
Why it matters
BizFinBench v2 is closer than most benchmarks to what investors actually ask an AI. The headline is that frontier models answer about six in ten such questions correctly where experts manage about eight and a half. The live test adds a familiar warning: one model's big return came with a drawdown most traders could not sit through, and a plain equal-weight basket did better per unit of risk.
Primary sources
- VerifiedBizFinBench.v2: Towards Reliable LLMs in Finance via Real-User Data and Offline/Online Bilingual Evaluation (arXiv:2601.06401, v2) (arXiv / HiThink Research, Shanghai University of Finance and Economics)Primary source. v2 posted July 13, 2026. Tables 2, 3 and 4, task sizes, windows and expert setup checked in the HTML version on October 14, 2026.
- VerifiedBizFinBench.v2 repository (README) (GitHub (HiThink-Research))States ICML 2026 acceptance (May 1, 2026) and a DeepSeek-R1 online result of +13.46% with -8% drawdown that differs from the paper's Table 4; checked October 14, 2026.
In our directories
How we used AI: AI tools helped research and draft this article. Every figure was then checked against the primary sources listed above before publication.
Not financial advice. For information and education only. AI model outputs and research results can be wrong and do not predict future returns. Nothing here is a recommendation to buy or sell any asset. See our disclaimer and methodology.