FrontierFinance: the tools around an AI analyst matter as much as the model

Benchmarks that test AI on finance usually ask for a number pulled from a filing. FrontierFinance, posted to arXiv on August 12, 2026 by researchers at Samaya AI, asks for something closer to a real analyst's deliverable: structured research reports across six stages of the investor workflow, from data extraction to screening for new ideas and sector and macro analysis.

How it is built

The benchmark has 220 queries written and audited by finance professionals, including former buy-side analysts, sell-side research associates and investment bankers. Each query comes with a long list of yes-or-no grading criteria, 11,543 in total, and each criterion is tied to a public data source. SEC filings are the largest source category at 39% of all rubrics (share of the 11,543 rubrics; no interval applies). nContext: n=11,543 rubrics · FrontierFinance public set · Samaya AI, arXiv:2608.11683 · margin: not applicable

A sample query: provide a breakdown of BP's upstream, downstream, and integrated gas and renewables segments' operating metrics over the past six years, as of October 9, 2025.

Answers are scored by majority vote of three AI judges (GPT 5.4, Gemini 3.1 Pro and Claude Sonnet 4.6). The headline metric is the share of rubrics an answer satisfies, averaged across queries.

The scoreboard

Systems were tested in three setups: models with their own built-in web search, models inside the open-source harness from the Vals AI Finance Agent v2 benchmark (with finance tools such as SEC search), and Samaya's in-house system.

Read this first: who made the test

The top two rows belong to Samaya AI, the company that built and published the benchmark. The paper says so openly and describes its system as included to show how a highly optimized finance harness performs. That does not make the result wrong, and the dataset and grading code are public so others can check it. But a vendor leading its own benchmark is a reason to wait for independent runs before treating the top line as settled.

The real finding: same model, different tools

The most useful comparison does not involve Samaya at all. Look at the same models in two setups:

• GPT 5.5 scored 20.7% with its built-in web search and 43.5% inside the Finance Agent v2 harness, more than double. nContext: n=220 queries, 11,543 rubrics · paper posted Aug 12, 2026 · Samaya AI, arXiv:2608.11683, Table 4 · margin: not disclosed
• Claude Opus 4.8 went from 33.0% to 45.0%. nContext: n=220 queries, 11,543 rubrics · paper posted Aug 12, 2026 · Samaya AI, arXiv:2608.11683, Table 4 · margin: not disclosed
• Gemini 3.1 Pro stayed nearly flat, from 30.7% to 30.5%. nContext: n=220 queries, 11,543 rubrics · paper posted Aug 12, 2026 · Samaya AI, arXiv:2608.11683, Table 4 · margin: not disclosed

The authors' conclusion is that the tool harness, not the model alone, strongly shapes quality and cost. For anyone choosing an AI research tool, that means a model's name on a product page says less than how the product feeds it data.

Open models are close, and cheaper

Kimi K3, an open-weight model, scored 46.4%, 0.4 points behind GPT 5.6 Sol and 2.8 points behind Claude Fable 5, at $0.90 per query against $3.03 and $4.06. GLM 5.2 delivered 42.8% at $0.63. The authors report that open models reached parity with older proprietary models within about two months of release. Paying more buys something, but not much: Claude Fable 5 scored a relative 9% higher than Claude Opus 4.8 at 56% more cost. nContext: n=220 queries, 11,543 rubrics · paper posted Aug 12, 2026 · Samaya AI, arXiv:2608.11683, Table 4 · margin: not disclosed

Where every system fails

Two use cases remain largely unsolved: Screening and Discovery, where the best system reached 33%, and Sector, Industry and Macro, where the best reached 39% (per-use-case query counts not restated in the abstract; no interval disclosed). These are the open-ended tasks, such as finding companies that fit a theme, closest to how traders actually use AI for idea generation. nContext: n: per-use-case query counts not stated (220 total) · paper Aug 12, 2026 · arXiv:2608.11683 · margin: not disclosed

The authors also found a failure pattern worth knowing. Models often tried to visit web addresses from memory instead of searching for them, and those remembered URLs had much higher error rates, because some were hallucinated or blocked. That wasted time and polluted the model's context with errors.

Limits the authors state

Each query is anchored to a date. As future models are trained on data after those dates, they may answer from memory rather than by researching, which would undermine what the benchmark tests. And screening tasks are subjective: two analysts can give different but equally valid answers, and a rubric may favor one framing.