Same earnings call, seven AI models, different scores: why your AI sentiment signal depends on the vendor
AI tools that score earnings calls for sentiment, uncertainty or management clarity are everywhere. A September 2026 study asked a simple question: if you swap the AI model, do you get the same score? Often, you do not.
Many AI trading tools turn text into numbers. Feed in an earnings call transcript and get back a sentiment score, an uncertainty score, a management clarity grade. Those numbers then drive screens, alerts and backtests. The hidden assumption is that the score describes the transcript. A paper posted to arXiv on September 25, 2026 shows that it also describes the model.
The experiment
Hamid Boustanifar (EDHEC Business School) and Sasan Mansouri (University of Groningen) took the question-and-answer sections of 1,946 earnings calls held by 501 S&P 500 companies in 2024, sourced from S&P Capital IQ. Management's answers averaged about 3,900 words per call.
As of November 2025, they picked one cost-efficient model from each of seven families: OpenAI's GPT, Anthropic's Claude, Google's Gemini, Meta's Llama, Mistral, DeepSeek and Alibaba's Qwen. Each model received identical definitions, scales and instructions, and returned a score, a confidence level and a written justification.
They measured 13 things, covering 12 constructs: sentiment, uncertainty, management clarity, climate risk, political risk, answer specificity, corporate culture, innovation, integrity and others, plus a rule-based specificity ratio where the model simply counts specific words.
The models disagree, a lot
Across the 13 measures, the average rank correlation between pairs of models was 0.52. For a measurement tool, that is low: two models scoring the same transcripts often ranked them quite differently. Only 34% of the total variation in scores came from differences between transcripts that all providers saw in common (1,946 calls; no interval disclosed). The rest was model-specific. Context: n=1,946 calls by 501 S&P 500 firms, 7 models · calls held in 2024 · Boustanifar and Mansouri, arXiv:2609.31013 · margin: not disclosed
Agreement depended heavily on what was being measured.
| Measure | Agreement between models |
|---|---|
| Sentiment | 0.85 |
| Innovation | 0.76 |
| Average across 13 measures | 0.52 |
| Integrity | 0.32 |
| Specificity ratio (counting task) | 0.23 |
The last row is the most striking. The specificity ratio is a precisely defined counting task, and the authors expected it to produce the most agreement. It produced the least. Score levels also differed widely: the average spread across providers for the same call was 0.75 times the overall standard deviation of scores.
Is the disagreement telling us something?
One charitable reading: maybe models disagree on calls that are genuinely ambiguous, and that ambiguity would show up in the market too. The authors tested this against analyst forecast dispersion, bid-ask spreads, trading volume and return variance after the call. Cross-model disagreement was not significantly associated with any of them. In their words, the disagreement is consistent with a substantial model-specific component rather than ambiguity in the disclosure.
Does it change conclusions?
This is where it matters for anyone running a backtest. The authors regressed the stock's abnormal return over the call day and the next day on each model's version of each measure, holding everything else fixed.
- For sentiment and uncertainty, the direction and significance of the relationship held across all seven models, but the magnitude varied by up to 60%. Context: n=1,946 calls by 501 S&P 500 firms, 7 models · calls held in 2024 · Boustanifar and Mansouri, arXiv:2609.31013 · margin: not disclosed
- For other measures, the choice of model could flip the sign of the relationship or change whether it was statistically significant at all.
Same transcripts, same regression, different conclusion, depending on which AI did the scoring.
Does agreeing with each other mean being right?
Not necessarily. The authors also compared the models' average scores with established dictionary-based measures from the academic literature, such as the Loughran and McDonald sentiment word lists. Correlations were positive but low: about 0.55 at best for corporate culture, innovation and specificity, and only about 0.3 for political risk and sentiment. So sentiment, the measure on which the models agree most, is also one where they diverge most from the traditional measure. The authors stress that dictionaries are alternative definitions, not ground truth.
Partial fixes
Averaging across all seven providers helps rankings. Using a demanding reliability threshold of 0.80, the seven-model average was reliable for ranking calls on 10 of the 13 measures, but stable in absolute levels on only 5. Newer frontier models narrowed the disagreement without closing it. The authors' recommendation is to treat AI-generated scores as model-contingent measurements and validate them across providers.
It echoes a related finding in our lookahead bias explainer: an AI signal can reflect properties of the model as much as properties of the market.
What it shows / What it doesn't
It shows
- That seven AI models scoring the same earnings calls with identical instructions produce substantially different rankings and levels.
- That model choice can change the size, sign or significance of the link between a text score and stock returns.
- That averaging across providers stabilizes rankings for most measures but not absolute score levels.
It doesn't show
- Which model is right. There is no ground truth, and agreement among models is not accuracy.
- That AI sentiment scores are useless. Sentiment showed the highest cross-model agreement, 0.85.
- How frontier models compare in full. The main tests used cost-efficient models chosen in November 2025, and newer models narrowed but did not close the gap.
- Results outside 2024 S&P 500 earnings calls.
Why it matters
If a tool sells you an AI sentiment or management-confidence score, that score is partly a property of the model behind it. Switching models, or a vendor quietly upgrading its model, can change your rankings and the backtest built on them. Sentiment is relatively robust; softer concepts such as integrity or specificity are not. Before relying on an AI text score, check whether a second model would tell the same story.
Primary sources
- VerifiedSame Text, Different Numbers: The Divergence of LLM-Based Measures (arXiv:2609.31013) (arXiv / EDHEC Business School, University of Groningen)Primary source. Preprint posted September 25, 2026. Sample, model list, correlations, return regressions, dictionary comparison and reliability results checked in the HTML version on October 14, 2026.
In our directories
How we used AI: AI tools helped research and draft this article. Every figure was then checked against the primary sources listed above before publication.
Not financial advice. For information and education only. AI model outputs and research results can be wrong and do not predict future returns. Nothing here is a recommendation to buy or sell any asset. See our disclaimer and methodology.