Same AI, different stopwatch: how earnings-call benchmarks can be graded on the wrong clock

Many AI finance benchmarks ask a model to read an earnings call and predict what the stock does next. A paper accepted to ICAIF 2026, the ACM AI in Finance conference in Milan in November, argues that the question has a hidden clock. If the return window starts before the call ends, the model is being graded on price moves that happened while the call was still going. We read the paper, arXiv 2610.21659, in full on its HTML version.

The problem in one example

A model that reads the complete question-and-answer section cannot act until the call is over. Yet some benchmarks pair complete-call inputs with returns that start at or before the call began. The authors say that can be fine for studying how markets react to a call. It is a different claim from "the model predicted the move after reading everything." They propose an Availability Contract: a benchmark should state when its input is complete, when the outcome window starts, and which claim that supports.

What they measured

The team built 552 earnings calls from 24 large US firms, 2018 to 2024, kept from a parent panel of 2,094 calls after requiring usable timestamps, audio and price data. For each call they held the end point fixed and compared start points: the previous close, a documentary call-start time, and a completion proxy equal to reconstructed audio length plus a stipulated 15-minute allowance.

Moving from the call-start clock to the completion clock flipped the sign of the return for 9.4% of calls.¹ Holding the completion clock's cutpoints fixed, 28.3% of calls landed in a different return quintile. Re-sorting each clock on its own gave 30.1%. Moving from the previous close to call start flipped 32.1% of signs, which shows how much a conventional event window already shifts the target.

For that same step, the average absolute change in return was 0.623 percentage points, but the median was only 0.179, so the effect is concentrated in a minority of calls. For the 206 after-close calls, both proxies map to the same next-session price bar, so nothing changed. That works as a built-in check on the method.

The leaderboard that did not move

The second test used a public benchmark of earnings-call volatility prediction, Dense Earnings Call, with four frozen prediction streams taken from the benchmark's original paper. The authors did not retrain anything. They shifted the targets of 1,191 before-market events forward by one trading session and re-scored the same predictions on 1,710 events.

Three-day mean squared error rose by 18.1% to 21.4% across the four streams.² The effect faded with longer horizons, to between 8.1% and 10.1% at seven days and between 3.8% and 6.2% at fifteen days.³ Yet the ranking of the four streams did not change under mean squared error, mean absolute error or rank correlation.

So the scoreboard looked stable while the numbers underneath moved. The authors add a caution: when they resampled by issuer, the best model at three days stayed best in 67.1% of draws, so near-ties are fragile too.⁴

Where the paper is careful, and where we should be

The completion time is a proxy. The authors did not observe when a transcript or a model was actually ready. Their 15-minute allowance is a stipulation, and they test allowances from zero to 120 minutes: the sign-flip rate then ranges from 6.5% to 15.0%.⁵ The 9.4% sits inside that range.

The sample is also selective. Only 26.4% of the parent panel survived the data requirements⁶, the price join removed 117 calls from five firms, and 2018 has just two calls. The Dense Earnings Call test is a counterfactual shift of targets, not a re-run of those models on a corrected benchmark. And the exploratory regressions with a small Qwen model, FinBERT and other text scores are secondary and labeled exploratory by the authors. The paper notes that Claude and Codex assisted with literature search and code review, while the reported estimates come from the authors' scripts.

What to take away

If you evaluate an AI that reads earnings calls, ask three questions of any result. When does the input end? When does the return window start? What claim does that timing support? A result that cannot answer them may be measuring the reaction to a call, not a forecast made after it. This is a measurement paper. It makes no claim that any model makes or loses money.