WallStreetDebug>_ Wall Street, debugged.
Markets
Subscribe
AI research

The backtest looked fine. Six bugs said otherwise: a field report from an AI-assisted trading stack

An intraday stock system, a parity harness, and a research log written with Claude's help. The authors report six defects that made the backtest drift from what the live account did, and the ten checks they now run. No profit claims, which is exactly why it is worth reading.

By WallStreetDebug DeskPublished 5 min readNot financial advice
6Defects that split one backtest from live trading
30%1Of the study period's days hit by the UTC clock bug
20 of 61Simulated fills sized past the live account's limit

Every AI trading pitch has a backtest. Almost none of them tell you whether the backtest does what the live system does. A short paper posted to arXiv on October 15, 2026, number 2610.21846, is a rare field report on exactly that gap. Its authors, independent researchers Alex Vidovich and Zivana Zerjal, ran one intraday US equity system live, replayed its live days in their backtest, and wrote down every place the two disagreed.

The AI angle is in the paper's own disclosure. The authors used Claude as a research assistant to help write the project's code and research log, run the logged analyses and draft the text. They say they chose the questions, made the keep-or-kill calls and verified the results. The paper does not blame the assistant for any defect. On the first one it says plainly: "The mistake was ours." Many retail quants now work this way, which is why the paper is worth a close read.

The setup

The system trades a fixed universe of 97 to 98 US stocks from one-minute bars and decides shortly after the open. It has run on a paper account and with real money. Its rules are not disclosed. Their main backtest, the parity harness, drives the same strategy code as the live runner and is checked against live fills.

The six defects

  • The default data feed. A historical request with no feed parameter returned single-exchange data instead of the consolidated tape. Opening-minute volume in the cache was 0.1% to 2% of the real figure for about 11 weeks.3
  • Dividend-adjusted history. A commit switched prices to fully adjusted. One bar sat 0.68% below the live price basis, and 37 of 98 symbols shifted the same way, every one a dividend payer.4 Live orders fill at unadjusted prices, so every backtest was off and live was fine.
  • Sizing past the limit. In the reference backtest, 20 of 61 simulated fills were sized at a notional the live account can never take, and dollar figures were about 18% too large.5 Only the dollars moved.
  • The clock. Research scans picked the opening bar with the string "13:30" in UTC, which is 9:30 New York time only in summer. The study period had 129 winter days, 30% of the sample, scored on a pre-market bar or silently dropped.1 Eleven scans were affected. No verdict flipped, but almost every number moved.
  • Missing data read as zero. The replay downloaded only the stocks it traded, not the extra data live uses. One input silently fell back to zero in every harness run for about four months. Restoring it changed one decision in 20 replayed real-money mornings.
  • "Enabled" is not "connected." An exit rule was switched on but ignored by live. A sizing rule ran only in the backtest. An audit found 53 settings read by production code but never declared, about a quarter of the configuration. One startup file was the only place the switch that turns order execution on existed.2

The pattern

All six sat where two paths that should agree had drifted apart: a historical request and a live one, research scans and the harness, the config file and the code that places orders. In five of the six, the path closer to live was right. None was found by reviewing summary results. The authors' line is the one to remember: a bug that makes results absurd gets caught; these made results look plausible.

Even matching decisions is not enough. A retired harness had matched the live decision on 7 of 8 real-money dates while misreading its inputs. On one real-money day, two errors cancelled and produced a false pass.

The checklist, condensed

The paper ends with ten checks, each traced to a defect. The gist:

  • Pin every vendor data parameter, and test the request, not just the response.
  • Simulate fills with unadjusted prices.
  • Re-apply the binding size limit after every step that can raise size.
  • Convert timestamps once, in one exchange-time helper, and test a winter day and a summer day.
  • Build the replay's data request from the live system's required inputs, and make missing inputs loud.
  • For every setting, test that it changes an action in the order path, and check live logs daily.

What to be careful about

This is one system, one data vendor and one account, reported by the team that made the errors. The log is not public, and some figures are rounded or withheld. The slippage bar is 5% of the replay result or a withheld dollar floor, so readers cannot judge how strict the real-money passes were.6 Every harness figure produced while the missing-data bug was live, including some in this paper, has not been re-run.

What this means for AI trading

AI coding assistants make it fast to build a backtest. They do not make it true. If a bot vendor, a course or your own AI-written script shows you a backtest, ask one question first: has it been replayed against real live fills, day by day? If the answer is no, the equity curve is a claim, not a record. Not advice. What should we test next?

What it shows / What it doesn't

It shows

  • Six concrete ways one intraday system's backtest drifted from live, each with its symptom, size, cause and the test that now guards it.2
  • That the drift did not show in summary statistics: a clock bug touched 129 winter days, 30% of a study period, without flipping a verdict.1
  • A ten-point checklist that applies to any backtest, including ones written with an AI coding assistant.

It doesn't show

  • Whether the system makes or loses money. The authors make no such claim.
  • That the AI assistant caused any defect. The authors take responsibility, and the paper does not attribute bugs to the tool.
  • How common these defects are in other systems. One system, one vendor, one account, self-reported.

Why it matters

A backtest is a claim about what a live system would have done. This paper shows how a careful team, with an AI assistant in the loop, still shipped six silent errors, and how they caught them: by replaying real live days, one fill at a time. Hold any AI trading backtest to that standard.

Data notes

Each small number in the text points to a note below: the sample size, the period, the source and, when the source publishes one, the margin of error. "Not disclosed" means the source does not give it.

  1. n=129 winter days in the study period; error shares from a 13-day winter sample · 2026 research scans · Vidovich and Zerjal, arXiv 2610.21846, Defect 4 · margin: not disclosed (small sample, self-reported)
  2. n=1 intraday US equity system, 97-98 stocks · research log through 2026, paper dated Sep 24, 2026 · Vidovich and Zerjal, arXiv 2610.21846, Section 2 and Table 5 · margin: not disclosed (self-reported; some figures rounded or withheld)
  3. n=98 cached 2026 trading days, 1 system · about 11 weeks in 2026 · Vidovich and Zerjal, arXiv 2610.21846, Defect 1 and Table 1 · margin: log entries disagree (0.1-0.5% vs 0.1-2%; authors use the wider range)
  4. n=98 symbols, one same-day scan · 2026 research log · Vidovich and Zerjal, arXiv 2610.21846, Defect 2 and Table 2 · margin: not disclosed (log examples range -0.68% to -1.65%)
  5. n=61 simulated fills, 1 reference backtest · 2026 research log · Vidovich and Zerjal, arXiv 2610.21846, Defect 3 · margin: not disclosed (log estimate)
  6. n=5 real-money days replayed (7 in the current harness record) · 2026 · Vidovich and Zerjal, arXiv 2610.21846, Sections 1 and 5 · margin: dollar floor of the slippage bar withheld by the authors

Sources

"Figures checked" means we opened the source and compared the figures we quote with it. It confirms what the source says, not that its claims are independently true. How we label sources

How we used AI: We read the full 8-page PDF of arXiv 2610.21846 (v1, submitted October 15, 2026) and its abstract page on October 18, 2026. All figures are the authors' own, from a research log that is not public.

Not financial advice. For information and education only. AI model outputs and research results can be wrong and do not predict future returns. Nothing here is a recommendation to buy or sell any asset. See our disclaimer and methodology.

BacktestingResearchTrading botsAI coding assistants

Suggest a tool or study

Know an AI tool, a study or a claim we should look at next? Email us at [email protected].

Related