A survey of 47 AI trading systems finds the same thing every time someone reruns them: the edge shrinks

Every few weeks a new LLM trading agent shows up on arXiv with a chart that climbs and a Sharpe ratio that would get a portfolio manager promoted. A survey posted to SSRN on October 15 asked the question those charts rarely invite: what happens when somebody else presses the button?¹

The answer, in the survey's own words: "the margins of independently rerun agents shrink or vanish."² Not every margin, and not for every system, because the authors are careful to add that the reruns "cover few systems." But the direction is the one that matters. When the people who built the agent stop being the people who grade it, the agent gets worse.

Who wrote it, and what it covers

The paper, "From Algorithms to Agents: A Survey of LLM-Based Trading Systems," runs 56 pages and comes from eight authors at Nanyang Technological University, the University of Hong Kong, University College London, the University of Waterloo and Cornell, among them the fintech economist Lin William Cong.¹ It sorts the literature two ways: by capability (perception, reasoning, action, memory, collaboration) and by market, six of them, from equities to prediction markets. Then it maps 47 systems onto that grid "to show where evidence is thin," audits how they were evaluated, and pulls together the head-to-head comparisons against traditional quant baselines.¹

It is a survey, not an experiment. Its authors did not build a new agent, and we did not run one either. What it offers is rarer and more useful: an inventory of what the field has actually shown, written by people inside the field.

The eight-month problem

The most quietly devastating line in the abstract is a statistics footnote promoted to a finding. Many reported margins, the authors write, "rest on windows of eight months or less, where an annualized Sharpe ratio has a standard error above one."²

Translate that. A standard error above one means a strategy reporting a Sharpe of 1.5 over such a window could plausibly have a true Sharpe of zero. Using the standard textbook approximation from Andrew Lo, our own arithmetic puts the standard error at about 1.22 for a strategy with no edge at all over eight months, which means pure luck alone produces a reading above 2 roughly one time in twenty.³ A Sharpe ratio of 2 on eight months of data is a coin with a nice haircut. It may be a good coin. Eight months cannot tell you.

This is not a flaw peculiar to AI. Quants have known for decades that short backtests flatter. What is peculiar is how many LLM agent papers still publish on windows that short, presumably in part because the models are new and their training cutoffs leave only a sliver of clean, out-of-sample history to test on.

Where the LLM seems to earn its keep

The survey does not conclude that language models are useless for trading. Its best-supported hypothesis puts the LLM "upstream of the trade": turning text into signals, writing factor and model code, and making "bounded execution tilts."⁴ In other words, the model as research assistant and code generator, feeding a disciplined process, rather than the model as autonomous portfolio manager.

Even here the authors flag a catch: "most upstream evidence comes from single author-run studies."⁴ The most plausible use case is also, so far, mostly graded by its own inventors.

The committee that doesn't outvote the chairman

Multi-agent architectures, where a "bull" agent argues with a "bear" agent while a "risk manager" agent referees, are among the field's favorite designs. They demo beautifully. The survey's third finding is that "controlled comparisons do not establish a multi-agent advantage."⁵ Risk controls do help "where ablated," meaning where someone bothered to switch them off and measure the difference.⁵ The useful part of the committee may be the compliance officer, not the debate.

The fourth finding should worry anyone shopping for a single best agent: structure and input design matter alongside the underlying model, and "the best structure depends on the backbone."⁵ Swap the model under a winning architecture and the architecture may stop winning. A leaderboard of agent designs is, on this reading, a leaderboard of pairings that may not survive the next model release.

What to ask the next agent that claims to beat the market

The survey's value for readers is that it turns into a checklist. How long is the test window, and does the reported Sharpe come with an uncertainty band? Has anyone outside the team rerun it? Is there a plain quant baseline in the same table? If it is a multi-agent system, did the authors compare it against a single agent under the same conditions? And if the margin depends on one specific model, what happens on the next one?

One caveat cuts against our own reading. We worked from the survey's abstract and SSRN listing; the full PDF did not open for us, so the counts behind "few systems" and "many reported margins" are the authors' characterizations, not figures we checked. The survey is also a preprint and has not been peer reviewed. Its conclusions are hedged by its own authors, and they deserve that hedge here too.

Still, the direction is clear, and it is not the one in the press releases. The evidence that LLMs help at the research end of trading is promising and mostly self-reported. The evidence that autonomous agents beat the market is thin, short and tends to shrink under someone else's hands. Who answers for the loss when an investor trusts an eight-month Sharpe ratio? Not the chart.