arXiv preprint ·

Short Windows, Unmodelled Costs, Unknown Cutoffs: A Pre-Registered Evidence Audit of 91 LLM Trading Papers

Henry Zhang

Published
Source
arXiv preprint, posted before peer review
Original
Read it at arXiv
Topics
Computational FinanceGeneral FinanceTrading & Market Microstructure

Abstract

Papers on large-language-model (LLM) trading agents and alpha mining report Sharpe ratios that would be remarkable in any other part of empirical finance. We audit the evidence behind those numbers. Before coding a single paper we pre-registered a population of 91 papers from a curated corpus of LLM trading research, a ten-field coding rubric, an analysis script and five directional hypotheses, and committed all of them to version control. Every code is backed by a verbatim quote that a script checks against the paper's text, and a second, blind coder recoded a random 20%. Of the 75 papers with a trading test, 51% deduct no trading costs and only 9% model costs from data; the median test window is 12 months, and a quarter of windows are six months or shorter. Among the 51 papers whose test window is not known to lie after the model's training cutoff, 98% neither test after the cutoff, nor evaluate live, nor use a chronologically consistent model. Only 9% report a multiplicity-adjusted result, and just five papers combine deducted costs, a leakage control and a reported trial count. Two of the five hypotheses pass (short windows; uncontrolled look-ahead). Three fail: more papers report a trial count than we predicted, though most of those counts are repeated random seeds rather than the number of strategies searched; and the predicted links between headline Sharpe and test-window length, and between headline Sharpe and costs, are not detectable in the 13 papers that state a headline Sharpe in their abstract or conclusion. We release the codes, quotes and script, and propose a six-line reporting minimum.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

Read it on ReadSearch

Each opens after you create an account, and brings you back to this paper.

Already a reader? Sign in

All new papers in Computational Finance

Get the few papers worth your time in Computational Finance every morning.

ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.

Papers belong to their authors and publishers; every page here links to the original. All research topics