The performance decay of LLM trading strategies

A couple of months ago, I wrote about an experiment where researchers from the San Francisco Fed asked ChatGPT to forecast inflation. The results were miserable. Once the researchers tried to forecast inflation outside of the training window of ChatGPT, the model broke down, indicating that even if you ringfence your model, the training data used for the model in the first place leaks through and gives you forecasts that look much better in a backtest than when applied to live markets.

A team of Chinese researchers now used state-of-the-art AI trading agents to assess if the same problem exists when developing trading strategies. They asked five LLM-based methods and restricted them to run on GTP-4o so that the training cutoff was known to be October 2023. Then they asked these five trading LLMs to trade in the constituents of the Nasdaq 100 for the period Q2 and Q3 2021 (i.e. a backtest that is within the training window of GPT-4o) and Q3 and Q4 of 2024 (a test after the training window closed).

Both periods had very similar returns for the Nasdaq 100 of about 13.5%, but the total return of the trading LLMs changed dramatically. In the backtest setting, the models achieved total returns between 30% and 44%, comfortably beating the index. But if the same models were asked to trade in late 2024 when training data could no longer help even if it inadvertently leaked into the task, the total return dropped to between 9% and 22%.

Trading agents applied to pre-training and post-training periods

The researchers then propose a way to salvage the problem, which asks the trading LLMs to come up with their own trading strategy but not only test it in a backtest, but apply it to counterfactual scenarios. Essentially, as far as I can see, this boils down to good old Monte Carlo simulation (with a few extra bells and whistles) to test a trading strategy in an artificial environment that has not been part of the training data for the LLM. If it works in the backtest and in the counterfactual environment, the trading strategy has a better chance of being profitable in real-life application. And according to their results (which are restricted to five stocks and Bitcoin, so not the full Nasdaq 100), that helps a lot.