From Knowing to Doing: A Memory-Controlled Benchmark for LLM Trading Agents on Stock Markets

• URL: http://arxiv.org/abs/2605.28359v1
• Date: Wed, 27 May 2026 11:57:10 GMT
• Title: From Knowing to Doing: A Memory-Controlled Benchmark for LLM Trading Agents on Stock Markets
• Abstract summary: KTD-Fin is an end-to-end stock-market trading benchmark.<n>It separates historical market memory from investment decision-making.<n>It incorporates a Barra-style performance attribution framework.
• Score: 34.699961175708815
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Evaluating whether large language model (LLM) agents can profit in capital markets is increasingly framed as end-to-end trading: place an agent in a historical market, let it trade, and measure portfolio returns. This setup is vulnerable to two evaluation failures. First, long backtests often overlap with the knowledge cutoffs of frontier LLMs, allowing memorized tickers, dates, prices, and market narratives to substitute for investment reasoning. Second, raw returns are a noisy proxy for stock-selection ability, since positive performance may come from market beta, style exposure, or favorable regimes rather than genuine alpha. We introduce KTD-Fin (Knowing-To-Doing Financial Benchmark), an end-to-end stock-market trading benchmark that addresses both issues. KTD-Fin uses a data-side masking protocol to anonymize key identifiers and calendar information consistently across prompts and tools, separating historical market memory from investment decision-making. It also incorporates a Barra-style performance attribution framework that decomposes portfolio returns into market, style, and stock-selection alpha components. Across ten frontier LLM agents evaluated on the Chinese CSI300 over a 2024--2026 window, masking substantially changes agent rationales, pushing them towards anonymized factor-based reasoning. Attribution analysis further shows that LLM agents' cumulative returns under leakage-controlled evaluation are largely explained by passive market and style exposure, with limited evidence of persistent stock-selection alpha. These findings suggest that financial LLM benchmarks should evaluate not only whether an agent makes money, but also whether the source of returns reflects transferable investment skill. We release KTD-Fin as a reproducible template for leakage-controlled and attribution-aware evaluation of LLM trading agents.

Related papers

• CN-Buzz2Portfolio: A Chinese-Market Dataset and Benchmark for LLM-Based Macro and Sector Asset Allocation from Daily Trending Financial News [21.498084305999992]
  Large Language Models (LLMs) are rapidly transitioning from static Natural Language Processing (NLP) tasks to acting as dynamic decision-making agents.<n>Direct live trading is irreproducible and prone to outcome bias by confounding luck with skill, whereas existing static benchmarks are often confined to entity-level stock picking and ignore broader market attention.<n>This work provides new insights into the alignment between general reasoning and financial decision-making, and all data, codes, and experiments are released to promote sustainable financial agent research.
  arXiv  Detail & Related papers  (2026-03-18T02:31:28Z)
• LiveTradeBench: Seeking Real-World Alpha with Large Language Models [26.976122048323873]
  Large language models (LLMs) achieve strong performance across benchmarks.<n>These tests occur in static settings, lacking real dynamics and uncertainty.<n>We introduce LiveTradeBench, a live trading environment for evaluating LLM agents in realistic and evolving markets.
  arXiv  Detail & Related papers  (2025-11-05T16:47:26Z)
• When Agents Trade: Live Multi-Market Trading Benchmark for LLM Agents [74.55061622246824]
  Agent Market Arena (AMA) is the first lifelong, real-time benchmark for evaluating Large Language Model (LLM)-based trading agents.<n>AMA integrates verified trading data, expert-checked news, and diverse agent architectures within a unified trading framework.<n>It evaluates agents across GPT-4o, GPT-4.1, Claude-3.5-haiku, Claude-sonnet-4, and Gemini-2.0-flash.
  arXiv  Detail & Related papers  (2025-10-13T17:54:09Z)
• StockBench: Can LLM Agents Trade Stocks Profitably In Real-world Markets? [44.10622904101254]
  Large language models (LLMs) have recently demonstrated strong capabilities as autonomous agents.<n>We introduce StockBench, a benchmark designed to evaluate LLM agents in realistic, multi-month stock trading environments.<n>Our evaluation shows that while most LLM agents struggle to outperform the simple buy-and-hold baseline, several models demonstrate the potential to deliver higher returns and manage risk more effectively.
  arXiv  Detail & Related papers  (2025-10-02T16:54:57Z)
• Your AI, Not Your View: The Bias of LLMs in Investment Analysis [62.388554963415906]
  In finance, Large Language Models (LLMs) face frequent knowledge conflicts arising from discrepancies between their pre-trained parametric knowledge and real-time market data.<n>These conflicts are especially problematic in real-world investment services, where a model's inherent biases can misalign with institutional objectives.<n>We propose an experimental framework to investigate emergent behaviors in such conflict scenarios, offering a quantitative analysis of bias in investment analysis.
  arXiv  Detail & Related papers  (2025-07-28T16:09:38Z)
• Agent Trading Arena: A Study on Numerical Understanding in LLM-Based Agents [69.58565132975504]
  Large language models (LLMs) have demonstrated remarkable capabilities in natural language tasks.<n>We present the Agent Trading Arena, a virtual zero-sum stock market in which LLM-based agents engage in competitive multi-agent trading.
  arXiv  Detail & Related papers  (2025-02-25T08:41:01Z)
• When AI Meets Finance (StockAgent): Large Language Model-based Stock Trading in Simulated Real-world Environments [55.19252983108372]
  We have developed a multi-agent AI system called StockAgent, driven by LLMs. The StockAgent allows users to evaluate the impact of different external factors on investor trading. It avoids the test set leakage issue present in existing trading simulation systems based on AI Agents.
  arXiv  Detail & Related papers  (2024-07-15T06:49:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.