Beyond Agent Architecture: Execution Assumptions and Reproducibility in LLM-Based Trading Systems

• URL: http://arxiv.org/abs/2606.08285v1
• Date: Sat, 06 Jun 2026 18:14:29 GMT
• Title: Beyond Agent Architecture: Execution Assumptions and Reproducibility in LLM-Based Trading Systems
• Abstract summary: This article presents a targeted topical review and audit of execution realism in large language models (LLMs) trading research.<n>A coded evidence matrix covering 30 trade-relevant primary studies is used to assess point-in-time controls, split transparency, held-out evaluation, cost and turnover treatment, execution semantics, universe definition, and artifact release.
• Score: 5.264406906731695
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Large language models (LLMs) and agentic systems are increasingly proposed for financial trading, yet their reported performance remains difficult to compare because studies vary in data provenance, temporal split discipline, execution timing, turnover treatment, and transaction-cost modeling. This article presents a targeted topical review and reproducibility audit of execution realism in LLM-based trading research. A coded evidence matrix covering 30 trade-relevant primary studies is used to assess point-in-time controls, split transparency, held-out evaluation, cost and turnover treatment, execution semantics, universe definition, and artifact release. Across the audited sample, architecture reporting is generally clearer than the evaluation assumptions needed to judge whether a trading result is economically interpretable or reproducible. A 10-equity worked example is included only as a methodological scaffold to illustrate how explicit friction and timing choices can materially compress active-strategy results. The main conclusion is that the next useful step for LLM trading research is not only better agent design, but also clearer reporting standards for execution realism, reproducibility, and evaluation comparability.

Related papers

• Can Agentic Trading Systems Pay for Their Own Intelligence? [42.600136171993874]
  We introduce TradeLens, a trace-grounded diagnostic toolkit for evaluating agentic trading systems.<n>We conduct extensive analysis across backbone models, capital scales, trading frequencies, and system architectures.<n>Our results show that viability hinges on intelligence-to-profit conversion.
  arXiv  Detail & Related papers  (2026-07-11T12:31:54Z)
• Teaching Language Models to Check Grounded Claim Factuality with Human Test-Taking Strategies [0.9867902290081937]
  Grounded claim factuality checking is important for large language model (LLM) applications such as retrieval-augmented generation.<n>We address this by formulating grounded claim factuality checking as a true/false reading comprehension task.<n>Our method reduces token usage by over 80% compared to unguided open-ended reasoning.
  arXiv  Detail & Related papers  (2026-05-28T10:11:42Z)
• Agentic Trading: When LLM Agents Meet Financial Markets [10.275260358522809]
  This paper reframes Large Language Models (LLMs) as expert-system decision pipelines.<n>A primary empirical subset satisfies the minimum boundary of Action Output plus Closed-Loop Evaluation.<n>The central empirical finding is protocol incomparability.
  arXiv  Detail & Related papers  (2026-05-19T04:20:07Z)
• Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents? [61.49434544687523]
  We introduce REFLECT, a meta-evaluation benchmark targeting fine-grained failure detection in agentic environments.<n> REFLECT defines a detailed taxonomy of process- and outcome-level failure modes, instantiated by performing controlled and localized interventions.<n>Our experiments show that even the best-performing models achieve overall accuracies below 55% across reasoning, tool-use, and report-quality failures.
  arXiv  Detail & Related papers  (2026-05-18T23:55:08Z)
• Toward Expert Investment Teams:A Multi-Agent LLM System with Fine-Grained Trading Tasks [6.55184070677326]
  We propose a multi-agent trading framework that decomposes investment analysis into fine-grained tasks.<n> Experimental results show that fine-grained task decomposition significantly improves risk-adjusted returns.<n>We exploit low correlation with the stock index and the variance of each system's output to achieve superior performance.
  arXiv  Detail & Related papers  (2026-02-26T18:37:36Z)
• Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation [76.5533899503582]
  Large language models (LLMs) are increasingly used as judges to evaluate agent performance.<n>We show this paradigm implicitly assumes that the agent's chain-of-thought (CoT) reasoning faithfully reflects both its internal reasoning and the underlying environment state.<n>We demonstrate that manipulated reasoning alone can inflate false positive rates of state-of-the-art VLM judges by up to 90% across 800 trajectories spanning diverse web tasks.
  arXiv  Detail & Related papers  (2026-01-21T06:07:43Z)
• StockBench: Can LLM Agents Trade Stocks Profitably In Real-world Markets? [44.10622904101254]
  Large language models (LLMs) have recently demonstrated strong capabilities as autonomous agents.<n>We introduce StockBench, a benchmark designed to evaluate LLM agents in realistic, multi-month stock trading environments.<n>Our evaluation shows that while most LLM agents struggle to outperform the simple buy-and-hold baseline, several models demonstrate the potential to deliver higher returns and manage risk more effectively.
  arXiv  Detail & Related papers  (2025-10-02T16:54:57Z)
• FinAgentBench: A Benchmark Dataset for Agentic Retrieval in Financial Question Answering [57.18367828883773]
  FinAgentBench is a benchmark for evaluating agentic retrieval with multi-step reasoning in finance.<n>The benchmark consists of 26K expert-annotated examples on S&P-500 listed firms.<n>We evaluate a suite of state-of-the-art models and demonstrate how targeted fine-tuning can significantly improve agentic retrieval performance.
  arXiv  Detail & Related papers  (2025-08-07T22:15:22Z)
• Source Attribution in Retrieval-Augmented Generation [3.579940498399598]
  This paper investigates the feasibility and effectiveness of adapting Shapley-based attribution to identify influential retrieved documents in RAG.<n>Our work aims to: (1) systematically apply established attribution principles to the RAG document-level setting; (2) quantify how well SHAP approximations can mirror exact attributions; and (3) evaluate their practical explainability in identifying critical documents.
  arXiv  Detail & Related papers  (2025-07-06T17:36:45Z)
• How do Large Language Models Understand Relevance? A Mechanistic Interpretability Perspective [64.00022624183781]
  Large language models (LLMs) can assess relevance and support information retrieval (IR) tasks.<n>We investigate how different LLM modules contribute to relevance judgment through the lens of mechanistic interpretability.
  arXiv  Detail & Related papers  (2025-04-10T16:14:55Z)
• Agent Trading Arena: A Study on Numerical Understanding in LLM-Based Agents [69.58565132975504]
  Large language models (LLMs) have demonstrated remarkable capabilities in natural language tasks.<n>We present the Agent Trading Arena, a virtual zero-sum stock market in which LLM-based agents engage in competitive multi-agent trading.
  arXiv  Detail & Related papers  (2025-02-25T08:41:01Z)
• FACT-AUDIT: An Adaptive Multi-Agent Framework for Dynamic Fact-Checking Evaluation of Large Language Models [79.41859481668618]
  Large Language Models (LLMs) have significantly advanced the fact-checking studies.<n>Existing automated fact-checking evaluation methods rely on static datasets and classification metrics.<n>We introduce FACT-AUDIT, an agent-driven framework that adaptively and dynamically assesses LLMs' fact-checking capabilities.
  arXiv  Detail & Related papers  (2025-02-25T07:44:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.