Time Travel is Cheating: Going Live with DeepFund for Real-Time Fund Investment Benchmarking

• URL: http://arxiv.org/abs/2505.11065v1
• Date: Fri, 16 May 2025 10:00:56 GMT
• Title: Time Travel is Cheating: Going Live with DeepFund for Real-Time Fund Investment Benchmarking
• Authors: Changlun Li, Yao Shi, Chen Wang, Qiqi Duan, Runke Ruan, Weijie Huang, Haonan Long, Lijun Huang, Yuyu Luo, Nan Tang,
• Abstract summary: Large Language Models (LLMs) have demonstrated notable capabilities across financial tasks.<n>Their real-world effectiveness in managing complex fund investment remains inadequately assessed.<n>We introduce DeepFund, a live fund benchmark tool designed to rigorously evaluate LLM in real-time market conditions.
• Score: 12.837781884216227
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Large Language Models (LLMs) have demonstrated notable capabilities across financial tasks, including financial report summarization, earnings call transcript analysis, and asset classification. However, their real-world effectiveness in managing complex fund investment remains inadequately assessed. A fundamental limitation of existing benchmarks for evaluating LLM-driven trading strategies is their reliance on historical back-testing, inadvertently enabling LLMs to "time travel"-leveraging future information embedded in their training corpora, thus resulting in possible information leakage and overly optimistic performance estimates. To address this issue, we introduce DeepFund, a live fund benchmark tool designed to rigorously evaluate LLM in real-time market conditions. Utilizing a multi-agent architecture, DeepFund connects directly with real-time stock market data-specifically data published after each model pretraining cutoff-to ensure fair and leakage-free evaluations. Empirical tests on nine flagship LLMs from leading global institutions across multiple investment dimensions-including ticker-level analysis, investment decision-making, portfolio management, and risk control-reveal significant practical challenges. Notably, even cutting-edge models such as DeepSeek-V3 and Claude-3.7-Sonnet incur net trading losses within DeepFund real-time evaluation environment, underscoring the present limitations of LLMs for active fund management. Our code is available at https://github.com/HKUSTDial/DeepFund.

Related papers

• Your AI, Not Your View: The Bias of LLMs in Investment Analysis [55.328782443604986]
  Large Language Models (LLMs) face frequent knowledge conflicts due to discrepancies between pre-trained parametric knowledge and real-time market data.<n>This paper offers the first quantitative analysis of confirmation bias in LLM-based investment analysis.<n>We observe a consistent preference for large-cap stocks and contrarian strategies across most models.
  arXiv  Detail & Related papers  (2025-07-28T16:09:38Z)
• EDINET-Bench: Evaluating LLMs on Complex Financial Tasks using Japanese Financial Statements [7.259647868714988]
  We introduce EDINET-Bench, an open-source Japanese financial benchmark to evaluate the performance of large language models (LLMs)<n>Our experiments reveal that even state-of-the-art LLMs struggle, performing only slightly better than logistic regression in binary classification for fraud detection and earnings forecasting.<n>Our dataset, benchmark construction code, and evaluation code is publicly available to facilitate future research in finance with LLMs.
  arXiv  Detail & Related papers  (2025-06-10T13:03:36Z)
• Your Offline Policy is Not Trustworthy: Bilevel Reinforcement Learning for Sequential Portfolio Optimization [82.03139922490796]
  Reinforcement learning (RL) has shown significant promise for sequential portfolio optimization tasks, such as stock trading, where the objective is to maximize cumulative returns while minimizing risks using historical data.<n>Traditional RL approaches often produce policies that merely memorize the optimal yet impractical buying and selling behaviors within the fixed dataset.<n>Our approach frames portfolio optimization as a new type of partial-offline RL problem and makes two technical contributions.
  arXiv  Detail & Related papers  (2025-05-19T06:37:25Z)
• Can LLM-based Financial Investing Strategies Outperform the Market in Long Run? [5.968528974532717]
  Large Language Models (LLMs) have been leveraged for asset pricing tasks and stock trading applications, enabling AI agents to generate investment decisions from unstructured financial data.<n>We critically assess their generalizability and robustness by proposing FINSABER, a backtesting framework evaluating timing-based strategies across longer periods and a larger universe of symbols.
  arXiv  Detail & Related papers  (2025-05-11T18:02:21Z)
• DeepFund: Will LLM be Professional at Fund Investment? A Live Arena Perspective [10.932591941137698]
  This paper introduces DeepFund, a comprehensive platform for evaluating Large Language Models (LLMs) in a simulated live environment.<n>Our approach implements a multi agent framework where LLMs serve as both analysts and managers, creating a realistic simulation of investment decision making.<n>We provide a web interface that visualizes model performance across different market conditions and investment parameters, enabling detailed comparative analysis.
  arXiv  Detail & Related papers  (2025-03-24T03:32:13Z)
• FinTSB: A Comprehensive and Practical Benchmark for Financial Time Series Forecasting [58.70072722290475]
  Financial time series (FinTS) record the behavior of human-brain-augmented decision-making.<n>FinTSB is a comprehensive and practical benchmark for financial time series forecasting.
  arXiv  Detail & Related papers  (2025-02-26T05:19:16Z)
• FinanceQA: A Benchmark for Evaluating Financial Analysis Capabilities of Large Language Models [0.0]
  FinanceQA is a testing suite that evaluates LLMs' performance on complex numerical financial analysis tasks that mirror real-world investment work.<n>Current LLMs fail to meet the strict accuracy requirements of financial institutions, with models failing approximately 60% of realistic tasks.<n>Results show that higher-quality training data is needed to support such tasks, which we experiment with using OpenAI's fine-tuning API.
  arXiv  Detail & Related papers  (2025-01-30T00:06:55Z)
• Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation [55.21013307734612]
  AoPS-Instruct is a dataset of more than 600,000 high-quality QA pairs.<n>LiveAoPSBench is an evolving evaluation set with timestamps, derived from the latest forum data.<n>Our work presents a scalable approach to creating and maintaining large-scale, high-quality datasets for advanced math reasoning.
  arXiv  Detail & Related papers  (2025-01-24T06:39:38Z)
• LLM-Powered Multi-Agent System for Automated Crypto Portfolio Management [9.9661459222949]
  We propose an explainable, multi-modal, multi-agent framework for cryptocurrency investment.<n>Our framework uses specialized agents that collaborate within and across teams to handle subtasks such as data analysis, literature integration, and investment decision-making.
  arXiv  Detail & Related papers  (2025-01-01T13:08:17Z)
• AI in Investment Analysis: LLMs for Equity Stock Ratings [0.2916558661202724]
  This paper explores the application of Large Language Models (LLMs) to generate multi-horizon stock ratings. Our study addresses these issues by leveraging LLMs to improve the accuracy and consistency of stock ratings. Our results show that our benchmark method outperforms traditional stock rating methods when assessed by forward returns.
  arXiv  Detail & Related papers  (2024-10-30T15:06:57Z)
• When AI Meets Finance (StockAgent): Large Language Model-based Stock Trading in Simulated Real-world Environments [55.19252983108372]
  We have developed a multi-agent AI system called StockAgent, driven by LLMs. The StockAgent allows users to evaluate the impact of different external factors on investor trading. It avoids the test set leakage issue present in existing trading simulation systems based on AI Agents.
  arXiv  Detail & Related papers  (2024-07-15T06:49:30Z)
• AlphaFin: Benchmarking Financial Analysis with Retrieval-Augmented Stock-Chain Framework [48.3060010653088]
  We release AlphaFin datasets, combining traditional research datasets, real-time financial data, and handwritten chain-of-thought (CoT) data. We then use AlphaFin datasets to benchmark a state-of-the-art method, called Stock-Chain, for effectively tackling the financial analysis task.
  arXiv  Detail & Related papers  (2024-03-19T09:45:33Z)
• FinBen: A Holistic Financial Benchmark for Large Language Models [75.09474986283394]
  FinBen is the first extensive open-source evaluation benchmark, including 36 datasets spanning 24 financial tasks. FinBen offers several key innovations: a broader range of tasks and datasets, the first evaluation of stock trading, novel agent and Retrieval-Augmented Generation (RAG) evaluation, and three novel open-source evaluation datasets for text summarization, question answering, and stock trading.
  arXiv  Detail & Related papers  (2024-02-20T02:16:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.