ShiJianBench: From Dialogue to Decision for Long-Horizon Evaluation of Investment Advisors

• URL: http://arxiv.org/abs/2608.01204v1
• Date: Sun, 02 Aug 2026 12:41:47 GMT
• Title: ShiJianBench: From Dialogue to Decision for Long-Horizon Evaluation of Investment Advisors
• Abstract summary: We introduce ShiJianBench, an offline framework for evaluating conversational investment advisors through matched investor trajectories.<n>At its core is a multi-agent investor simulator with explicit evolving state variables, motive-driven deliberation, long-term memory, and dialogue-grounded updates.<n>Experiments on Chinese fund-market traces from 2021 to 2026 identify a stable leading group of LLM advisors that combines substantially stronger personalized content with competitive investor-side trajectory outcomes.
• Score: 44.51855262774783
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Conversational investment advisors influence not only what users know, but also how they make subsequent decisions as market conditions evolve. Existing evaluations primarily assess response quality or observed outcomes, leaving the long-horizon pathway from advisor language to investor behavior difficult to audit. We introduce ShiJianBench, an offline framework for evaluating conversational investment advisors through matched investor trajectories under fixed historical market feedback. At its core is a multi-agent investor simulator with explicit evolving state variables, motive-driven deliberation, long-term memory, and dialogue-grounded updates. The simulator is calibrated against aggregate behavioral patterns from 7,199 real users, and advisor policies are evaluated using separate investor-side, service-side, and content-side metrics under a hard compliance gate. Experiments on Chinese fund-market traces from 2021 to 2026 identify a stable leading group of LLM advisors that combines substantially stronger personalized content with competitive investor-side trajectory outcomes. These results reveal a systematic distinction between producing a high-quality response and delivering an effective long-horizon intervention, motivating trajectory-aware evaluation of conversational advisors.

Related papers

• Regime-Aware Portfolio Management via Retrieval-Augmented LLM-Guided Expert Switching [3.0795668932789515]
  This work proposes a retrieval-augmented expert-switching framework.<n>It selects experts based on their historical performance under similar market situations.<n>Experiments across cryptocurrency, stock, and foreign-exchange markets show that the proposed selector achieves the highest cumulative return and Sharpe ratio.
  arXiv  Detail & Related papers  (2026-08-28T12:15:54Z)
• Your AI, On a Dial: Controlling Investment Bias in LLMs with a Single Neuron [76.4407829130164]
  We study whether a model's overall investment stance can be calibrated to a specified direction and strength.<n>We introduce an investment-bias dial, an inference-time intervention on a single neuron.<n>We find that the dial produces monotonic changes in investment stance without modifying prompts or model parameters.
  arXiv  Detail & Related papers  (2026-08-24T06:33:02Z)
• Fund2Persona: A Framework for Building and Refining Financial Advisor Personas from Fund Disclosure Data [92.72860874355746]
  We propose Fund2Persona, a framework that grounds financial-advisor personas in fund disclosures, holdings transitions, market context, and manager commentary.<n>We evaluate the resulting personas on held-out holdings-transition reconstruction and manager-commentary alignment.<n>These results suggest that fund-data-grounded financial-advisor personas can make manager-specific investment expertise portable.
  arXiv  Detail & Related papers  (2026-06-29T05:14:43Z)
• From Knowing to Doing: A Memory-Controlled Benchmark for LLM Trading Agents on Stock Markets [34.699961175708815]
  KTD-Fin is an end-to-end stock-market trading benchmark.<n>It separates historical market memory from investment decision-making.<n>It incorporates a Barra-style performance attribution framework.
  arXiv  Detail & Related papers  (2026-05-27T11:57:10Z)
• DIALECTIC: A Multi-Agent System for Startup Evaluation [2.5358402189416704]
  We introduce DIALECTIC, an LLM-based multi-agent system for startup evaluation.<n>It first gathers factual knowledge about a startup and organizes these facts into a hierarchical question tree.<n>It then synthesizes the facts into natural-language arguments for and against an investment and iteratively critiques and refines these arguments through a simulated debate.<n>Our system also produces numeric decision scores that allow investors to rank and thus efficiently prioritize opportunities.
  arXiv  Detail & Related papers  (2026-02-19T11:40:36Z)
• Conv-FinRe: A Conversational and Longitudinal Benchmark for Utility-Grounded Financial Recommendation [33.09732020656996]
  We introduce Conv-FinRe, a benchmark for stock recommendation that evaluates LLMs beyond behavior matching.<n>We build the benchmark from real market data and human decision trajectories, instantiate controlled advisory conversations, and evaluate a suite of state-of-the-art LLMs.<n>Results reveal a persistent tension between rational decision quality and behavioral alignment.
  arXiv  Detail & Related papers  (2026-02-19T01:29:50Z)
• LLM Agents as VC investors: Predicting Startup Success via RolePlay-Based Collective Simulation [13.180519641845398]
  SimVC-CAS is a novel collective agent system that simulates venture capital decision-making as a multi-agent interaction process.<n>We show that SimVC-CAS significantly improves predictive accuracy while providing interpretable, multiperspective reasoning.
  arXiv  Detail & Related papers  (2025-12-27T14:34:44Z)
• Pathways of Thoughts: Multi-Directional Thinking for Long-form Personalized Question Answering [57.12316804290369]
  Personalization is essential for adapting question answering systems to user-specific information needs.<n>We propose Pathways of Thoughts (PoT), an inference-stage method that applies to any large language model (LLM) without requiring task-specific fine-tuning.<n>PoT consistently outperforms competitive baselines, achieving up to a 13.1% relative improvement.
  arXiv  Detail & Related papers  (2025-09-23T14:44:46Z)
• LeCoDe: A Benchmark Dataset for Interactive Legal Consultation Dialogue Evaluation [68.26911563519162]
  Legal consultation is essential for safeguarding individual rights and ensuring access to justice.<n>Current systems fall short in handling the interactive and knowledge-intensive nature of real-world consultations.<n>LeCoDe is a real-world multi-turn benchmark dataset comprising 3,696 legal consultation dialogues with 110,008 dialogue turns.
  arXiv  Detail & Related papers  (2025-05-26T08:24:32Z)
• When AI Meets Finance (StockAgent): Large Language Model-based Stock Trading in Simulated Real-world Environments [55.19252983108372]
  We have developed a multi-agent AI system called StockAgent, driven by LLMs. The StockAgent allows users to evaluate the impact of different external factors on investor trading. It avoids the test set leakage issue present in existing trading simulation systems based on AI Agents.
  arXiv  Detail & Related papers  (2024-07-15T06:49:30Z)
• Learning Risk Preferences from Investment Portfolios Using Inverse Optimization [25.19470942583387]
  This paper presents a novel approach of measuring risk preference from existing portfolios using inverse optimization. We demonstrate our methods on real market data that consists of 20 years of asset pricing and 10 years of mutual fund portfolio holdings.
  arXiv  Detail & Related papers  (2020-10-04T21:29:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.