From Comprehension to Reasoning: A Hierarchical Benchmark for Automated Financial Research Reporting

• URL: http://arxiv.org/abs/2603.19254v1
• Date: Wed, 25 Feb 2026 13:44:58 GMT
• Title: From Comprehension to Reasoning: A Hierarchical Benchmark for Automated Financial Research Reporting
• Abstract summary: FinReasoning is a benchmark that decomposes Chinese research-report generation into three stages aligned with real analyst.<n>Based on the evaluation results, FinReasoning reveals that most models exhibit a understanding-execution gap.
• Score: 19.0993436440595
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) are increasingly used to generate financial research reports, shifting from auxiliary analytic tools to primary content producers. Yet recent real-world deployments reveal persistent failures--factual errors, numerical inconsistencies, fabricated references, and shallow analysis--that can distort assessments of corporate fundamentals and ultimately trigger severe economic losses. However, existing financial benchmarks focus on comprehension over completed reports rather than evaluating whether a model can produce reliable analysis. Moreover, current evaluation frameworks merely flag hallucinations and lack structured measures for deeper analytical skills, leaving key analytical bottlenecks undiscovered. To address these gaps, we introduce FinReasoning, a benchmark that decomposes Chinese research-report generation into three stages aligned with real analyst workflows, assessing semantic consistency, data alignment, and deep insight. We further propose a fine-grained evaluation framework that strengthens hallucination-correction assessment and incorporates a 12-indicator rubric for core analytical skills. Based on the evaluation results, FinReasoning reveals that most models exhibit a understanding-execution gap: they can identify errors but struggle to generate accurate corrections; they can retrieve data but have difficulty returning it in correct format. Furthermore, no model achieves overwhelming superiority across all three tracks; Doubao-Seed-1.8, GPT-5, and Kimi-K2 rank as the top three in overall performance, yet each exhibits a distinct capability distribution. The evaluation resource is available at https://github.com/TongjiFinLab/FinReasoning.

Related papers

• Holistic Evaluation and Failure Diagnosis of AI Agents [0.2922148761619506]
  We present a holistic agent evaluation framework that pairs top-down diagnosis with bottom-up span-level evaluation.<n>This decomposition scales to traces of arbitrary length and produces span-level rationales for each verdict.<n>On the TRAIL benchmark, our framework achieves state-of-the-art results across all metrics.
  arXiv  Detail & Related papers  (2026-05-14T14:12:39Z)
• Evian: Towards Explainable Visual Instruction-tuning Data Auditing [14.93566912726999]
  We introduce a novel "Decomposition-then-Evaluation" paradigm that breaks model responses into constituent cognitive components.<n>We instantiate this paradigm via EVIAN, an automated framework that evaluates these components along the axes of Image-Text Consistency, Logical Coherence, and Factual Accuracy.
  arXiv  Detail & Related papers  (2026-04-22T13:28:27Z)
• Fin-RATE: A Real-world Financial Analytics and Tracking Evaluation Benchmark for LLMs on SEC Filings [31.87459935714186]
  Fin-RATE is a benchmark built on U.S. Securities and Exchange Commission (SEC) filings and mirroring financial analyst.<n>We benchmark 17 leading Large Language Models (LLMs) across open-source, closed-source, and finance-specialized models.<n>Results show substantial performance degradation, with accuracy dropping by 18.60% and 14.35% as tasks shift from single-document reasoning to longitudinal and cross-entity analysis.
  arXiv  Detail & Related papers  (2026-02-07T00:54:37Z)
• Understanding Structured Financial Data with LLMs: A Case Study on Fraud Detection [17.04809129025246]
  FinFRE-RAG is a two-stage approach that applies importance-guided feature reduction to serialize a compact subset of numeric/categorical attributes into natural language.<n>LLMs can produce human-readable explanations and facilitate feature analysis, potentially reducing the manual workload of fraud analysts.
  arXiv  Detail & Related papers  (2025-12-15T07:09:11Z)
• FinLFQA: Evaluating Attributed Text Generation of LLMs in Financial Long-Form Question Answering [57.43420753842626]
  FinLFQA is a benchmark designed to evaluate the ability of Large Language Models to generate long-form answers to complex financial questions.<n>We provide an automatic evaluation framework covering both answer quality and attribution quality.
  arXiv  Detail & Related papers  (2025-10-07T20:06:15Z)
• Curse of Knowledge: When Complex Evaluation Context Benefits yet Biases LLM Judges [72.3356133063925]
  The paradigm of large language models (LLMs) as judges has emerged as a scalable solution, yet prior work primarily focuses on simple settings.<n>Our in-depth analysis offers crucial insights for improving the accuracy and verifiability of evaluation signals.
  arXiv  Detail & Related papers  (2025-09-03T15:48:33Z)
• Assessing Consistency and Reproducibility in the Outputs of Large Language Models: Evidence Across Diverse Finance and Accounting Tasks [0.0]
  We generate over 3.4 million outputs from diverse financial source texts and data.<n>More advanced models do not consistently demonstrate better consistency and maintain high agreement.<n>Simple aggregation strategies across 3-5 runs dramatically improve consistency.
  arXiv  Detail & Related papers  (2025-03-21T09:43:37Z)
• Causality can systematically address the monsters under the bench(marks) [64.36592889550431]
  Benchmarks are plagued by various biases, artifacts, or leakage.<n>Models may behave unreliably due to poorly explored failure modes.<n> causality offers an ideal framework to systematically address these challenges.
  arXiv  Detail & Related papers  (2025-02-07T17:01:37Z)
• Self-Evolving Critique Abilities in Large Language Models [59.861013614500024]
  This paper explores enhancing critique abilities of Large Language Models (LLMs)<n>We introduce SCRIT, a framework that trains LLMs with self-generated data to evolve their critique abilities.<n>Our analysis reveals that SCRIT's performance scales positively with data and model size.
  arXiv  Detail & Related papers  (2025-01-10T05:51:52Z)
• A Comprehensive Evaluation and Analysis Study for Chinese Spelling Check [53.152011258252315]
  We show that using phonetic and graphic information reasonably is effective for Chinese Spelling Check. Models are sensitive to the error distribution of the test set, which reflects the shortcomings of models. The commonly used benchmark, SIGHAN, can not reliably evaluate models' performance.
  arXiv  Detail & Related papers  (2023-07-25T17:02:38Z)
• Preserving Knowledge Invariance: Rethinking Robustness Evaluation of Open Information Extraction [49.15931834209624]
  We present the first benchmark that simulates the evaluation of open information extraction models in the real world.<n>We design and annotate a large-scale testbed in which each example is a knowledge-invariant clique.<n>By further elaborating the robustness metric, a model is judged to be robust if its performance is consistently accurate on the overall cliques.
  arXiv  Detail & Related papers  (2023-05-23T12:05:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.