FinRiskAtlas: Decision-Aligned Evaluation of Large Language Models for Financial Risk Review

• URL: http://arxiv.org/abs/2608.25325v1
• Date: Wed, 26 Aug 2026 03:13:15 GMT
• Title: FinRiskAtlas: Decision-Aligned Evaluation of Large Language Models for Financial Risk Review
• Abstract summary: We introduce FinRiskAtlas, a Chinese-language benchmark that evaluates financial LLMs along two complementary dimensions.<n>We extend this framework through offline replay of 680 pre-action states from 104 de-identified professional trajectories.<n>FinRisk-Ask shows that entering the Ask branch more frequently does not necessarily improve request targeting or end-to-end evidence acquisition.
• Score: 25.300288794507377
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deploying large language models for professional financial review requires more than measuring general financial competence: models must perform the specific review operation required by a workflow and determine whether available evidence is sufficient for a defensible decision. Existing financial benchmarks cover knowledge, reasoning, compliance, and professional tasks, but their evaluation units are often organized around datasets or task formulations rather than the decisions that deployed systems support. We introduce FinRiskAtlas, a Chinese-language benchmark that evaluates financial LLMs along two complementary dimensions: operation execution under fixed evidence states and evidence-state control under evolving review conditions. The static benchmark contains 9,742 instances across 53 task families, including 42 Domain Knowledge families and eleven downstream review operations defined by explicit evaluation contracts. FinRisk-Ask extends this framework through offline replay of 680 pre-action states from 104 de-identified professional trajectories, withholding future evidence during inference and using it only to construct expert-verified evidence targets. Across 33 model configurations, operation-level evaluation yields non-redundant rankings (mean pairwise Spearman correlation 0.42 across downstream operations), and knowledge-based shortlisting can incur up to 18.01 points of regret on individual operations. FinRisk-Ask further shows that entering the Ask branch more frequently does not necessarily improve request targeting or end-to-end evidence acquisition. These results show that broad financial capability scores do not fully capture where models are reliable in professional workflows, motivating evaluation units aligned with the decisions and evidence states that deployed systems must support.

Related papers

• BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agents [5.690389797295356]
  BigFinanceBench is a 928-item benchmark of open-ended rubric financial-research tasks.<n>It decomposes the derivation into independently checkable steps.<n>It supports partial-credit evaluation and localization of failures across the analyst workflow.
  arXiv  Detail & Related papers  (2026-06-02T16:12:34Z)
• Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents? [61.49434544687523]
  We introduce REFLECT, a meta-evaluation benchmark targeting fine-grained failure detection in agentic environments.<n> REFLECT defines a detailed taxonomy of process- and outcome-level failure modes, instantiated by performing controlled and localized interventions.<n>Our experiments show that even the best-performing models achieve overall accuracies below 55% across reasoning, tool-use, and report-quality failures.
  arXiv  Detail & Related papers  (2026-05-18T23:55:08Z)
• FINESSE-Bench: A Hierarchical Benchmark Suite for Financial Domain Knowledge and Technical Analysis in Large Language Models [0.33504916505452914]
  FINESSE-Bench is a suite of eight specialized benchmarks comprising 3,993 questions for hierarchical evaluation of financial competencies in large language models.<n>We describe a unified evaluation protocol covering multiple-choice questions, numerical answers, and short open-ended responses.
  arXiv  Detail & Related papers  (2026-05-14T23:53:51Z)
• Herculean: An Agentic Benchmark for Financial Intelligence [114.33879935277217]
  We introduce Herculean, the first skilled benchmark for agentic financial intelligence spanning four representative skill sets.<n>Across frontier agents, we find agents perform relatively well on Trading and Market Insights, but struggle substantially on Hedging and Auditing.<n>Overall, our results point to a key gap in current agents in turning financial reasoning into dependable workflow execution.
  arXiv  Detail & Related papers  (2026-05-14T04:30:49Z)
• SAHM: A Benchmark for Arabic Financial and Shari'ah-Compliant Reasoning [59.20983102634086]
  We introduce SAHM, a document-grounded benchmark and instruction-tuning dataset for Arabic financial NLP and Shari'ah-compliant reasoning.<n> SAHM contains 14,380 expert-verified instances spanning seven tasks: AAOIFI standards QA, fatwa-based QA/MCQ, accounting and business exams, financial sentiment analysis, extractive summarization, and event-cause reasoning.<n>We find that Arabic fluency does not reliably translate to evidence-grounded financial reasoning.
  arXiv  Detail & Related papers  (2026-04-21T05:24:08Z)
• Fact4ac at the Financial Misinformation Detection Challenge Task: Reference-Free Financial Misinformation Detection via Fine-Tuning and Few-Shot Prompting of Large Language Models [4.8239913268889]
  This paper presents our winning methodology for the "Reference-Free Financial Misinformation Detection" shared task.<n>We propose a comprehensive framework that capitalizes on the reasoning capabilities of state-of-the-art Large Language Models.<n>Our proposed system demonstrated superior efficacy, successfully securing the first-place ranking on both official leaderboards.
  arXiv  Detail & Related papers  (2026-04-16T05:38:58Z)
• Evaluating LLMs in Finance Requires Explicit Bias Consideration [88.38155218924999]
  Finance-specific biases can inflate performance, contaminate backtests, and make reported results useless for deployment claims.<n>No single bias is discussed in more than 28 percent of studies.<n>We propose a Structural Validity Framework and an evaluation checklist with minimal requirements for bias diagnosis and future system design.
  arXiv  Detail & Related papers  (2026-02-15T17:02:01Z)
• PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning [18.32501228579171]
  Professional Reasoning Bench (PRBench) is a realistic, open-ended, and difficult benchmark of real-world problems in Finance and Law.<n>We open-source its 1,100 expert-authored tasks and 19,356 expert-curated criteria, making it the largest public, rubric-based benchmark for both legal and finance domains.
  arXiv  Detail & Related papers  (2025-11-14T18:55:12Z)
• Evaluating Large Language Models for Financial Reasoning: A CFA-Based Benchmark Study [1.6770212301915661]
  This study presents the first comprehensive evaluation of state-of-the-art LLMs using 1,560 multiple-choice questions from official mock exams across Levels I-III of CFA.<n>We compare models distinguished by core design priorities: multi-modal and computationally powerful, reasoning-specialized and highly accurate, and lightweight efficiency-optimized.
  arXiv  Detail & Related papers  (2025-08-29T06:13:21Z)
• FinAgentBench: A Benchmark Dataset for Agentic Retrieval in Financial Question Answering [57.18367828883773]
  FinAgentBench is a benchmark for evaluating agentic retrieval with multi-step reasoning in finance.<n>The benchmark consists of 26K expert-annotated examples on S&P-500 listed firms.<n>We evaluate a suite of state-of-the-art models and demonstrate how targeted fine-tuning can significantly improve agentic retrieval performance.
  arXiv  Detail & Related papers  (2025-08-07T22:15:22Z)
• FinDER: Financial Dataset for Question Answering and Evaluating Retrieval-Augmented Generation [65.04104723843264]
  We present FinDER, an expert-generated dataset tailored for Retrieval-Augmented Generation (RAG) in finance.<n>FinDER focuses on annotating search-relevant evidence by domain experts, offering 5,703 query-evidence-answer triplets.<n>By challenging models to retrieve relevant information from large corpora, FinDER offers a more realistic benchmark for evaluating RAG systems.
  arXiv  Detail & Related papers  (2025-04-22T11:30:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.