FinRCA-Bench: Benchmarking Evidence Retrieval and Reasoning for Financial AI Systems

• URL: http://arxiv.org/abs/2608.18534v1
• Date: Wed, 19 Aug 2026 04:36:01 GMT
• Title: FinRCA-Bench: Benchmarking Evidence Retrieval and Reasoning for Financial AI Systems
• Abstract summary: FinRCA-Bench is a benchmark for 2,250 accounts-payable-to-bank reconciliation cases spanning 14 operational tables.<n>We show that structural retrieval failures outnumber reasoning failures with sufficient retrieval by 95 to 15.<n>On FinRCA-Bench, retrieval architecture strongly observed AI-system performance, and root-cause label is a weak proxy for an auditable diagnosis.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models are increasingly used to support financial operations, but their apparent reasoning performance can depend on whether they receive the right evidence. In financial reconciliation, the evidence needed for diagnosis is distributed across invoices, purchase orders, approvals, allocations, payments, ledger entries, and bank activity, linked by transactional relationships rather than textual similarity. End-to-end accuracy can therefore conflate evidence access with reasoning quality. We introduce FinRCA-Bench, a deterministic synthetic benchmark of 2,250 accounts-payable-to-bank reconciliation cases spanning 14 operational tables, including 1,500 injected failures across 15 causal categories and 750 legitimate or hard-negative cases. Root-cause labels and record-level evidence contracts are hidden from the model, allowing retrieval to be evaluated independently of answer correctness. We compare Rules/SQL, classical machine learning, dense semantic retrieval, deterministic relational expansion, and Typed Provenance Graph Retrieval (TPGR), a typed traversal restricted to persisted transaction relationships. Rules/SQL reaches 84.97% held-out exact accuracy and classical ML reaches 95.44%. Holding the reasoning model, prompt, and generation settings fixed while changing only retrieval increases macro required-record recall from 0.83% to 77.70% and exact 16-class accuracy from 2.05% to 72.44%. Structural retrieval failures outnumber reasoning failures with sufficient retrieval by 95 to 15; 254 correct predictions occur despite incomplete retrieval, and strict returned-evidence contract accuracy is only 5.72%. On FinRCA-Bench, retrieval architecture strongly shapes observed AI-system performance, and a correct root-cause label is a weak proxy for an auditable diagnosis.

Related papers

• State of Thought Enables Endogenous Reasoning [48.902117018115256]
  Test-time reasoning has emerged as a major approach to improving the capabilities of Large Language Models (LLMs)<n>We propose State of Thought, a new reasoning paradigm that enables endogenous reasoning in LLMs.<n>We show that SoT consistently improves mean-baseline accuracy while reducing generated tokens by 62.6% and end-to-end latency by 44.6%.
  arXiv  Detail & Related papers  (2026-09-13T05:55:03Z)
• TrustDABench: Benchmarking Reliability and Robustness of LLMs for Structured Data Analysis [42.62413276719373]
  A trustworthy result should be supported by a valid path from the user question to the relevant data evidence.<n>This requirement creates two diagnostic questions: whether an LLM can refuse to answer or ask for clarification when such a path does not exist, and whether it can preserve the correct analysis when the same evidence is expressed in different table forms.<n>We introduce TrustDABench, a benchmark that operationalizes these questions as reliability and robustness.
  arXiv  Detail & Related papers  (2026-08-25T07:09:01Z)
• Auditable by Construction: An Ontology-Driven Framework for Trustworthy LLM Analytics in Enterprise Finance [0.0]
  This paper argues that retrieval-augmented generation for enterprise finance should be evaluated on auditability alongside accuracy.<n>We present the Knowledge-Driven Analytics Framework (KDAF), which builds ontology-driven knowledge systems through six iterative stages.<n>An evaluation on FinanceBench (145 questions) compares KDAF against zero-context inference, BM25, concept-weighted lexical retrieval, and ungrounded graph traversal.
  arXiv  Detail & Related papers  (2026-08-21T01:35:34Z)
• V-FiLLM: Verified Financial LLM Reasoning Benchmark [0.37189423451031356]
  We introduce V-FiLLM, a framework that generates financial reasoning benchmarks from computation trees grounded in real tables.<n>V-FiLLM exposes four independently controllable axes of difficulty including computation depth, expression breadth, financial concept complexity, and context size.<n>We show that lightweight LoRA fine-tuning on verified chain-of-thought traces improves accuracy from 81.1% to 85.6% on held-out problems.
  arXiv  Detail & Related papers  (2026-08-11T15:18:47Z)
• Diagnosing Fine-Grained Inconsistency Classification in Financial Disclosure Text [3.2712318205295907]
  Financial disclosures contain numerical claims, temporal statements, entity references, policy commitments, and risk descriptions that may conflict in qualitatively different ways.<n>We study this problem as fine-grained inconsistency.<n>A fine-tuned 300M encoder reaches 61.9% accuracy, compared with 61.5% for a LoRA-adapted Qwen3.5-9B model and 61.3% for GPT-adapted generative models.
  arXiv  Detail & Related papers  (2026-07-29T01:03:53Z)
• AgentFinVQA: A Deployable Multi-Agent Pipeline for Auditable Financial Chart QA [7.135926644099901]
  We present AgentFinVQA, a multi-agent pipeline that decomposes each query into planning, OCR, legend grounding, visual inspection, and verification.<n>On FinMME, AgentFinVQA improves $+7.68$ pp over a primary-backbone matched zero-shot baseline with a proprietary backbone.<n>Error analysis shows that question misunderstanding, legend confusion and extraction error account for nearly two-thirds of failures.
  arXiv  Detail & Related papers  (2026-06-18T04:33:07Z)
• FinVerBench: Benchmark Validity and Calibration in Large Language Model Financial Statement Verification [0.0]
  FinVerBench is a benchmark and validity study for financial statement verification.<n>It is built from SEC 10-K variant filings for 43 S&P 500 companies.
  arXiv  Detail & Related papers  (2026-05-28T08:30:15Z)
• Agentic Proving for Program Verification [44.663012714194025]
  Agentic systems have emerged as state-of-the-art approaches for automated theorem proving in formal mathematics.<n>We evaluate Claude Code in an agentic proving framework on CLEVER, a Lean 4 benchmark for verifiable code generation.
  arXiv  Detail & Related papers  (2026-05-22T15:41:27Z)
• Replayable Financial Agents: A Determinism-Faithfulness Assurance Harness for Tool-Using LLM Agents [0.7699235580548228]
  LLM agents struggle with regulatory audit replay: when asked to reproduce a transaction flagged decision with identical inputs, most deployments fail to return consistent results.<n>This paper introduces the DeterminismFaithfulness Assurance Harness (DFAH), a framework for measuring trajectory determinism and evidence-conditioned faithfulness in tool-using agents deployed in financial services.
  arXiv  Detail & Related papers  (2026-01-17T19:47:55Z)
• SSR: Safeguarding Staking Rewards by Defining and Detecting Logical Defects in DeFi Staking [55.62033436283969]
  Decentralized Finance (DeFi) staking is one of the most prominent applications within the DeFi ecosystem.<n> logical defects in DeFi staking could enable attackers to claim unwarranted rewards.<n>We developed SSR (Safeguarding Staking Reward), a static analysis tool designed to detect logical defects in DeFi staking contracts.
  arXiv  Detail & Related papers  (2026-01-09T15:01:41Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• Causal Understanding by LLMs: The Role of Uncertainty [43.87879175532034]
  Recent papers show LLMs achieve near-random accuracy in causal relation classification.<n>We investigate whether pretraining exposure to causal examples improves causal understanding.
  arXiv  Detail & Related papers  (2025-09-24T13:06:35Z)
• AGENTICT$^2$S:Robust Text-to-SPARQL via Agentic Collaborative Reasoning over Heterogeneous Knowledge Graphs for the Circular Economy [42.73610751710192]
  AgenticT$2$S is a framework that decomposes knowledge graphs into subtasks managed by specialized agents.<n>A two-stage verifier detects structurally invalid and semantically underspecified queries.<n>Experiments on real-world circular economy KGs demonstrate that AgenticT$2$S improves execution accuracy by 17.3%.
  arXiv  Detail & Related papers  (2025-08-03T15:58:54Z)
• Benchmarking Reasoning Robustness in Large Language Models [76.79744000300363]
  We find significant performance degradation on novel or incomplete data.<n>These findings highlight the reliance on recall over rigorous logical inference.<n>This paper introduces a novel benchmark, termed as Math-RoB, that exploits hallucinations triggered by missing information to expose reasoning gaps.
  arXiv  Detail & Related papers  (2025-03-06T15:36:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.