FinReflectKG - EvalBench: Benchmarking Financial KG with Multi-Dimensional Evaluation

• URL: http://arxiv.org/abs/2510.05710v1
• Date: Tue, 07 Oct 2025 09:22:48 GMT
• Title: FinReflectKG - EvalBench: Benchmarking Financial KG with Multi-Dimensional Evaluation
• Abstract summary: FinReflectKG - EvalBench is a benchmark and evaluation framework for financial knowledge graphs.<n>It links audited triples to source chunks from S&P 100 filings and supports single-pass, multi-pass, and reflection-agent-based extraction modes.<n>Our findings suggest that, when equipped with explicit bias controls, LLM-as-Judge protocols provide a reliable and cost-efficient alternative to human annotation.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Large language models (LLMs) are increasingly being used to extract structured knowledge from unstructured financial text. Although prior studies have explored various extraction methods, there is no universal benchmark or unified evaluation framework for the construction of financial knowledge graphs (KG). We introduce FinReflectKG - EvalBench, a benchmark and evaluation framework for KG extraction from SEC 10-K filings. Building on the agentic and holistic evaluation principles of FinReflectKG - a financial KG linking audited triples to source chunks from S&P 100 filings and supporting single-pass, multi-pass, and reflection-agent-based extraction modes - EvalBench implements a deterministic commit-then-justify judging protocol with explicit bias controls, mitigating position effects, leniency, verbosity and world-knowledge reliance. Each candidate triple is evaluated with binary judgments of faithfulness, precision, and relevance, while comprehensiveness is assessed on a three-level ordinal scale (good, partial, bad) at the chunk level. Our findings suggest that, when equipped with explicit bias controls, LLM-as-Judge protocols provide a reliable and cost-efficient alternative to human annotation, while also enabling structured error analysis. Reflection-based extraction emerges as the superior approach, achieving best performance in comprehensiveness, precision, and relevance, while single-pass extraction maintains the highest faithfulness. By aggregating these complementary dimensions, FinReflectKG - EvalBench enables fine-grained benchmarking and bias-aware evaluation, advancing transparency and governance in financial AI applications.

Related papers

• KGCQual: An Interpretable Framework for Evaluating the Knowledge Graph Construction Quality from Text [6.373904244962351]
  We propose a novel, interpretable metric for intrinsic Knowledge Graph quality assessment.<n>It measures how closely an automatically extracted graph approximates an "ideal" graph.<n>We evaluate our metric across multiple state-of-the-art triple extraction systems and datasets.
  arXiv  Detail & Related papers  (2026-07-11T08:45:15Z)
• Agentic Retrieval-Augmented Generation for Financial Document Question Answering [7.56842616602779]
  FinAgent-RAG is an agentic RAG framework that orchestrates iterative retrieval-reasoning loops with self-verification.<n>It integrates a Contrastive Financial Retriever trained with hard negative mining to distinguish semantically similar but numerically distinct financial passages.<n>It achieves 76.81%, 78.46%, and 74.96% execution accuracy respectively, outperforming the strongest baseline by 5.62--9.32 percentage points.
  arXiv  Detail & Related papers  (2026-05-06T19:59:51Z)
• MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome [109.15093810810214]
  MiroEval is a benchmark and evaluation framework for deep research systems.<n>The benchmark comprises 100 tasks grounded in real user needs.<n>The proposed evaluation suite assesses deep research systems along three complementary dimensions.
  arXiv  Detail & Related papers  (2026-03-30T13:16:03Z)
• An Optimised Greedy-Weighted Ensemble Framework for Financial Loan Default Prediction [0.8941624592392746]
  This study proposes an Optimised Greedy-Weighted Ensemble framework for loan default prediction.<n>The proposed framework dynamically allocates model weights based on empirical predictive performance.<n>It provides a scalable data-driven approach to support credit assessment, risk monitoring, and financial decision-making.
  arXiv  Detail & Related papers  (2026-03-19T14:04:37Z)
• FinReflectKG -- HalluBench: GraphRAG Hallucination Benchmark for Financial Question Answering Systems [0.0]
  Current Knowledge Graph (KG)-augmented QA systems lack systematic mechanisms to detect hallucinations.<n>We introduce FinBench-QA-Hallucination, a benchmark for evaluating hallucination detection methods in KG-augmented financial QA over SEC 10-K filings.<n>Our findings highlight vulnerabilities in current KG-augmented systems and provide insights for building reliable financial information systems.
  arXiv  Detail & Related papers  (2026-03-11T04:37:53Z)
• From Comprehension to Reasoning: A Hierarchical Benchmark for Automated Financial Research Reporting [19.0993436440595]
  FinReasoning is a benchmark that decomposes Chinese research-report generation into three stages aligned with real analyst.<n>Based on the evaluation results, FinReasoning reveals that most models exhibit a understanding-execution gap.
  arXiv  Detail & Related papers  (2026-02-25T13:44:58Z)
• A Comprehensive Evaluation of LLM Reasoning: From Single-Model to Multi-Agent Paradigms [20.241519889633285]
  Large Language Models (LLMs) are increasingly deployed as reasoning systems, where reasoning paradigms play a critical role.<n>We conduct a comprehensive and unified evaluation of reasoning paradigms, spanning direct single-model generation, CoT-augmented single-model reasoning, and representative MAS.<n>We introduce MIMeBench, a new open-ended benchmark that targets two foundational yet underexplored semantic capabilities.
  arXiv  Detail & Related papers  (2026-01-19T17:23:45Z)
• Enhancing Business Analytics through Hybrid Summarization of Financial Reports [0.152292571922932]
  Financial reports and earnings communications contain large volumes of structured and semi structured information.<n>We present a hybrid summarization framework that combines extractive and abstractive techniques to produce concise and factually reliable summaries.<n>These findings support the development of practical summarization systems for distilling lengthy financial texts into usable business insights.
  arXiv  Detail & Related papers  (2025-12-28T16:25:12Z)
• FinVet: A Collaborative Framework of RAG and External Fact-Checking Agents for Financial Misinformation Detection [0.0]
  FinVet is a novel framework that integrates two Retrieval-Augmented Generation pipelines with external fact-checking.<n>It provides evidence-backed verdicts, source attribution, confidence scores, and explicit uncertainty flags when evidence is insufficient.
  arXiv  Detail & Related papers  (2025-10-13T17:31:49Z)
• Enhancing Credit Risk Prediction: A Meta-Learning Framework Integrating Baseline Models, LASSO, and ECOC for Superior Accuracy [7.254744067646655]
  This research proposes a comprehensive meta-learning framework that synthesizes multiple complementary models.<n>We implement Permutation Feature Importance analysis for each prediction class across all constituent models.<n>Results demonstrate that our framework significantly enhances the accuracy of financial entity classification.
  arXiv  Detail & Related papers  (2025-09-26T14:09:04Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them [58.04324690859212]
  Large Language Models (LLMs) as automated evaluators (LLM-as-a-judge) has revealed critical inconsistencies in current evaluation frameworks.<n>We identify two fundamental types of inconsistencies: Score-Comparison Inconsistency and Pairwise Transitivity Inconsistency.<n>We propose TrustJudge, a probabilistic framework that addresses these limitations through two key innovations.
  arXiv  Detail & Related papers  (2025-09-25T13:04:29Z)
• Evaluating Large Language Models for Financial Reasoning: A CFA-Based Benchmark Study [1.6770212301915661]
  This study presents the first comprehensive evaluation of state-of-the-art LLMs using 1,560 multiple-choice questions from official mock exams across Levels I-III of CFA.<n>We compare models distinguished by core design priorities: multi-modal and computationally powerful, reasoning-specialized and highly accurate, and lightweight efficiency-optimized.
  arXiv  Detail & Related papers  (2025-08-29T06:13:21Z)
• FinAgentBench: A Benchmark Dataset for Agentic Retrieval in Financial Question Answering [57.18367828883773]
  FinAgentBench is a benchmark for evaluating agentic retrieval with multi-step reasoning in finance.<n>The benchmark consists of 26K expert-annotated examples on S&P-500 listed firms.<n>We evaluate a suite of state-of-the-art models and demonstrate how targeted fine-tuning can significantly improve agentic retrieval performance.
  arXiv  Detail & Related papers  (2025-08-07T22:15:22Z)
• CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward [50.97588334916863]
  We develop CompassVerifier, an accurate and robust lightweight verifier model for evaluation and outcome reward.<n>It demonstrates multi-domain competency spanning math, knowledge, and diverse reasoning tasks, with the capability to process various answer types.<n>We introduce VerifierBench benchmark comprising model outputs collected from multiple data sources, augmented through manual analysis of metaerror patterns to enhance CompassVerifier.
  arXiv  Detail & Related papers  (2025-08-05T17:55:24Z)
• Probabilistically Tightened Linear Relaxation-based Perturbation Analysis for Neural Network Verification [83.25968588249776]
  We present a novel framework that combines over-approximation techniques from LiRPA-based approaches with a sampling-based method to compute tight intermediate reachable sets.<n>With negligible computational overhead, $textttPT-LiRPA$ exploiting the estimated reachable sets, significantly tightens the lower and upper linear bounds of a neural network's output.
  arXiv  Detail & Related papers  (2025-07-07T18:45:53Z)
• Supervised Optimism Correction: Be Confident When LLMs Are Sure [91.7459076316849]
  We establish a novel theoretical connection between supervised fine-tuning and offline reinforcement learning.<n>We show that the widely used beam search method suffers from unacceptable over-optimism.<n>We propose Supervised Optimism Correction, which introduces a simple yet effective auxiliary loss for token-level $Q$-value estimations.
  arXiv  Detail & Related papers  (2025-04-10T07:50:03Z)
• FinTSB: A Comprehensive and Practical Benchmark for Financial Time Series Forecasting [58.70072722290475]
  Financial time series (FinTS) record the behavior of human-brain-augmented decision-making.<n>FinTSB is a comprehensive and practical benchmark for financial time series forecasting.
  arXiv  Detail & Related papers  (2025-02-26T05:19:16Z)
• The Certainty Ratio $C_ρ$: a novel metric for assessing the reliability of classifier predictions [0.0]
  This paper introduces the Certainty Ratio ($C_rho$), a novel metric designed to quantify the contribution of confident (certain) versus uncertain predictions to any classification performance measure.<n> Experimental results across 21 datasets and multiple classifiers, including Decision Trees, Naive-Bayes, 3-Nearest Neighbors, and Random Forests, demonstrate that $C_rho$rho reveals critical insights that conventional metrics often overlook.
  arXiv  Detail & Related papers  (2024-11-04T10:50:03Z)
• Self-Evaluation Guided Beam Search for Reasoning [61.523627290397556]
  We introduce a stepwise self-evaluation mechanism to guide and calibrate the reasoning process of Large Language Model (LLM) We propose a decoding algorithm integrating the self-evaluation guidance via beam search. Our approach surpasses the corresponding Codex-backboned baselines in few-shot accuracy by $6.34%$, $9.56%$, and $5.46%$ on the GSM8K, AQuA, and StrategyQA.
  arXiv  Detail & Related papers  (2023-05-01T02:37:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.