FinReflectKG - MultiHop: Financial QA Benchmark for Reasoning with Knowledge Graph Evidence

• URL: http://arxiv.org/abs/2510.02906v1
• Date: Fri, 03 Oct 2025 11:19:31 GMT
• Title: FinReflectKG - MultiHop: Financial QA Benchmark for Reasoning with Knowledge Graph Evidence
• Abstract summary: We present FinReflectKG - MultiHop, a benchmark built on FinReflectKG, a temporally indexed financial Knowledge Graph.<n>We generate financial analyst style questions with exact supporting evidence from the KG.<n>Across both reasoning and non-reasoning models, KG-guided precise retrieval yields substantial gains.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Multi-hop reasoning over financial disclosures is often a retrieval problem before it becomes a reasoning or generation problem: relevant facts are dispersed across sections, filings, companies, and years, and LLMs often expend excessive tokens navigating noisy context. Without precise Knowledge Graph (KG)-guided selection of relevant context, even strong reasoning models either fail to answer or consume excessive tokens, whereas KG-linked evidence enables models to focus their reasoning on composing already retrieved facts. We present FinReflectKG - MultiHop, a benchmark built on FinReflectKG, a temporally indexed financial KG that links audited triples to source chunks from S&P 100 filings (2022-2024). Mining frequent 2-3 hop subgraph patterns across sectors (via GICS taxonomy), we generate financial analyst style questions with exact supporting evidence from the KG. A two-phase pipeline first creates QA pairs via pattern-specific prompts, followed by a multi-criteria quality control evaluation to ensure QA validity. We then evaluate three controlled retrieval scenarios: (S1) precise KG-linked paths; (S2) text-only page windows centered on relevant text spans; and (S3) relevant page windows with randomizations and distractors. Across both reasoning and non-reasoning models, KG-guided precise retrieval yields substantial gains on the FinReflectKG - MultiHop QA benchmark dataset, boosting correctness scores by approximately 24 percent while reducing token utilization by approximately 84.5 percent compared to the page window setting, which reflects the traditional vector retrieval paradigm. Spanning intra-document, inter-year, and cross-company scopes, our work underscores the pivotal role of knowledge graphs in efficiently connecting evidence for multi-hop financial QA. We also release a curated subset of the benchmark (555 QA Pairs) to catalyze further research.

Related papers

• Theseus in the Graph: Towards Traceable Multi-Hop Graph Navigation [11.103813686369685]
  Multi-Hop Knowledge Graph Question Answering tasks require models to assemble relational evidence along paths in a KG to answer natural-language questions.<n>We re-frame multi-hop KGQA as a question-conditioned graph navigation problem.<n>In this setting, an agent receives a KG, a question, and a topic entity, and traverses a sequence of relations towards the answer, making the reasoning path explicit.
  arXiv  Detail & Related papers  (2026-09-13T14:06:43Z)
• Enhancing Financial Question Answering: A Novel Benchmark Dataset of Banks' financial statements [6.625637186756314]
  We introduce FinRAG-QA, a novel benchmark dataset for financial question answering.<n>FinRAG-QA targets cross-institutional retrieval over documents averaging 198k words.<n>Contextual chunk enrichment combined with a retrieval-optimised embedding model raises NDCG@10 from 0.322 to 0.710.
  arXiv  Detail & Related papers  (2026-09-03T10:52:56Z)
• Travel-Oriented Reasoning Large Language Model via Domain-Specific Knowledge Graphs [0.0]
  Large language models (LLMs) demonstrate broad reasoning abilities but struggle with accuracy and reliability in specialized domains such as travel.<n>We propose a modular pipeline for building a travel-domain reasoning LLM grounded in an expert-designed knowledge graph.<n>Our pipeline integrates a travel KG that encodes domain entities and their relationships, a supervised fine-tuning stage that embeds the domain knowledge into a reasoning-capable LLM.
  arXiv  Detail & Related papers  (2026-06-28T07:53:11Z)
• PluriHop: Exhaustive, Recall-Sensitive QA over Distractor-Rich Corpora [0.0]
  We introduce PluriHopWIND, a diagnostic multilingual dataset of 48 pluri-hop questions built from 191 real-world wind industry reports in German and English.<n>We show that PluriHopWIND is 8-40% more repetitive than other common datasets and thus has higher density of distractor documents.<n>We propose PluriHopRAG, a RAG architecture that follows a "check all documents individually, filter cheaply" approach.
  arXiv  Detail & Related papers  (2025-10-16T07:22:58Z)
• FinLFQA: Evaluating Attributed Text Generation of LLMs in Financial Long-Form Question Answering [57.43420753842626]
  FinLFQA is a benchmark designed to evaluate the ability of Large Language Models to generate long-form answers to complex financial questions.<n>We provide an automatic evaluation framework covering both answer quality and attribution quality.
  arXiv  Detail & Related papers  (2025-10-07T20:06:15Z)
• FinAgentBench: A Benchmark Dataset for Agentic Retrieval in Financial Question Answering [57.18367828883773]
  FinAgentBench is a benchmark for evaluating agentic retrieval with multi-step reasoning in finance.<n>The benchmark consists of 26K expert-annotated examples on S&P-500 listed firms.<n>We evaluate a suite of state-of-the-art models and demonstrate how targeted fine-tuning can significantly improve agentic retrieval performance.
  arXiv  Detail & Related papers  (2025-08-07T22:15:22Z)
• AGENTICT$^2$S:Robust Text-to-SPARQL via Agentic Collaborative Reasoning over Heterogeneous Knowledge Graphs for the Circular Economy [42.73610751710192]
  AgenticT$2$S is a framework that decomposes knowledge graphs into subtasks managed by specialized agents.<n>A two-stage verifier detects structurally invalid and semantically underspecified queries.<n>Experiments on real-world circular economy KGs demonstrate that AgenticT$2$S improves execution accuracy by 17.3%.
  arXiv  Detail & Related papers  (2025-08-03T15:58:54Z)
• The benefits of query-based KGQA systems for complex and temporal questions in LLM era [55.20230501807337]
  Large language models excel in question-answering (QA) yet still struggle with multi-hop reasoning and temporal questions.<n> Query-based knowledge graph QA (KGQA) offers a modular alternative by generating executable queries instead of direct answers.<n>We explore multi-stage query-based framework for WikiData QA, proposing multi-stage approach that enhances performance on challenging multi-hop and temporal benchmarks.
  arXiv  Detail & Related papers  (2025-07-16T06:41:03Z)
• PGDA-KGQA: A Prompt-Guided Generative Framework with Multiple Data Augmentation Strategies for Knowledge Graph Question Answering [3.8623708225544755]
  Knowledge Graph Question Answering (KGQA) is a crucial task in natural language processing.<n>We propose PGDA-KGQA, a prompt-guided generative framework with multiple data augmentation strategies for KGQA.<n> Experiments demonstrate that PGDA-KGQA outperforms state-of-the-art methods on standard KGQA.
  arXiv  Detail & Related papers  (2025-06-11T05:56:03Z)
• Diagnosing and Addressing Pitfalls in KG-RAG Datasets: Toward More Reliable Benchmarking [63.84117489519164]
  Knowledge Graph Question Answering systems rely on high-quality benchmarks to evaluate complex multi-hop reasoning.<n>Despite their widespread use, popular datasets such as WebQSP and CWQ suffer from critical quality issues.<n>We introduce KGQAGen, an LLM-in-the-loop framework that systematically resolves these pitfalls.<n>Our findings advocate for more rigorous benchmark construction and position KGQAGen as a scalable framework for advancing KGQA evaluation.
  arXiv  Detail & Related papers  (2025-05-29T14:44:52Z)
• KG-QAGen: A Knowledge-Graph-Based Framework for Systematic Question Generation and Long-Context LLM Evaluation [3.618621510356872]
  KG-QAGen is a framework that extracts QA pairs at multiple complexity levels.<n>We construct a dataset of 20,139 QA pairs and open-source a part of it.<n>We evaluate 13 proprietary and open-source LLMs and observe that even the best-performing models are struggling with set-based comparisons.
  arXiv  Detail & Related papers  (2025-05-18T16:46:39Z)
• FAMMA: A Benchmark for Financial Domain Multilingual Multimodal Question Answering [18.821122274064116]
  We introduce FAMMA, an open-source benchmark for underlinefinunderlineancial underlinemultilingual underlinemultimodal question underlineanswering (QA)<n>Our benchmark aims to evaluate the abilities of large language models (LLMs) in answering complex reasoning questions that require advanced financial knowledge.
  arXiv  Detail & Related papers  (2024-10-06T15:41:26Z)
• UniKGQA: Unified Retrieval and Reasoning for Solving Multi-hop Question Answering Over Knowledge Graph [89.98762327725112]
  Multi-hop Question Answering over Knowledge Graph(KGQA) aims to find the answer entities that are multiple hops away from the topic entities mentioned in a natural language question. We propose UniKGQA, a novel approach for multi-hop KGQA task, by unifying retrieval and reasoning in both model architecture and parameter learning.
  arXiv  Detail & Related papers  (2022-12-02T04:08:09Z)
• QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering [122.84513233992422]
  We propose a new model, QA-GNN, which addresses the problem of answering questions using knowledge from pre-trained language models (LMs) and knowledge graphs (KGs) We show its improvement over existing LM and LM+KG models, as well as its capability to perform interpretable and structured reasoning.
  arXiv  Detail & Related papers  (2021-04-13T17:32:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.