Auditable by Construction: An Ontology-Driven Framework for Trustworthy LLM Analytics in Enterprise Finance
- URL: http://arxiv.org/abs/2608.20661v1
- Date: Fri, 21 Aug 2026 01:35:34 GMT
- Title: Auditable by Construction: An Ontology-Driven Framework for Trustworthy LLM Analytics in Enterprise Finance
- Abstract summary: This paper argues that retrieval-augmented generation for enterprise finance should be evaluated on auditability alongside accuracy.<n>We present the Knowledge-Driven Analytics Framework (KDAF), which builds ontology-driven knowledge systems through six iterative stages.<n>An evaluation on FinanceBench (145 questions) compares KDAF against zero-context inference, BM25, concept-weighted lexical retrieval, and ungrounded graph traversal.
- Score: 0.0
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Enterprise adoption of large language models in finance is constrained less by fluency than by trust: in Financial Planning and Analysis (FP&A) and other regulated workflows, an answer is usable only if it is traceable to authoritative sources and auditable after the fact. This paper argues that retrieval-augmented generation for enterprise finance should be evaluated on auditability alongside accuracy, and presents the Knowledge-Driven Analytics Framework (KDAF), which builds ontology-driven knowledge systems through six iterative stages and retrieves evidence via Context-Aware Relevance Propagation (CARP), so that every retrieved fact carries its relationship type, confidence, and source lineage. An evaluation on FinanceBench (145 questions) compares KDAF against zero-context inference, BM25, concept-weighted lexical retrieval, and ungrounded graph traversal. First, retrieval is necessary: zero-context inference reaches 4.1% correctness against 10-12% for retrieval-augmented conditions. Second, on answer correctness the retrieval conditions are statistically indistinguishable (KDAF vs BM25: -0.007, 95% CI [-0.021, 0.000]), so accuracy alone does not justify structured retrieval here -- a negative result we report explicitly. Third, on auditability the ordering reverses: KDAF attains the highest citation traceability F1 (0.515), exceeding ungrounded traversal by +0.027 (CI [0.006, 0.050]) and BM25 by +0.052 (CI [0.024, 0.083]), intervals excluding zero. Graph-structured retrieval also admits no evidence from outside the question subject entity (0 of 426 items, against 16.8% and 20.2% for lexical baselines), and every selected item resolves to a complete provenance chain. We argue that auditability, not accuracy, is the axis on which ontology-grounded retrieval earns its cost.
Related papers
- FinRCA-Bench: Benchmarking Evidence Retrieval and Reasoning for Financial AI Systems [0.0]
FinRCA-Bench is a benchmark for 2,250 accounts-payable-to-bank reconciliation cases spanning 14 operational tables.<n>We show that structural retrieval failures outnumber reasoning failures with sufficient retrieval by 95 to 15.<n>On FinRCA-Bench, retrieval architecture strongly observed AI-system performance, and root-cause label is a weak proxy for an auditable diagnosis.
arXiv Detail & Related papers (2026-08-19T04:36:01Z) - ReliableTableQA:How Much Supervision Does Reliability Annotation Need? [1.3152347996965301]
We introduce ReliableTableQA, a framework for training an LLM to annotate the statistical reliability of QA results.<n>We quantify the Unreliable Confident Answer Rate (UCAR), a failure we quantify as the Unreliable Confident Answer Rate (UCAR)<n>Our findings reframe reliability annotation as a data-efficiency problem and delineate precisely when reinforcement fine-tuning does and does not pay off.
arXiv Detail & Related papers (2026-07-10T16:57:30Z) - ITPEval: Benchmarking Formal Translation Across Interactive Theorem Provers [77.95933562142014]
ITPEval is the first benchmark for evaluating automated formal proof translation across four major ITPs.<n>We release itpeval, a unified multi-ITP verification infrastructure with state-isolated warm backends.
arXiv Detail & Related papers (2026-07-07T22:05:55Z) - FinVerBench: Benchmark Validity and Calibration in Large Language Model Financial Statement Verification [0.0]
FinVerBench is a benchmark and validity study for financial statement verification.<n>It is built from SEC 10-K variant filings for 43 S&P 500 companies.
arXiv Detail & Related papers (2026-05-28T08:30:15Z) - Confidence-Aware Alignment Makes Reasoning LLMs More Reliable [65.44962502963378]
We introduce CASPO, a framework that aligns token-level confidence with step-wise logical correctness without training a separate reward model.<n>During inference, we propose Confidence-aware Thought (CaT), which dynamically prunes uncertain reasoning branches with negligible O(V) latency.<n>Experiments across ten benchmarks and multiple model families show that CASPO consistently improves reasoning reliability and inference efficiency.
arXiv Detail & Related papers (2026-05-08T07:08:25Z) - Replayable Financial Agents: A Determinism-Faithfulness Assurance Harness for Tool-Using LLM Agents [0.7699235580548228]
LLM agents struggle with regulatory audit replay: when asked to reproduce a transaction flagged decision with identical inputs, most deployments fail to return consistent results.<n>This paper introduces the DeterminismFaithfulness Assurance Harness (DFAH), a framework for measuring trajectory determinism and evidence-conditioned faithfulness in tool-using agents deployed in financial services.
arXiv Detail & Related papers (2026-01-17T19:47:55Z) - All That Glisters Is Not Gold: A Benchmark for Reference-Free Counterfactual Financial Misinformation Detection [67.89888669159899]
RFC Bench is a benchmark for evaluating large language models on financial misinformation under realistic news.<n>The benchmark defines two complementary tasks: reference free misinformation detection and comparison based diagnosis.
arXiv Detail & Related papers (2026-01-07T18:18:28Z) - FinCARE: Financial Causal Analysis with Reasoning and Evidence [39.146761527401424]
Portfolio managers rely on correlation-based analysis and methods that fail to capture true causal relationships driving performance.<n>We present a hybrid framework that integrates statistical causal discovery algorithms with domain knowledge from two complementary sources: a financial knowledge graph extracted from SEC 10-K filings and large language model reasoning.
arXiv Detail & Related papers (2025-10-23T05:14:28Z) - Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
arXiv Detail & Related papers (2025-09-25T14:05:55Z) - TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them [58.04324690859212]
Large Language Models (LLMs) as automated evaluators (LLM-as-a-judge) has revealed critical inconsistencies in current evaluation frameworks.<n>We identify two fundamental types of inconsistencies: Score-Comparison Inconsistency and Pairwise Transitivity Inconsistency.<n>We propose TrustJudge, a probabilistic framework that addresses these limitations through two key innovations.
arXiv Detail & Related papers (2025-09-25T13:04:29Z) - FinAgentBench: A Benchmark Dataset for Agentic Retrieval in Financial Question Answering [57.18367828883773]
FinAgentBench is a benchmark for evaluating agentic retrieval with multi-step reasoning in finance.<n>The benchmark consists of 26K expert-annotated examples on S&P-500 listed firms.<n>We evaluate a suite of state-of-the-art models and demonstrate how targeted fine-tuning can significantly improve agentic retrieval performance.
arXiv Detail & Related papers (2025-08-07T22:15:22Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.