FRED: Financial Retrieval-Enhanced Detection and Editing of Hallucinations in Language Models

• URL: http://arxiv.org/abs/2507.20930v2
• Date: Wed, 30 Jul 2025 17:19:41 GMT
• Title: FRED: Financial Retrieval-Enhanced Detection and Editing of Hallucinations in Language Models
• Abstract summary: Hallucinations in large language models pose a critical challenge for applications requiring factual reliability.<n>This work presents an effective approach for detecting and editing factually incorrect content in model-generated responses.
• Score: 1.9894117371899613
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Hallucinations in large language models pose a critical challenge for applications requiring factual reliability, particularly in high-stakes domains such as finance. This work presents an effective approach for detecting and editing factually incorrect content in model-generated responses based on the provided context. Given a user-defined domain-specific error taxonomy, we construct a synthetic dataset by inserting tagged errors into financial question-answering corpora and then fine-tune four language models, Phi-4, Phi-4-mini, Qwen3-4B, and Qwen3-14B, to detect and edit these factual inaccuracies. Our best-performing model, fine-tuned Phi-4, achieves an 8% improvement in binary F1 score and a 30% gain in overall detection performance compared to OpenAI-o3. Notably, our fine-tuned Phi-4-mini model, despite having only 4 billion parameters, maintains competitive performance with just a 2% drop in binary detection and a 0.1% decline in overall detection compared to OpenAI-o3. Our work provides a practical solution for detecting and editing factual inconsistencies in financial text generation while introducing a generalizable framework that can enhance the trustworthiness and alignment of large language models across diverse applications beyond finance. Our code and data are available at https://github.com/pegasi-ai/shield.

Related papers

• From Comprehension to Reasoning: A Hierarchical Benchmark for Automated Financial Research Reporting [19.0993436440595]
  FinReasoning is a benchmark that decomposes Chinese research-report generation into three stages aligned with real analyst.<n>Based on the evaluation results, FinReasoning reveals that most models exhibit a understanding-execution gap.
  arXiv  Detail & Related papers  (2026-02-25T13:44:58Z)
• Robust and Real-Time Bangladeshi Currency Recognition: A Dual-Stream MobileNet and EfficientNet Approach [2.3053825622580133]
  We build a new Bangladeshi banknote dataset that includes both controlled and real-world scenarios.<n>We incorporate four additional datasets, including public benchmarks, to cover various complexities and improve the model's generalization.<n>The proposed model achieves 97.95% accuracy on controlled datasets, 92.84% on complex backgrounds, and 94.98% accuracy when combining all datasets.
  arXiv  Detail & Related papers  (2026-01-31T17:37:16Z)
• Scaling Truth: The Confidence Paradox in AI Fact-Checking [0.8201655885319955]
  Large language models (LLMs) hold promise in automating fact verification, yet their effectiveness across global contexts remains uncertain.<n>We systematically evaluate nine established LLMs across multiple categories using 5,000 claims previously assessed by 174 professional fact-checking organizations across 47 languages.<n>Findings reveal a concerning pattern resembling the Dunning-Kruger effect: smaller, models show high confidence despite lower accuracy, while larger models demonstrate higher accuracy but lower confidence.
  arXiv  Detail & Related papers  (2025-09-10T17:36:25Z)
• Reliable Decision Support with LLMs: A Framework for Evaluating Consistency in Binary Text Classification Applications [0.7124971549479361]
  This study introduces a framework for evaluating consistency in large language model (LLM) binary text classification.<n>We determine sample size requirements, develop metrics for invalid responses, and evaluate intra- and inter-rater reliability.
  arXiv  Detail & Related papers  (2025-05-20T21:12:58Z)
• NeKo: Toward Post Recognition Generative Correction Large Language Models with Task-Oriented Experts [57.53692236201343]
  We propose a Multi-Task Correction MoE, where we train the experts to become an expert'' of speech-to-text, language-to-text and vision-to-text datasets. NeKo performs competitively on grammar and post-OCR correction as a multi-task model.
  arXiv  Detail & Related papers  (2024-11-08T20:11:24Z)
• Unearthing Skill-Level Insights for Understanding Trade-Offs of Foundation Models [61.467781476005435]
  skill-wise performance is obscured when inspecting aggregate accuracy, under-utilizing the rich signal modern benchmarks contain. We propose an automatic approach to recover the underlying skills relevant for any evaluation instance, by way of inspecting model-generated rationales. Our skill-slices and framework open a new avenue in model evaluation, leveraging skill-specific analyses to unlock a more granular and actionable understanding of model capabilities.
  arXiv  Detail & Related papers  (2024-10-17T17:51:40Z)
• Advancing Anomaly Detection: Non-Semantic Financial Data Encoding with LLMs [49.57641083688934]
  We introduce a novel approach to anomaly detection in financial data using Large Language Models (LLMs) embeddings. Our experiments demonstrate that LLMs contribute valuable information to anomaly detection as our models outperform the baselines.
  arXiv  Detail & Related papers  (2024-06-05T20:19:09Z)
• $C^3$: Confidence Calibration Model Cascade for Inference-Efficient Cross-Lingual Natural Language Understanding [28.853593305486832]
  Cross-lingual natural language understanding (NLU) is a critical task in natural language processing (NLP) Recent advancements have seen multilingual pre-trained language models (mPLMs) significantly enhance the performance of these tasks. Existing model cascade methods seek to enhance inference efficiency by greedily selecting the lightest model capable of processing the current input from a variety of models.
  arXiv  Detail & Related papers  (2024-02-25T05:07:56Z)
• FinLLM-B: When Large Language Models Meet Financial Breakout Trading [13.465954970263502]
  FinLLM-B is the premier large language model for financial breakout detection.<n>We have developed a novel framework for large language models, namely multi-stage structure.<n>Compared to GPT-3.5, FinLLM-B improves the average accuracy of answers and rational by 49.97%, with the multi-stage structure contributing 9.72% to the improvement.
  arXiv  Detail & Related papers  (2024-02-12T10:04:07Z)
• Fine-tuning Language Models for Factuality [96.5203774943198]
  Large pre-trained language models (LLMs) have led to their widespread use, sometimes even as a replacement for traditional search engines. Yet language models are prone to making convincing but factually inaccurate claims, often referred to as 'hallucinations' In this work, we fine-tune language models to be more factual, without human labeling.
  arXiv  Detail & Related papers  (2023-11-14T18:59:15Z)
• Preserving Knowledge Invariance: Rethinking Robustness Evaluation of Open Information Extraction [49.15931834209624]
  We present the first benchmark that simulates the evaluation of open information extraction models in the real world.<n>We design and annotate a large-scale testbed in which each example is a knowledge-invariant clique.<n>By further elaborating the robustness metric, a model is judged to be robust if its performance is consistently accurate on the overall cliques.
  arXiv  Detail & Related papers  (2023-05-23T12:05:09Z)
• Prompting GPT-3 To Be Reliable [117.23966502293796]
  This work decomposes reliability into four facets: generalizability, fairness, calibration, and factuality. We find that GPT-3 outperforms smaller-scale supervised models by large margins on all these facets.
  arXiv  Detail & Related papers  (2022-10-17T14:52:39Z)
• Plex: Towards Reliability using Pretrained Large Model Extensions [69.13326436826227]
  We develop ViT-Plex and T5-Plex, pretrained large model extensions for vision and language modalities, respectively. Plex greatly improves the state-of-the-art across reliability tasks, and simplifies the traditional protocol. We demonstrate scaling effects over model sizes up to 1B parameters and pretraining dataset sizes up to 4B examples.
  arXiv  Detail & Related papers  (2022-07-15T11:39:37Z)
• Error Detection in Large-Scale Natural Language Understanding Systems Using Transformer Models [0.0]
  Large-scale conversational assistants like Alexa, Siri, Cortana and Google Assistant process every utterance using multiple models for domain, intent and named entity recognition. We address this challenge to detect domain classification errors using offline Transformer models. We combine utterance encodings from a RoBERTa model with the Nbest hypothesis produced by the production system. We then fine-tune end-to-end in a multitask setting using a small dataset of humanannotated utterances with domain classification errors.
  arXiv  Detail & Related papers  (2021-09-04T00:10:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.