KARL: Mitigating Hallucinations in LLMs via Knowledge-Boundary-Aware Reinforcement Learning

• URL: http://arxiv.org/abs/2604.22779v1
• Date: Fri, 03 Apr 2026 11:30:56 GMT
• Title: KARL: Mitigating Hallucinations in LLMs via Knowledge-Boundary-Aware Reinforcement Learning
• Abstract summary: KARL is a novel framework that aligns an LLM's abstention behavior with its evolving knowledge boundary.<n>It achieves a superior accuracy-hallucination trade-off, effectively suppressing hallucinations while maintaining high accuracy across both in-distribution and out-of-distribution scenarios.
• Score: 46.007379982024474
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Enabling large language models (LLMs) to appropriately abstain from answering questions beyond their knowledge is crucial for mitigating hallucinations. While existing reinforcement learning methods foster autonomous abstention, they often compromise answer accuracy because their static reward mechanisms, agnostic to models' knowledge boundaries, drive models toward excessive caution. In this work, we propose KARL, a novel framework that continuously aligns an LLM's abstention behavior with its evolving knowledge boundary. KARL introduces two core innovations: a Knowledge-Boundary-Aware Reward that performs online knowledge boundary estimation using within-group response statistics, dynamically rewarding correct answers or guided abstention; and a Two-Stage RL Training Strategy that first explores the knowledge boundary and bypasses the "abstention trap", and subsequently converts incorrect answers beyond the knowledge boundary into abstentions without sacrificing accuracy. Extensive experiments on multiple benchmarks demonstrate that KARL achieves a superior accuracy-hallucination trade-off, effectively suppressing hallucinations while maintaining high accuracy across both in-distribution and out-of-distribution scenarios.

Related papers

• Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs [60.80435138270317]
  We apply reinforcement learning with metacognitive feedback to the problem of faithful calibration.<n>Experiments show RLMF achieves generalizable, state-of-the-art FC on diverse tasks.<n> RLMF surpasses standard RL by up to 63% while enhancing models' ability to assess and express their own capability limits.
  arXiv  Detail & Related papers  (2026-06-30T17:56:01Z)
• KbSD: Knowledge Boundary aware Self-Distillation for Behavioral Calibration in Agentic Search [13.024432404905198]
  We propose KbSD (Knowledge boundary Self-Distillation), a framework that tackles the limitation through dense token-level supervision.<n>KbSD constructs a hint-augmented teacher, architecturally identical to the student, that receives explicit knowledge boundary signals.<n>Experiments show that KbSD consistently improves both task accuracy and hallucination mitigation over strong baselines.
  arXiv  Detail & Related papers  (2026-06-29T06:56:18Z)
• Distinguishable Deletion: Unifying Knowledge Erasure and Refusal for Large Language Model Unlearning [58.725080160369494]
  Distinguishable Deletion ($mathrmD2$) is a paradigm that restricts the response distribution in the latent representation rather than specific tokens.<n>We introduce an energy index that quantifies the presence of knowledge and the separation between unlearned and retained content.<n>Experiments demonstrate that EUA significantly outperforms previous methods, indicating the superiority of $mathrmD2$.
  arXiv  Detail & Related papers  (2026-05-16T03:15:35Z)
• GRADE: Probing Knowledge Gaps in LLMs through Gradient Subspace Dynamics [51.647572539626715]
  We present GRADE (Gradient Dynamics for knowledge gap detection), which quantifies the knowledge gap via the cross-layer rank ratio of the gradient to that of the corresponding hidden state subspace.<n>We validate modelname on six benchmarks, demonstrating its effectiveness and robustness to input perturbations.<n>In addition, we present a case study showing how the gradient chain can generate interpretable explanations of knowledge gaps for long-form answers.
  arXiv  Detail & Related papers  (2026-04-03T07:52:26Z)
• Agentic Uncertainty Quantification [76.94013626702183]
  We propose a unified Dual-Process Agentic UQ (AUQ) framework that transforms verbalized uncertainty into active, bi-directional control signals.<n>Our architecture comprises two complementary mechanisms: System 1 (Uncertainty-Aware Memory, UAM), which implicitly propagates verbalized confidence and semantic explanations to prevent blind decision-making; and System 2 (Uncertainty-Aware Reflection, UAR), which utilizes these explanations as rational cues to trigger targeted inference-time resolution only when necessary.
  arXiv  Detail & Related papers  (2026-01-22T07:16:26Z)
• Hindsight Distillation Reasoning with Knowledge Encouragement Preference for Knowledge-based Visual Question Answering [55.368681418311894]
  Existing Knowledge-based Visual Question Answering (KBVQA) methods either utilize implicit knowledge in multimodal large language models (MLLMs) via in-context learning or explicit knowledge via retrieval augmented generation.<n>We provide a Hindsight Distilled Reasoning (HinD) framework with Knowledge Encouragement Preference Optimization (KEPO)<n> Experiments on OK-VQA and A-OKVQA validate the effectiveness of HinD, showing that HinD with elicited reasoning from 7B-size MLLM achieves superior performance without commercial model APIs or outside knowledge.
  arXiv  Detail & Related papers  (2025-11-14T10:03:23Z)
• Wisdom is Knowing What not to Say: Hallucination-Free LLMs Unlearning via Attention Shifting [11.725875396424927]
  We introduce Attention-Shifting (AS) framework for selective unlearning.<n>AS is driven by two design objectives: (1) context-preserving suppression that attenuates attention to fact-bearing tokens without disrupting LLMs' linguistic structure; and (2) hallucination-resistant response shaping that discourages fabricated completions when queried about unlearning content.<n> Experimental results show that AS improves performance over the state-of-the-art unlearning methods, achieving up to 15% higher accuracy on the ToFU benchmark and 10% on the TDEC benchmark, while maintaining competitive hallucination-free unlearning effectiveness.
  arXiv  Detail & Related papers  (2025-10-20T06:50:03Z)
• Harnessing RLHF for Robust Unanswerability Recognition and Trustworthy Response Generation in LLMs [2.217239320172707]
  This paper introduces Self-Aware LLM for Unanswerability (SALU), a novel approach that deeply integrates unanswerability detection directly within the generative process.<n>SALU is trained using a multi-task learning framework for both standard Question Answering (QA) and explicit abstention generation for unanswerable queries.<n>It consistently outperforms strong baselines, including hybrid LLM-classifier systems, in overall accuracy for correctly answering or abstaining from questions.
  arXiv  Detail & Related papers  (2025-07-22T18:44:18Z)
• Know the Unknown: An Uncertainty-Sensitive Method for LLM Instruction Tuning [18.283963879468466]
  Large language models (LLMs) demonstrate remarkable capabilities but face challenges from hallucinations.<n>We introduce Uncertainty-and-Sensitivity-Aware Tuning (US-Tuning), a novel two-stage approach for contextual question answering.<n>Our experimental results demonstrate that US-Tuning not only significantly reduces incorrect answers in contextual QA but also improves models' faithfulness to their parametric knowledge.
  arXiv  Detail & Related papers  (2024-06-14T14:56:04Z)
• Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedback [14.120154004011084]
  Large Language Models (LLMs) often generate erroneous outputs, known as hallucinations. We present a novel alignment framework called Reinforcement Learning from Knowledge Feedback (RLKF)
  arXiv  Detail & Related papers  (2024-03-27T08:39:56Z)
• Knowledge Verification to Nip Hallucination in the Bud [69.79051730580014]
  We demonstrate the feasibility of mitigating hallucinations by verifying and minimizing the inconsistency between external knowledge present in the alignment data and the intrinsic knowledge embedded within foundation LLMs. We propose a novel approach called Knowledge Consistent Alignment (KCA), which employs a well-aligned LLM to automatically formulate assessments based on external knowledge. We demonstrate the superior efficacy of KCA in reducing hallucinations across six benchmarks, utilizing foundation LLMs of varying backbones and scales.
  arXiv  Detail & Related papers  (2024-01-19T15:39:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.