Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RL

• URL: http://arxiv.org/abs/2604.17073v1
• Date: Sat, 18 Apr 2026 17:21:40 GMT
• Title: Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RL
• Abstract summary: We study queries that are clear in meaning but cannot be reliably resolved from the given information.<n>We propose a clarification-aware RLVR reward that rewards explicit abstention and semantically aligned post-refusal clarification on unanswerable queries.<n>Using this reward, we train Abstain-R1, a 3B model that improves abstention and clarification on unanswerable queries while preserving strong performance on answerable ones.
• Score: 18.462748083613757
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement fine-tuning improves the reasoning ability of large language models, but it can also encourage them to answer unanswerable queries by guessing or hallucinating missing information. Existing abstention methods either train models to produce generic refusals or encourage follow-up clarifications without verifying whether those clarifications identify the key missing information. We study queries that are clear in meaning but cannot be reliably resolved from the given information, and argue that a reliable model should not only abstain, but also explain what is missing. We propose a clarification-aware RLVR reward that, while rewarding correct answers on answerable queries, jointly optimizes explicit abstention and semantically aligned post-refusal clarification on unanswerable queries. Using this reward, we train Abstain-R1, a 3B model that improves abstention and clarification on unanswerable queries while preserving strong performance on answerable ones. Experiments on Abstain-Test, Abstain-QA, and SelfAware show that Abstain-R1 substantially improves over its base model and achieves unanswerable-query behavior competitive with larger systems including DeepSeek-R1, suggesting that calibrated abstention and clarification can be learned through verifiable rewards rather than emerging from scale alone.

Related papers

• To Answer or to Abstain: Mitigating Search-Agent Hallucinations via Abstention-Aware Reinforcement Learning [20.267226736362158]
  AWA-RL dynamically shapes the abstention reward using the model's query-specific prior capabilities.<n>AWA-RL boosts absolute precision by up to 10.3% and overall RA-F1 by 2.9%, with only marginal sacrifice in raw accuracy.<n>Results confirm that AWA-RL successfully yields highly capable and reliable search agents.
  arXiv  Detail & Related papers  (2026-07-12T12:43:12Z)
• Bridging the Detection-to-Abstention Gap in Reasoning Models under Insufficient Information [24.973717050514207]
  We formalize the detection-to-abstention gap, where detected insufficiency fails to translate into final abstention.<n>This gap is especially concerning in high-risk domains such as medical AI, where incomplete evidence can be more harmful than refusal.<n>We propose Judge-Then-Then (JTS), a trajectory-level reasoning-control framework that trains models to make an explicit answerability commitment.
  arXiv  Detail & Related papers  (2026-05-27T07:28:25Z)
• Knowing When Not to Answer: Evaluating Abstention in Multimodal Reasoning Systems [52.83669998269706]
  Abstention has been studied in text-only settings but remains underexplored multimodally.<n>Current benchmarks either ignore unanswerability or rely on coarse methods that miss realistic failure modes.<n>We introduce MM-AQA, a benchmark that constructs unanswerable instances from answerable ones via transformations along two axes.
  arXiv  Detail & Related papers  (2026-04-16T09:23:22Z)
• PassiveQA: A Three-Action Framework for Epistemically Calibrated Question Answering via Supervised Finetuning [0.0]
  We study decision-aware query resolution under incomplete information, where a model must determine whether to Answer, Ask for clarification, or Abstain.<n>We propose PassiveQA, a three-action framework that aligns model behaviour with information sufficiency through supervised finetuning.<n>Our approach integrates structured information-state representations, knowledge graph-grounded context, and a finetuned planner that explicitly models missing variables and decision reasoning.
  arXiv  Detail & Related papers  (2026-04-06T09:54:16Z)
• I-CALM: Incentivizing Confidence-Aware Abstention for LLM Hallucination Mitigation [2.688126466709795]
  Large language models (LLMs) frequently produce confident but incorrect answers.<n>We study whether prompt-only interventions can reduce hallucination risk without modifying the model.<n>Results show the framework can improve selective answering on factual questions without retraining.
  arXiv  Detail & Related papers  (2026-04-05T00:15:41Z)
• Answering the Wrong Question: Reasoning Trace Inversion for Abstention in LLMs [24.31388053995892]
  Reasoning models have been shown to have worse abstention abilities.<n>Hallucinations resulting in failed abstention can be reinterpreted as LLMs answering the wrong question.<n>We develop a new class of state-of-the-art abstention methods called Trace Inversion.
  arXiv  Detail & Related papers  (2026-04-02T16:23:46Z)
• APR: Penalizing Structural Redundancy in Large Reasoning Models via Anchor-based Process Rewards [61.52322047892064]
  Test-Time Scaling (TTS) has significantly enhanced the capabilities of Large Reasoning Models (LRMs)<n>We observe that LRMs frequently conduct repetitive self-verification without revision even after obtaining the final answer during the reasoning process.<n>We propose Anchor-based Process Reward (APR), a structure-aware reward shaping method that localizes the reasoning anchor and penalizes exclusively the post-anchor AST.
  arXiv  Detail & Related papers  (2026-01-31T14:53:20Z)
• Rewarding Intellectual Humility Learning When Not To Answer In Large Language Models [0.0]
  Large Language Models (LLMs) often produce hallucinated or unverifiable content, undermining their reliability in factual domains.<n>This work investigates Reinforcement Learning with Verifiable Rewards (RLVR) as a training paradigm that explicitly rewards abstention alongside correctness to promote intellectual humility.
  arXiv  Detail & Related papers  (2026-01-27T23:42:07Z)
• From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generation [52.62655622099456]
  We propose reinforcement learning with verifiable reference-based rewards (RLVRR)<n>Instead of checking the final answer, RLVRR extracts an ordered linguistic signal from high-quality references (i.e., reward chain)<n>In this way, RLVRR decomposes rewards into two dimensions: content, which preserves deterministic core concepts, and style, which evaluates adherence to stylistic properties.
  arXiv  Detail & Related papers  (2026-01-26T14:39:58Z)
• Critique to Verify: Accurate and Honest Test-Time Scaling with RL-Trained Verifiers [63.99316853136304]
  Mirror-Critique is a framework that trains a verifier with informative critiques.<n>We deploy a small instruction-tuned model to synthesize high-quality critique data.<n>The resulting Mirror-Verifier is deployed to evaluate candidate solutions by generating multiple critiques per solution.
  arXiv  Detail & Related papers  (2025-09-27T06:50:24Z)
• RULE: Reinforcement UnLEarning Achieves Forget-Retain Pareto Optimality [24.299312059430704]
  Unlearning is the task of selectively removing specific information from a model without retraining from scratch or degrading overall utility.<n>Existing methods often rely on large-scale forget and retain datasets, and suffer from unnatural responses, poor generalization, or catastrophic utility loss.<n>We propose Reinforcement UnLearning (RULE), an efficient framework that formulates unlearning as a refusal boundary optimization problem.
  arXiv  Detail & Related papers  (2025-06-08T14:38:39Z)
• GRAIT: Gradient-Driven Refusal-Aware Instruction Tuning for Effective Hallucination Mitigation [62.63014905981601]
  Refusal-Aware Instruction Tuning (RAIT) aims to enhance Large Language Models (LLMs) by improving their ability to refuse responses to questions beyond their knowledge.<n>Effective RAIT must address two key challenges: firstly, effectively reject unknown questions to minimize hallucinations; secondly, avoid over-refusal to ensure questions that can be correctly answered are not rejected.<n>GraIT employs gradient-driven sample selection to effectively minimize hallucinations and (2) introduces an adaptive weighting mechanism during fine-tuning to reduce the risk of over-refusal.
  arXiv  Detail & Related papers  (2025-02-09T14:11:30Z)
• R-Tuning: Instructing Large Language Models to Say `I Don't Know' [66.11375475253007]
  Large language models (LLMs) have revolutionized numerous domains with their impressive performance but still face their challenges. Previous instruction tuning methods force the model to complete a sentence no matter whether the model knows the knowledge or not. We present a new approach called Refusal-Aware Instruction Tuning (R-Tuning) Experimental results demonstrate R-Tuning effectively improves a model's ability to answer known questions and refrain from answering unknown questions.
  arXiv  Detail & Related papers  (2023-11-16T08:45:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.