Reinforcement Learning for Large Language Model Selective Evidence Adoption from Contaminated Retrieval Results

• URL: http://arxiv.org/abs/2607.20090v1
• Date: Wed, 22 Jul 2026 12:45:04 GMT
• Title: Reinforcement Learning for Large Language Model Selective Evidence Adoption from Contaminated Retrieval Results
• Abstract summary: Retrieval-augmented large language models face contexts that interleave useful evidence with misleading statements or instruction-like content.<n>We introduce SelectBench, a benchmark and training set for selective evidence adoption, and post-train Qwen3.5-4B directly with DAPO.<n>On the corrected 325-example SelectBench-v2 test set, strict success rises from 22.46% for the original checkpoint to 25.54% with DAPO-Rule and 26.46% with DAPO-DeepSeek.
• Score: 3.9964880913101677
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Retrieval-augmented large language models frequently face contexts that interleave useful evidence with misleading statements or instruction-like content. Blanket refusal discards valid evidence, whereas uncritical adoption yields incorrect or unsafe answers. The ability to selectively adopt relevant information while rejecting deceptive or harmful content is therefore critical for reliable deployment in real-world retrieval settings. We introduce SelectBench, a controlled benchmark and training set for selective evidence adoption, and post-train Qwen3.5-4B directly with DAPO using either deterministic rule rewards or a frozen semantic judge. On the corrected 325-example SelectBench-v2 test set, strict success rises from 22.46% for the original checkpoint to 25.54% with DAPO-Rule and 26.46% with DAPO-DeepSeek. Both trained policies reduce forbidden-content adoption and produce shorter, more focused responses, yet prompt-injection following does not improve. The paired gains are modest and fail to survive Holm correction, suggesting that stronger reward shaping or additional training iterations may be needed for more robust gains. DAPO-DeepSeek exhibits no material degradation on MMLU or clean HotpotQA, indicating that the post-training procedure preserves general capabilities. These results demonstrate a directional improvement in selective evidence use, while identifying injection resistance and statistical robustness as important remaining challenges for future work.

Related papers

• AERA: Adaptive Evidence Residual Allocation for Efficient Test-Time Reasoning [22.964476507912327]
  Test-time scaling improves language-model reasoning by generating additional candidate solutions, but allocating the same inference budget to every problem is computationally wasteful.<n>We show that checkpoint-level correctness evolves non-monotonically, and observable evidence may strengthen before an answer collapses or weaken before it recovers.<n>We introduce Adaptive Evidence Residual Allocation (AERA), a sequential controller that learns whether additional computation is likely to recover a better answer from checkpoint-observable evidence.
  arXiv  Detail & Related papers  (2026-08-28T06:21:07Z)
• TTPO: Test-Time Policy Optimization [53.81524570216593]
  Test-Time Policy Optimization (TTPO) is an asymmetric objective that distills agreeing rollouts via OPSD and penalizes disagreeing rollouts with Grouped RL.<n>We show that TTPO matches label-supervised OPSD on five competition-level benchmarks, raises Qwen3-1.7B from 38.0% to 45.2% in TTT, yields +25.2% to +36.4% without thinking, and shows strong cross-task generalization.
  arXiv  Detail & Related papers  (2026-08-27T17:58:10Z)
• Learning What to Fail On: Failure-Mode Contextual Bandits for Adversarial Data Curation [14.914697247710622]
  We introduce a failure-aware adversarial retrieval-augmented framework for improving robustness in natural language understanding.<n>Our approach improves RoBERTa-base accuracy from 88.48% to 92.60% on SNLI, from 75.04% to 80.95% on ANLI, and from 54.67% to 71.99% on MultiNLI.<n>We demonstrate transfer to FEVER fact verification, achieving up to 79.86% FEVER score and 82.45% accuracy with RoBERTa-large.
  arXiv  Detail & Related papers  (2026-08-19T08:31:15Z)
• An Empirical Study of Reward Specification and Benchmark Reliability in GRPO-based LLM Unlearning [0.0]
  Unlearning is evaluated through two objectives: suppress target-specific knowledge and preserve non-target utility.<n>In generative QA, this leaves a third behavior underspecified: when a target-adjacent prompt admits a broader answer without target-specific leakage, the model should answer at that level rather than leak, evade, or refuse.<n>We compare four reward designs that span lexical suppression, anti-refusal shaping, rubric-based broad answering, and an explicit refusal contrast, with and without SFT warm-up.
  arXiv  Detail & Related papers  (2026-08-18T14:04:29Z)
• EvidentialRAG: Quantifying and Mitigating Information Conflict in Multi-Source Retrieval-Augmented Generation via Evidential Deep Learning [0.2730969268472861]
  This paper introduces ERAG, an uncertainty-aware RAG framework that converts retrieved chunks into probabilistic evidence before generation.<n>Experiments on CRAG, ConflictQA, and MuSiQue show that ERAG remains competitive with the strongest matched baseline on standard question answering.
  arXiv  Detail & Related papers  (2026-07-11T21:56:15Z)
• When Does Learning to Stop Help? A Cost-Aware Study of Early Exits in Reasoning Models [4.8190992438931035]
  LearnStop is a hidden-state-free logistic stopper over prefix-observable features.<n>A trajectory decomposition predicts the regime: learning pays where answers oscillate and correctness evidence is spread.<n>Cost accounting sharpens the picture further -- the same policy that saves 32% of tokens under KV-cache forking costs 121% extra under black-box repeated prefilling.
  arXiv  Detail & Related papers  (2026-06-29T19:33:42Z)
• Exploring Agentic Tool-Calling Decisions via Uncertainty-Aligned Reinforcement Learning [63.24624171679711]
  Large language model (LLM)-based agents often make suboptimal tool-use decisions.<n>We propose TRUST, which incorporates uncertainty quantification into reward design as a repulsive force for maintaining uncertainty separation.
  arXiv  Detail & Related papers  (2026-06-05T07:08:34Z)
• CEPO: RLVR Self-Distillation using Contrastive Evidence Policy Optimization [50.59956036193097]
  A model produces a correct solution under reinforcement learning with verifiable rewards (RLVR)<n>Every token receives the same reward signal regardless of whether it was a decisive reasoning step or a grammatical filler.<n>We propose Contrastive Evidence Policy Optimization (CEPO)<n>CEPO asks a sharper question at every token: "does the correct answer favor this token?" but "does the correct answer favor it while the wrong answer disfavors it?"
  arXiv  Detail & Related papers  (2026-05-19T06:46:19Z)
• Correct Answers from Sound Reasoning: Verifiable Process Supervision for Language Models [94.68358825189738]
  We propose a post-training framework for verifiable domains that jointly optimize prediction accuracy and reasoning quality.<n>We evaluate VPS on chess, a controlled testbed where reasoning steps can be deterministically verified against engine signals.<n>VPS preserves accuracy while significantly improving reasoning quality, reducing win-rate error by up to 30% and restoring consistency to near saturation.
  arXiv  Detail & Related papers  (2026-04-03T15:19:46Z)
• Reinforcement Inference: Leveraging Uncertainty for Self-Correcting Language Model Reasoning [0.0]
  Reinforcement Inference uses the model's own uncertainty to selectively invoke a second, more deliberate reasoning attempt.<n>On 12,032 MMLU-Pro questions across 14 subjects, using DeepSeek-v3.2 with deterministic decoding in a zero-shot setting, Reinforcement Inference improves accuracy from 60.72% to 84.03%.
  arXiv  Detail & Related papers  (2026-02-09T11:08:24Z)
• GRACE: Reinforcement Learning for Grounded Response and Abstention under Contextual Evidence [9.80421132842862]
  Retrieval-Augmented Generation (RAG) integrates external knowledge to enhance Large Language Models (LLMs)<n>RAG is susceptible to two critical flaws: providing correct answers without explicit grounded evidence and producing fabricated responses when the retrieved context is insufficient.<n>We propose GRACE, a reinforcement-learning framework that simultaneously mitigates both types of flaws.
  arXiv  Detail & Related papers  (2026-01-08T02:47:33Z)
• Knowing the Answer Isn't Enough: Fixing Reasoning Path Failures in LVLMs [85.37131922131657]
  We reveal a critical yet underexplored flaw in Large Vision-Language Models (LVLMs)<n>Even when these models know the correct answer, they frequently arrive there through incorrect reasoning paths.<n>We propose PSO (Path-Select Optimization), a two-stage post-training framework designed to enhance both the reasoning performance and stability of existing LVLMs.
  arXiv  Detail & Related papers  (2025-12-06T03:02:55Z)
• RefusalBench: Generative Evaluation of Selective Refusal in Grounded Language Models [43.76961935990733]
  The ability of language models to refuse to answer based on flawed systems remains a significant failure point.<n>We introduce RefusalBench, a generative methodology that creates diagnostic test cases through controlled linguistic context.<n>We find that selective refusal is a train, alignmentsensitive capability offering a clear path to improvement.
  arXiv  Detail & Related papers  (2025-10-12T00:53:42Z)
• Uncertainty-Penalized Direct Preference Optimization [52.387088396044206]
  We develop a pessimistic framework for DPO by introducing preference uncertainty penalization schemes. The penalization serves as a correction to the loss which attenuates the loss gradient for uncertain samples. We show improved overall performance compared to vanilla DPO, as well as better completions on prompts from high-uncertainty chosen/rejected responses.
  arXiv  Detail & Related papers  (2024-10-26T14:24:37Z)
• Improving Selective Visual Question Answering by Learning from Your Peers [74.20167944693424]
  Visual Question Answering (VQA) models can have difficulties abstaining from answering when they are wrong. We propose Learning from Your Peers (LYP) approach for training multimodal selection functions for making abstention decisions. Our approach uses predictions from models trained on distinct subsets of the training data as targets for optimizing a Selective VQA model.
  arXiv  Detail & Related papers  (2023-06-14T21:22:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.