What Makes a Medical Checker Trainable? Diagnosing Signal Collapse and Reward Hacking in Checker-Guided RAG for Biomedical QA

• URL: http://arxiv.org/abs/2605.25988v1
• Date: Mon, 25 May 2026 16:06:06 GMT
• Title: What Makes a Medical Checker Trainable? Diagnosing Signal Collapse and Reward Hacking in Checker-Guided RAG for Biomedical QA
• Abstract summary: Medical RAG needs evidence-grounded claims, so plugging a claim-level NLI checker into retrieval-augmented RL is intuitive.<n>We compare four NLI checker back-ends as process rewards inside a GRPO-trained medical RAG agent.
• Score: 6.163604976615509
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Medical RAG needs evidence-grounded claims, so plugging a claim-level NLI checker into retrieval-augmented RL is intuitive. \textbf{We find that the checker's \emph{output distribution} during training, not its held-out accuracy, decides whether it provides trainable gradient.} We compare four NLI checker back-ends as process rewards inside a GRPO-trained medical RAG agent (Qwen2.5-7B, replicated on Qwen3-4B and Llama-3.1-8B) across four held-out medical QA benchmarks. Three diagnostic findings emerge. \textbf{(i)} Signal collapse is log-prob-specific: LLM log-probability scoring labels over 97\% of claims neutral -- collapsing the RL gradient to zero -- while a calibrated MedNLI classifier scores the same pairs non-degenerately. \textbf{(ii)} Moderate signal beats strong signal on answer quality: a strong proprietary checker triggers a three-step reward-hacking cascade -- ultra-short answers, search avoidance, language collapse -- so a moderate-signal local classifier trains a higher-quality model (\textbf{+12\% BERTScore over zero-shot, no GPT dependency}). \textbf{(iii)} Signal strength is policy-dependent: the same checker registers as moderate on one policy but strong on another without triggering the cascade end-state. We frame these as boundary conditions for verifier-as-reward systems.

Related papers

• OrchestraBench: Evaluating Multi-Agent Orchestration Failure Modes, Recovery, and Decomposition Quality [0.0]
  OrchestraBench evaluates failure, recovery, and decomposition through a controlled, seed-reproducible failure-injection harness over templated enterprise.<n>It introduces cascade radius and per-failure-mode recovery as primary metrics and compares routing policies with bootstrap confidence intervals and paired tests.
  arXiv  Detail & Related papers  (2026-08-05T17:27:15Z)
• When RLVR Shrinks the Reasoning Boundary: Diagnosing Pass@k Inversion [0.0]
  Reinforcement learning with verifiable rewards (RLVR) can improve one-sample accuracy while making a model worse under repeated sampling.<n>We study this pass@k inversion: after training, the policy may solve fewer distinct problems than its base model at large $k$.<n>The failure concentrates on boundary prompts, where the base model contains rare correct trajectories that are recoverable by sampling but too sparse to reliably appear in finite RLVR rollout groups.
  arXiv  Detail & Related papers  (2026-07-12T17:17:18Z)
• Detecting Hallucinations in Retrieval-Augmented Generation through Grounding-Aware Sensitivity by Perturbation (GASP) [1.827510863075184]
  Grounding-Aware Sensitivity by Perturbation (GASP) is a span-level detector that scores each answer sentence by how strongly its likelihood depends on the retrieved evidence.<n>GASP holds the answer fixed and re-scores it under the full context, under no context, and with each chunk removed, then measures the log-likelihood drops and Jensen-Shannon divergences.<n>We evaluate GASP on three benchmarks (RAGTruth, TofuEval, RAGBench) with three instruction-tuned scorers from two model families.
  arXiv  Detail & Related papers  (2026-07-05T10:35:30Z)
• Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifiers [25.040339775169883]
  We propose a verifier-free training method in which the base policy, conditioned on the rubric, serves as the teacher for the unconditioned student.<n>Across Qwen-2.5 (3B, 7B) and Qwen3-Thinking (4B, 8B) models on medical and science domains, RGSD achieves satisfaction comparable to judge-based rubricO.
  arXiv  Detail & Related papers  (2026-06-10T17:53:19Z)
• Auditing CoT Answer-Hijack Patches: Source-Control Certificates with Type-I Guarantees [0.0]
  Chain-of-thought (CoT) answer-hijack templates can flip the final numeric answer of a 7B-8B language model on GSM8K or MATH-500.<n>We show that a successful clean-source patch is often read as evidence that the patched activation carries the recovered content.<n>We construct an audit that turns each candidate patch into a source-control certificate with a pre-registered Type-I guarantee.
  arXiv  Detail & Related papers  (2026-06-03T10:49:43Z)
• What Are We Actually Decoding? Source Attribution for Non-Invasive Brain-to-Language Retrieval [42.66754319854329]
  We recast stimulus-locked MEG-to-audio retrieval as an auditing framework.<n>We use structural shortcuts, window-level stimulus-locked evidence, and cross-window contextual aggregation.<n>These results suggest that brain-to-language performance should be source-attributed, not merely reported.
  arXiv  Detail & Related papers  (2026-05-23T11:23:39Z)
• Needle-in-RAG: Prompt-Conditioned Character-Level Traceback of Poisoned Spans in Retrieved Evidence [4.039934762896615]
  RAGCharacter is a two-pass forensic framework that localizes the responsible retrieved span for a concrete misgeneration event.<n>We study black-box character-level poison traceback in RAG and present RAGCharacter, a two-pass forensic framework that localizes the responsible retrieved span for a concrete misgeneration event.
  arXiv  Detail & Related papers  (2026-05-03T08:42:29Z)
• What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time [57.533031432715084]
  Test-Time Reinforcement Learning (TTRL) enables Large Language Models (LLMs) to enhance reasoning capabilities on unlabeled test streams.<n>Existing TTRL methods rely exclusively on positive pseudo-labeling strategies.<n>We propose SCRL, a robust test-time reinforcement learning framework that effectively mitigates label noise amplification.
  arXiv  Detail & Related papers  (2026-03-20T11:47:12Z)
• Reward Under Attack: Analyzing the Robustness and Hackability of Process Reward Models [68.45272703833209]
  We show that state-of-the-art PRMs are systematically exploitable under adversarial optimization pressure.<n>We introduce a three-tiered diagnostic framework that applies increasing adversarial pressure to quantify these vulnerabilities.<n>We release PRM-BiasBench and a diagnostic toolkit to enable robustness evaluation before deployment.
  arXiv  Detail & Related papers  (2026-02-20T23:38:03Z)
• SpatialBench-UC: Uncertainty-Aware Evaluation of Spatial Prompt Following in Text-to-Image Generation [0.0]
  SpatialBench-UC is a small, reproducible benchmark for pairwise spatial relations.<n>We release a benchmark package, versioned prompts, pinned configs, per-sample checker outputs, and report tables.<n>We evaluate three baselines, Stable Diffusion 1.5, SD 1.5 BoxDiff, and SD 1.4 GLIGEN.
  arXiv  Detail & Related papers  (2026-01-19T23:37:10Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers [90.50039419576807]
  Reinforcement Learning with Verifiable Rewards (RLVR) trains policies against automated verifiers to avoid costly human labeling.<n>To reduce vulnerability to verifier hacking, many RLVR systems collapse rewards to binary $0,1$ during training.<n>This choice carries a cost: it introduces textitfalse negatives (rejecting correct answers, FNs) and textitfalse positives (accepting incorrect ones, FPs)
  arXiv  Detail & Related papers  (2025-10-01T13:56:44Z)
• Reference-Free Rating of LLM Responses via Latent Information [53.463883683503106]
  We study the common practice of asking a judge model to assign Likert-scale scores to free-text responses.<n>We then propose and evaluate Latent Judges, which derive scalar ratings from internal model signals.<n>Across a broad suite of pairwise and single-rating benchmarks, latent methods match or surpass standard prompting.
  arXiv  Detail & Related papers  (2025-09-29T12:15:52Z)
• Unsupervised Conformal Inference: Bootstrapping and Alignment to Control LLM Uncertainty [49.19257648205146]
  We propose an unsupervised conformal inference framework for generation.<n>Our gates achieve close-to-nominal coverage and provide tighter, more stable thresholds than split UCP.<n>The result is a label-free, API-compatible gate for test-time filtering.
  arXiv  Detail & Related papers  (2025-09-26T23:40:47Z)
• Deep Omni-supervised Learning for Rib Fracture Detection from Chest Radiology Images [41.62893318123283]
  Deep learning (DL)-based rib fracture detection has shown promise of playing an important role in preventing mortality and improving patient outcome. DL-based object detection models requires a huge amount of bounding box annotation. Annotating medical data is time-consuming and expertise-demanding, making obtaining a large amount of fine-grained annotations extremely infeasible. We present a novel omni-supervised object detection network, ORF-Netv2, to leverage as much available supervision as possible.
  arXiv  Detail & Related papers  (2023-06-23T05:36:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.