Drop the Act: Probe-Filtered RL for Faithful Chain-of-Thought Reasoning

• URL: http://arxiv.org/abs/2605.11467v1
• Date: Tue, 12 May 2026 03:30:23 GMT
• Title: Drop the Act: Probe-Filtered RL for Faithful Chain-of-Thought Reasoning
• Abstract summary: Reasoning models post-hoc rationalize answers they have already committed to internally.<n>ProFIL reduces post-commitment theater by **11--100%**, raises faithful-fraction (e.g., +24pp on LiveCodeBench under an independent Claude 3.7 Sonnet judge)<n>ProFIL also beats a matched length-penalty GRPO baseline, isolating the gain as semantic commitment-detection rather than chain compression.
• Score: 1.3011345529764784
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reasoning models post-hoc rationalize answers they have already committed to internally, producing chains of *reasoning theater*: deliberative-looking steps that contribute nothing to correctness. This wastes inference tokens, pollutes interpretability, and obscures what the model actually computed. We introduce **ProFIL** (**Pro**be-**Fil**tered Reinforcement Learning) to *reduce theater, increase chain-of-thought faithfulness, and shrink chain length* in a single, drop-in extension to Group Relative Policy Optimization (GRPO). A multi-head attention probe is trained *once* on the *frozen* base model to detect post-commitment steps from internal activations alone; during GRPO, rollouts whose probe score exceeds a threshold have their advantage zeroed. *Our central finding is that a probe trained on a frozen base, with verifier-derived labels and no human annotation, provides a stable signal that suppresses theater while resisting the RL-obfuscation failure mode predicted by prior work.* Across four reasoning domains (GSM8K, LiveCodeBench, ToolUse, MMLU-Redux) and two model architectures (Llama-8B, Qwen-7B), ProFIL reduces post-commitment theater by **11--100%**, raises faithful-fraction (e.g., +24pp on LiveCodeBench under an independent Claude 3.7 Sonnet judge), and shortens chains by 4--19%, all while preserving or improving task accuracy. ProFIL also beats a matched length-penalty GRPO baseline, isolating the gain as semantic commitment-detection rather than chain compression. Probe weights, training configurations, and rollouts are released across all four domains.

Related papers

• Explore More, Drift Less: Outcome-Only Reinforcement Learning Can Suffice for Long-Horizon Interactive Agents [3.3656546156010942]
  Reinforcement learning is a natural way to post-train agents for long-horizon interactive tasks judged only by end-of-task verification.<n>Recent work compensates around the training with denser rewards, SFT priors, skill libraries, curated memory, or multi-agent orchestration.<n>We present CANOPY, a minimalist protocol attacking both directly: scale same-task exploration until the natural signal reappears, keep every update on-policy, KL-anchored, and confined to the agent's own action tokens.
  arXiv  Detail & Related papers  (2026-09-01T13:44:40Z)
• Training-Free Inference-Time Self-Reflection and Cost-Bounded Early Stopping for Large Language Models [15.961083103517872]
  EvoResearcher is a training-free, inference-time protocol that adds cost-bounded self-reflection to a single frozen LLM backbone.<n>Four self-reflective meta-reward components act as design principles instantiated as prompt-level mechanisms.
  arXiv  Detail & Related papers  (2026-08-19T13:09:57Z)
• Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL [55.05743310621117]
  Co-RL is a framework in which multiple decoupled models, sharing no parameters, are simultaneously optimized through RL using rewards derived from their peers.<n>We show that increasing cohort diversity, through heterogeneous model families, sizes, and rephrased training samples, reduces the correlated errors that drive self-reinforcing feedback loops.
  arXiv  Detail & Related papers  (2026-08-18T01:16:02Z)
• Doomed from the Start: Early Abort of LLM Agent Episodes via a Recall-Controlled Probe Cascade [14.63102636181275]
  Large language model (LLM) agents often waste inference compute by continuing multi-step trajectories that are already doomed to fail.<n>We study early failure prediction and inference-time early stopping for LLM agents using hidden-state probes.<n>We turn this signal into a recall-controlled abort cascade for reducing LLM agent inference costs.
  arXiv  Detail & Related papers  (2026-07-07T17:03:55Z)
• Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning [55.264863369127774]
  Current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states.<n>We propose Hidden-Align, an auxiliary loss function that aligns the last-layer hidden states of correct rollouts at the anchor token during RL training, with zero overhead in both training and inference.<n>On eight mathematical reasoning benchmarks, Hidden-Align improves average pass@1 over the DAPO baseline by 3.8, 6.2, and 5.4 percentage points on Qwen3-1.7B, 4B, and 14B respectively, with consistent pass@k gains across all three scales.
  arXiv  Detail & Related papers  (2026-06-02T06:51:15Z)
• The Chain Holds, the Answer Folds: Trace-Answer Dissociation in Reasoning Models Under Adversarial Pressure [8.86745721473138]
  Reasoning models are evaluated on single-turn benchmarks but deployed in multi-turn dialogue, where users push back on correct answers.<n>We call this unfaithful capitulation (UC) and isolate it with a $2times 2$ latent-versus-behavioral framework that flip-rate metrics and single-turn probes both miss.
  arXiv  Detail & Related papers  (2026-05-27T20:41:08Z)
• Teaching LLMs Program Semantics via Symbolic Execution Traces [0.7046782561282057]
  We introduce an evaluation framework of 500 C verification tasks built on SV-COMP 2025.<n>We evaluate 14 models across six families and find high overall accuracy masks a critical weakness.<n>Just $sim$3,000 bug traces combined with chain-of-thought reasoning at inference time improve violation detection by over 17 percentage points.
  arXiv  Detail & Related papers  (2026-05-07T13:01:06Z)
• CoVerRL: Breaking the Consensus Trap in Label-Free Reasoning via Generator-Verifier Co-Evolution [52.691495954442985]
  CoVerRL is a framework where a single model alternates between generator and verifier roles, with each capability bootstrapping the other.<n> Experiments across Qwen and Llama model families demonstrate that CoVerRL outperforms label-free baselines by 4.7-5.9% on mathematical reasoning benchmarks.<n>Self-verification accuracy improves from around 55% to over 85%, confirming that both capabilities genuinely co-evolve.
  arXiv  Detail & Related papers  (2026-03-18T14:38:55Z)
• Reinforcement Inference: Leveraging Uncertainty for Self-Correcting Language Model Reasoning [0.0]
  Reinforcement Inference uses the model's own uncertainty to selectively invoke a second, more deliberate reasoning attempt.<n>On 12,032 MMLU-Pro questions across 14 subjects, using DeepSeek-v3.2 with deterministic decoding in a zero-shot setting, Reinforcement Inference improves accuracy from 60.72% to 84.03%.
  arXiv  Detail & Related papers  (2026-02-09T11:08:24Z)
• Are Reasoning LLMs Robust to Interventions on Their Chain-of-Thought? [79.86483056611105]
  Reasoning LLMs generate step-by-step chains of thought before giving an answer.<n>How robust are these reasoning traces to disruptions that occur within them?<n>We introduce a controlled evaluation framework that perturbs a model's own CoT at fixed timesteps.
  arXiv  Detail & Related papers  (2026-02-07T10:02:58Z)
• Stop Rewarding Hallucinated Steps: Faithfulness-Aware Step-Level Reinforcement Learning for Small Reasoning Models [59.6715047267181]
  Small reasoning models (SRMs) are prone to hallucinations, especially in intermediate reasoning steps.<n>Existing mitigation methods based on online reinforcement learning rely on outcome-based rewards or coarse-grained chain-of-thought evaluation.<n>We propose Faithfulness-Aware Step-Level Reinforcement Learning (FaithRL), introducing step-level supervision via explicit faithfulness rewards from a process reward model.
  arXiv  Detail & Related papers  (2026-02-05T17:15:12Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• LaSeR: Reinforcement Learning with Last-Token Self-Rewarding [54.72617309922891]
  Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a core paradigm for enhancing the reasoning capabilities of Large Language Models (LLMs)<n>Previous practice requires the LLM to sequentially generate solutions and self-verifications using two separate prompt templates, which significantly reduces efficiency.<n>We propose LaSeR (Reinforcement Learning with Last-Token Self-Rewarding), an algorithm that simply augments the original RLVR loss with a MSE loss.
  arXiv  Detail & Related papers  (2025-10-16T17:55:11Z)
• RESTRAIN: From Spurious Votes to Signals -- Self-Driven RL with Self-Penalization [52.01526898310723]
  We introduce RESTRAIN, a self-penalizing RL framework that converts the absence of gold labels into a useful learning signal.<n>Instead of overcommitting to spurious majority votes, RESTRAIN exploits signals from the model's entire answer distribution.<n>On challenging reasoning benchmarks, RESTRAIN delivers large gains using only unlabeled data.
  arXiv  Detail & Related papers  (2025-10-02T16:24:01Z)
• Mitigating Deceptive Alignment via Self-Monitoring [15.365589693661823]
  We develop a framework that embeds a Self-Monitor inside the chain-of-thought process itself, named CoT Monitor+.<n>During generation, the model produces (i) ordinary reasoning steps and (ii) an internal self-evaluation signal trained to flag and suppress misaligned strategies.<n>The signal is used as an auxiliary reward in reinforcement learning, creating a feedback loop that rewards honest reasoning and discourages hidden goals.
  arXiv  Detail & Related papers  (2025-05-24T17:41:47Z)
• Stable Reinforcement Learning for Efficient Reasoning [2.838966689544288]
  GRPO-$lambda$ is an efficient and stabilized variant of GRPO.<n>It dynamically adjusts the reward strategy by monitoring the correctness ratio.<n>It improves average accuracy by 1.48% while reducing CoT sequence length by 47.3%.
  arXiv  Detail & Related papers  (2025-05-23T16:43:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.