A Pre-Registered Causal Partition of Self-Consistency Elicitation and Reward Design in RLVR

• URL: http://arxiv.org/abs/2606.05932v2
• Date: Mon, 08 Jun 2026 06:52:09 GMT
• Title: A Pre-Registered Causal Partition of Self-Consistency Elicitation and Reward Design in RLVR
• Abstract summary: Reinforcement learning from rewards improves reasoning even when the reward signal is spurious.<n> Practitioners commonly interpret naive = acc(TRUE) - acc(R) as the reward-design effect.<n>We prove this estimand is systematically biased.
• Score: 1.2958054117511815
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Reinforcement learning from verifiable rewards (RLVR) improves reasoning even when the reward signal is spurious -- assigning credit to the group-plurality answer rather than a ground-truth verifier. Practitioners commonly interpret naive = acc(TRUE) - acc(RANDOM) as the reward-design effect. We prove this estimand is systematically biased: it conflates self-consistency elicitation (sharpening the policy toward its modal answer via majority pseudo-reward) with genuine reward-design signal. Using a controlled tabular-GRPO simulator we derive an exact telescoping decomposition total = null + elicit + rd and measure each term across five prior-strength levels. The reward-design fraction of the naive estimator ranges from 0.139 at weak prior (ps=0.20) to 0.05 at strong prior (ps=0.80), with the elicitation term flipping sign at the self-consistency crossover. A pre-registered 2x2x2 factorial confirms non-additivity (interaction ratio 0.385; AxC effect -0.089). A points-vs-bounds pilot gate shows strong-prior regimes are point-identified while near-crossover regimes are only bounded. Re-audits of two named published results yield ELICITATION DOMINATED (elicitation share 0.98) and REWARD DESIGN DOMINATED (rd share 1.18) verdicts respectively, demonstrating the diagnostic value of the partition. We pre-commit to submit regardless of flip outcome; a non-flip is a finding of equal standing. We release a reusable one-command harness for any alignment paper to run the same audit.

Related papers

• Proof-Carrying Cognition: Closing the Verification Gap with Reality-Settled Reward [0.0]
  Frontier gains in language-model reasoning come from reinforcement learning on reasoning traces.<n>We argue the field's binding constraint is the verification gap.
  arXiv  Detail & Related papers  (2026-09-09T06:24:53Z)
• Accurate Ensembles, Fragile Narratives: Multi-Scale Stacking and a Fidelity Audit of LLM-Generated Explanations for Credit Risk [0.0]
  Credit scoring increasingly relies on models whose decision logic cannot be read off their parameters.<n>A common proposal closes that gap with a language model: compute feature attributions, hand them to an LLM, and let it write the rationale.<n>We build such a system end to end and test whether the second half of the promise holds.
  arXiv  Detail & Related papers  (2026-08-08T13:22:14Z)
• UpliftBench: Revealing Outcome-Regime and Objective Mismatch in Uplift Evaluation [0.0]
  UpliftBench evaluates 12 uplift estimators under an outer-test-isolated, multi-objective protocol.<n>We show the disagreement is substantially about metrics, not models.
  arXiv  Detail & Related papers  (2026-08-02T00:51:44Z)
• Verifier-Induced Support Reshaping in On-Policy Optimization [27.782412748608255]
  We show that on-policy reinforcement learning with verifiable rewards (RLVR) can improve the current objective while making successful behaviors for later objectives too rare to sample and reinforce.<n>We call this verifier-induced support reshaping and define effective rewardable support as successful trajectories reachable within a fixed rollout budget.
  arXiv  Detail & Related papers  (2026-07-31T19:05:43Z)
• SVR: Self-Verifying Refinement via Joint Verdict-Confidence Reinforcement Learning for Adaptive Test-Time Compute [62.3458279176813]
  Self-Verifying Refinement is an oracle-free multi-turn reinforcement learning framework.<n>It learns to use self-verification as a compute-control policy.<n>It achieves a macro-average accuracy of 0.563 with only 2.99 inference turns on average.
  arXiv  Detail & Related papers  (2026-07-30T16:20:58Z)
• CRAFT: Counterfactual Credit Assignment from Free Sibling Rollouts for Self-Distilled Agentic Reinforcement Learning [4.663685189987781]
  Self-distilled agentic reinforcement learning augments trajectory-level reward with a token-level distillation loss.<n>The prevailing recipe gates this loss by a single scalar, the teacher-student log-probability gap.<n>We introduce CRAFT, a threepillar credit-assignment scheme that addresses both limitations.
  arXiv  Detail & Related papers  (2026-06-28T16:11:47Z)
• Reasoning Arena: Trace Tournaments When Verifiable Rewards Fall Short [51.667769734342635]
  Reinforcement learning with verifiable rewards (RLVR) has become a leading paradigm for improving the reasoning ability of large language models.<n>We propose Reasoning Arena, an adaptive training framework that routes such non-diverse reward groups to a judge system instead of discarding them.<n>We show that Reasoning Arena consistently outperforms the RLVR baseline by 7.6% on average in competition mathematics and coding benchmarks.
  arXiv  Detail & Related papers  (2026-06-08T11:57:17Z)
• Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning [55.264863369127774]
  Current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states.<n>We propose Hidden-Align, an auxiliary loss function that aligns the last-layer hidden states of correct rollouts at the anchor token during RL training, with zero overhead in both training and inference.<n>On eight mathematical reasoning benchmarks, Hidden-Align improves average pass@1 over the DAPO baseline by 3.8, 6.2, and 5.4 percentage points on Qwen3-1.7B, 4B, and 14B respectively, with consistent pass@k gains across all three scales.
  arXiv  Detail & Related papers  (2026-06-02T06:51:15Z)
• The Cancellation Hypothesis in Critic-Free RL: From Outcome Rewards to Token Credits [56.379030343338776]
  We study criticfree RL from a token-level perspective, revealing the token-flipping phenomenon.<n>We show that a token's change in probability is not fully determined by its own advantage; coupled gradient interactions with other tokens also play a non-negligible role.
  arXiv  Detail & Related papers  (2026-05-09T04:07:20Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers [90.50039419576807]
  Reinforcement Learning with Verifiable Rewards (RLVR) trains policies against automated verifiers to avoid costly human labeling.<n>To reduce vulnerability to verifier hacking, many RLVR systems collapse rewards to binary $0,1$ during training.<n>This choice carries a cost: it introduces textitfalse negatives (rejecting correct answers, FNs) and textitfalse positives (accepting incorrect ones, FPs)
  arXiv  Detail & Related papers  (2025-10-01T13:56:44Z)
• Quantile Advantage Estimation for Entropy-Safe Reasoning [44.192277495613695]
  Reinforcement Learning with Verifiable Rewards (RLVR) strengthens LLM reasoning, but training often oscillates between entropy collapse and entropy explosion<n>We trace both hazards to the mean baseline used in value-free RL, which improperly penalizes negative-advantage samples under reward outliers.<n>We propose Quantile Advantage Estimation (QAE), replacing the mean with a group-wise K-quantile baseline.
  arXiv  Detail & Related papers  (2025-09-26T17:37:52Z)
• RLPR: Extrapolating RLVR to General Domains without Verifiers [103.14103272635893]
  We propose RLPR, a simple verifier-free framework that extrapolates RLVR to broader general domains.<n>We find that addressing the high variance of this noisy probability reward is crucial to make it work.<n>RLPR consistently improves reasoning capabilities in both areas for Gemma, Llama, and Qwen based models.
  arXiv  Detail & Related papers  (2025-06-23T02:56:36Z)
• TRUST-LAPSE: An Explainable and Actionable Mistrust Scoring Framework for Model Monitoring [4.262769931159288]
  We propose TRUST-LAPSE, a "mistrust" scoring framework for continuous model monitoring. We assess the trustworthiness of each input sample's model prediction using a sequence of latent-space embeddings. Our latent-space mistrust scores achieve state-of-the-art results with AUROCs of 84.1 (vision), 73.9 (audio), and 77.1 (clinical EEGs)
  arXiv  Detail & Related papers  (2022-07-22T18:32:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.