JURY-RL: Votes Propose, Proofs Dispose for Label-Free RLVR

• URL: http://arxiv.org/abs/2604.25419v1
• Date: Tue, 28 Apr 2026 09:29:00 GMT
• Title: JURY-RL: Votes Propose, Proofs Dispose for Label-Free RLVR
• Abstract summary: JURY-RL is a label-free RLVR framework that decouples answer proposal from reward disposal.<n>It consistently outperforms other label-free baselines on mathematical reasoning benchmarks.<n>It attains pass@1 performance comparable to supervised ground-truth training.
• Score: 39.03968285406107
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning with verifiable rewards (RLVR) enhances the reasoning of large language models (LLMs), but standard RLVR often depends on human-annotated answers or carefully curated reward specifications. In machine-checkable domains, label-free alternatives such as majority voting or LLM-as-a-judge remove annotation cost but can introduce false positives that destabilize training. We introduce JURY-RL, a label-free RLVR framework that decouples answer proposal from reward disposal: votes from model rollouts propose a candidate answer, and a formal verifier determines whether that candidate can receive positive reward. Concretely, only rollouts matching the plurality-voted answer are rewarded when that answer is successfully verified in Lean. When verification is inconclusive, we invoke ResZero (Residual-Zero), a fallback reward that discards the unverified plurality proposal and redistributes a zero-mean, variance-preserving signal over the residual answers. This design maintains a stable optimization gradient without reinforcing unverifiable consensus. Across three backbone models trained on mathematical data, JURY-RL consistently outperforms other label-free baselines on mathematical reasoning benchmarks and transfers competitively to code generation and general benchmarks. It attains pass@1 performance comparable to supervised ground-truth training, with superior generalization demonstrated by higher pass@k and response diversity.

Related papers

• Don't Peek at the Answer: Outcome-Masked Group Relative Policy Optimization for Label-Free RLVR [21.781083492293913]
  Voting-based label-free RLVR replace gold supervision with answer-level consensus from model samples.<n>We propose OM-GRPO, a label-free RLVR framework that decouples reward estimation from policy optimization.
  arXiv  Detail & Related papers  (2026-08-04T04:41:20Z)
• Soft-SVeRL: Self-Verified Reinforcement Learning with Soft Rewards [24.040621624203823]
  Reinforcement Learning from Verifiable Rewards (RLVR) has improved language models in domains such as mathematics and code.<n>We introduce Soft-RLVR, a framework for reinforcement learning from decomposed, learned verification signals.<n>We show that self-verification is prone to reward inflation from overly permissive self-judgments, and that explicit stabilization is needed to prevent this collapse.
  arXiv  Detail & Related papers  (2026-05-27T14:50:53Z)
• ResRL: Boosting LLM Reasoning via Negative Sample Projection Residual Reinforcement Learning [60.411097231348975]
  Reinforcement Learning with Verifiable Rewards (RLVR) enhances reasoning of Large Language Models (LLMs)<n>This paper proposes negative sample projection Residual Reinforcement Learning (ResRL) that decouples similar semantic distributions among positive and negative responses.
  arXiv  Detail & Related papers  (2026-05-01T03:57:44Z)
• From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generation [52.62655622099456]
  We propose reinforcement learning with verifiable reference-based rewards (RLVRR)<n>Instead of checking the final answer, RLVRR extracts an ordered linguistic signal from high-quality references (i.e., reward chain)<n>In this way, RLVRR decomposes rewards into two dimensions: content, which preserves deterministic core concepts, and style, which evaluates adherence to stylistic properties.
  arXiv  Detail & Related papers  (2026-01-26T14:39:58Z)
• Hybrid Reinforcement: When Reward Is Sparse, It's Better to Be Dense [36.71358559780692]
  HERO is a reinforcement learning framework that integrates verifier signals with reward-model scores in a structured way.<n> HERO consistently outperforms RM-only and verifier-only baselines, with strong gains on both verifiable and hard-to-verify tasks.
  arXiv  Detail & Related papers  (2025-10-08T17:09:41Z)
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers [90.50039419576807]
  Reinforcement Learning with Verifiable Rewards (RLVR) trains policies against automated verifiers to avoid costly human labeling.<n>To reduce vulnerability to verifier hacking, many RLVR systems collapse rewards to binary $0,1$ during training.<n>This choice carries a cost: it introduces textitfalse negatives (rejecting correct answers, FNs) and textitfalse positives (accepting incorrect ones, FPs)
  arXiv  Detail & Related papers  (2025-10-01T13:56:44Z)
• Critique to Verify: Accurate and Honest Test-Time Scaling with RL-Trained Verifiers [63.99316853136304]
  Mirror-Critique is a framework that trains a verifier with informative critiques.<n>We deploy a small instruction-tuned model to synthesize high-quality critique data.<n>The resulting Mirror-Verifier is deployed to evaluate candidate solutions by generating multiple critiques per solution.
  arXiv  Detail & Related papers  (2025-09-27T06:50:24Z)
• RLPR: Extrapolating RLVR to General Domains without Verifiers [103.14103272635893]
  We propose RLPR, a simple verifier-free framework that extrapolates RLVR to broader general domains.<n>We find that addressing the high variance of this noisy probability reward is crucial to make it work.<n>RLPR consistently improves reasoning capabilities in both areas for Gemma, Llama, and Qwen based models.
  arXiv  Detail & Related papers  (2025-06-23T02:56:36Z)
• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs [35.27561531876348]
  This paper systematically investigates the impact of Reinforcement Learning with Verifiable Rewards (RLVR) on Large Language Models (LLMs)<n>We show that RLVR can extend the reasoning boundary for both mathematical and coding tasks.<n>We present a theoretical framework explaining RLVR's incentive mechanism, demonstrating how it can encourage correct reasoning even when rewards are based solely on answer correctness.
  arXiv  Detail & Related papers  (2025-06-17T07:06:56Z)
• Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewards [11.149294285483782]
  We propose a unified RLVR-based training paradigm that bridges the gap between non-verifiable tasks and verifiable rewards.<n>We introduce a writing-principle-based pairwise Generative Reward Model (GenRM) and a novel Bootstrapped Relative Policy Optimization (BRPO) algorithm.<n>Our approach empowers LLMs to develop robust writing capabilities without supervised fine-tuning.
  arXiv  Detail & Related papers  (2025-05-30T14:34:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.