Aletheia: What Makes RLVR For Code Verifiers Tick?

• URL: http://arxiv.org/abs/2601.12186v1
• Date: Sat, 17 Jan 2026 22:30:45 GMT
• Title: Aletheia: What Makes RLVR For Code Verifiers Tick?
• Abstract summary: Verifiers trained via Reinforcement Learning from Verifiable Rewards (RLVR) are a prominent fixture of the Large Language Model (LLM) post-training pipeline.<n>Code verifiers remain valuable toward judging model outputs in scenarios where execution feedback is hard to obtain.<n>We examine components of the RLVR-based verifier training recipe widely credited for its success.
• Score: 51.371034079170435
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-domain thinking verifiers trained via Reinforcement Learning from Verifiable Rewards (RLVR) are a prominent fixture of the Large Language Model (LLM) post-training pipeline, owing to their ability to robustly rate and rerank model outputs. However, the adoption of such verifiers towards code generation has been comparatively sparse, with execution feedback constituting the dominant signal. Nonetheless, code verifiers remain valuable toward judging model outputs in scenarios where execution feedback is hard to obtain and are a potentially powerful addition to the code generation post-training toolbox. To this end, we create and open-source Aletheia, a controlled testbed that enables execution-grounded evaluation of code verifiers' robustness across disparate policy models and covariate shifts. We examine components of the RLVR-based verifier training recipe widely credited for its success: (1) intermediate thinking traces, (2) learning from negative samples, and (3) on-policy training. While experiments show the optimality of RLVR, we uncover important opportunities to simplify the recipe. Particularly, despite code verification exhibiting positive training- and inference-time scaling, on-policy learning stands out as the key component at small verifier sizes, and thinking-based training emerges as the most important component at larger scales.

Related papers

• From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement [56.26277769790695]
  Reinforcement Learning with Verifiable Rewards (RLVR) has driven recent progress in reasoning-oriented large language models (LLMs)<n>We propose Reinforcement Learning with Self-Verifiable Rewards (RLSVR), a task-transformation-based training paradigm for extending RLVR to open-ended tasks.<n>We instantiate RLSVR with SpyRL, a Self-PlaY Reinforcement Learning method inspired by social deduction game Who Is the Spy?
  arXiv  Detail & Related papers  (2026-07-26T19:04:33Z)
• When Do Intrinsic Rewards Work for Code Reasoning? A Comprehensive Study [77.64957118012423]
  Reinforcement learning with verifiable rewards (RLVR) has driven substantial progress in large language model reasoning.<n>Recent work addresses this by deriving rewards from a model's own signals, such as majority voting or confidence-based scores.<n>We present a systematic empirical study of intrinsic reward methods for code generation.
  arXiv  Detail & Related papers  (2026-06-18T19:15:50Z)
• When Self-Belief Misleads: Active Label Acquisition for Reinforcement Learning with Verifiable Rewards [49.25249414962884]
  We propose Reinforcement Learning with Active Verifiable Rewards (RLAVR)<n>RLAVR actively acquires ground-truth labels for a small set of selected samples and integrates them with pseudo-labels.<n>Building on this, we introduce Correction-Aware Reliability Estimation for RLAVR (CARE)
  arXiv  Detail & Related papers  (2026-05-25T13:55:12Z)
• CVeDRL: An Efficient Code Verifier via Difficulty-aware Reinforcement Learning [57.24524263804788]
  Code verifiers play a critical role in post-verification for LLM-based code generation.<n>Existing supervised fine-tuning methods suffer from data scarcity, high failure rates, and poor inference efficiency.<n>We show that naive RL with only functionality rewards fails to generate effective unit tests for difficult branches and samples.
  arXiv  Detail & Related papers  (2026-01-30T10:33:29Z)
• Reinforcement Learning via Self-Distillation [37.078107691613155]
  Large language models are increasingly post-trained with reinforcement learning in verifiable domains such as code and math.<n>Current methods for reinforcement learning with verifiable rewards (RLVR) learn only from a scalar outcome reward per attempt, creating a severe credit-assignment bottleneck.<n>We formalize this setting as reinforcement learning with rich feedback and introduce Self-Distillation Policy Optimization (SDPO)<n>SDPO converts tokenized feedback into a dense learning signal without any external teacher or explicit reward model.
  arXiv  Detail & Related papers  (2026-01-28T17:45:12Z)
• From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generation [52.62655622099456]
  We propose reinforcement learning with verifiable reference-based rewards (RLVRR)<n>Instead of checking the final answer, RLVRR extracts an ordered linguistic signal from high-quality references (i.e., reward chain)<n>In this way, RLVRR decomposes rewards into two dimensions: content, which preserves deterministic core concepts, and style, which evaluates adherence to stylistic properties.
  arXiv  Detail & Related papers  (2026-01-26T14:39:58Z)
• ConfClip: Confidence-Weighted and Clipped Reward for Reinforcement Learning in LLMs [32.13266235550995]
  Reinforcement learning (RL) has become a standard paradigm for refining large language models (LLMs)<n>Inspired by observations from human learning, we introduce a RL technique that integrates verifiable outcomes with the model's own confidence estimates.
  arXiv  Detail & Related papers  (2025-09-22T13:00:35Z)
• ReVeal: Self-Evolving Code Agents via Reliable Self-Verification [11.875519107421312]
  We introduce ReVeal, a reinforcement learning framework that evolves code generation through self-verification and tool-based evaluation.<n>At inference, this strengthened self-verification enables the model to use self-constructed tests and tool feedback to continuously evolve code for 20+ turns on LiveCodeBench despite training on only three.<n>These findings highlight the promise of ReVeal as a scalable paradigm for RL training and test-time scaling, paving the way for more robust and autonomous AI agents.
  arXiv  Detail & Related papers  (2025-06-13T03:41:04Z)
• VerIF: Verification Engineering for Reinforcement Learning in Instruction Following [55.60192044049083]
  Reinforcement learning with verifiable rewards (RLVR) has become a key technique for enhancing large language models (LLMs)<n>We propose VerIF, a verification method that combines rule-based code verification with LLM-based verification from a large reasoning model.<n>We apply RL training with VerIF to two models, achieving significant improvements across several representative instruction-following benchmarks.
  arXiv  Detail & Related papers  (2025-06-11T17:10:36Z)
• Learning to Reason without External Rewards [100.27210579418562]
  Training large language models (LLMs) for complex reasoning via Reinforcement Learning with Verifiable Rewards (RLVR) is effective but limited by reliance on costly, domain-specific supervision.<n>We explore Reinforcement Learning from Internal Feedback (RLIF), a framework that enables LLMs to learn from intrinsic signals without external rewards or labeled data.<n>We propose Intuitor, an RLIF method that uses a model's own confidence, termed self-certainty, as its sole reward signal.
  arXiv  Detail & Related papers  (2025-05-26T07:01:06Z)
• RL Tango: Reinforcing Generator and Verifier Together for Language Reasoning [29.36492073947041]
  Tango is a novel framework that usesReinforcement learning to concurrently train both an LLM generator and a verifier.<n>A central innovation of Tango is its generative, process-level LLM verifier, which is trained via RL and co-evolves with the generator.<n>Experiments demonstrate that both components of Tango achieve state-of-the-art results among 7B/8B-scale models.
  arXiv  Detail & Related papers  (2025-05-21T02:43:15Z)
• Process Supervision-Guided Policy Optimization for Code Generation [15.943210767010045]
  Reinforcement learning (RL) with unit test feedback has enhanced large language models' (LLMs) code generation, but relies on sparse rewards provided only after complete code evaluation.<n>We propose a Process Reward Model (PRM) that delivers dense, line-level feedback on code correctness during generation, mimicking human code refinement.
  arXiv  Detail & Related papers  (2024-10-23T07:22:33Z)
• CCLF: A Contrastive-Curiosity-Driven Learning Framework for Sample-Efficient Reinforcement Learning [56.20123080771364]
  We develop a model-agnostic Contrastive-Curiosity-Driven Learning Framework (CCLF) for reinforcement learning. CCLF fully exploit sample importance and improve learning efficiency in a self-supervised manner. We evaluate this approach on the DeepMind Control Suite, Atari, and MiniGrid benchmarks.
  arXiv  Detail & Related papers  (2022-05-02T14:42:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.