Reinforcement Learning with Conditional Expectation Reward

• URL: http://arxiv.org/abs/2603.10624v1
• Date: Wed, 11 Mar 2026 10:36:27 GMT
• Title: Reinforcement Learning with Conditional Expectation Reward
• Abstract summary: Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective in enhancing the reasoning capabilities of large language models.<n>We propose Conditional Expectation Reward (CER), which leverages the large language model itself as an implicit verifier.<n>CER is defined as the expected likelihood of generating the reference answer conditioned on the generated answer.
• Score: 11.862379897558556
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective in enhancing the reasoning capabilities of large language models, particularly in domains such as mathematics where reliable rule-based verifiers can be constructed. However, the reliance on handcrafted, domain-specific verification rules substantially limits the applicability of RLVR to general reasoning domains with free-form answers, where valid answers often exhibit significant variability, making it difficult to establish complete and accurate rules. To address this limitation, we propose Conditional Expectation Reward (CER), which leverages the large language model itself as an implicit verifier, and is therefore applicable to general domains and eliminates the need for external verifiers or auxiliary models. CER is defined as the expected likelihood of generating the reference answer conditioned on the generated answer. In contrast to rule-based verifiers that yield binary feedback, CER provides a soft, graded reward signal that reflects varying degrees of correctness, making it better suited to tasks where answers vary in correctness. Experimental results demonstrate that CER is effective across a wide range of reasoning tasks, spanning both mathematical and general domains, indicating that CER serves as a flexible and general verification mechanism. The code is available at https://github.com/changyi7231/CER.

Related papers

• From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement [56.26277769790695]
  Reinforcement Learning with Verifiable Rewards (RLVR) has driven recent progress in reasoning-oriented large language models (LLMs)<n>We propose Reinforcement Learning with Self-Verifiable Rewards (RLSVR), a task-transformation-based training paradigm for extending RLVR to open-ended tasks.<n>We instantiate RLSVR with SpyRL, a Self-PlaY Reinforcement Learning method inspired by social deduction game Who Is the Spy?
  arXiv  Detail & Related papers  (2026-07-26T19:04:33Z)
• GeneralThinker: Domain-General Reasoning through Likelihood-Guided Answer-Conditioned Optimization [4.732347368043908]
  GeneralThinker is an on-policy framework that reformulates reasoning supervision as dense answer-conditioned optimization.<n>GeneralThinker evaluates generated reasoning trajectories using the likelihood of the ground-truth answer.<n>It constrains token-level updates through clipping and direction-preserving modulation.
  arXiv  Detail & Related papers  (2026-05-27T04:07:26Z)
• Verifiable Process Rewards for Agentic Reasoning [21.195739597726142]
  Reinforcement learning from verifiable rewards (RLVR) has improved the reasoning abilities of large language models (LLMs)<n>Most existing approaches rely on sparse outcome-level feedback.<n>We propose Verifiable Process Rewards (VPR), a framework that converts such oracles into dense turn-level supervision for reinforcement learning.
  arXiv  Detail & Related papers  (2026-05-11T10:30:53Z)
• Justified or Just Convincing? Error Verifiability as a Dimension of LLM Quality [24.614192550852277]
  No standard measure exists for whether model-generated justifications help users distinguish correct answers from incorrect ones.<n>We formalize this idea as error verifiability and propose $v_textbal$, a balanced metric that measures whether justifications enable raters to accurately assess answer correctness.<n>We introduce two methods that succeed at improving verifiability: reflect-and-rephrase (RR) for mathematical reasoning and oracle-rephrase (OR) for factual QA.
  arXiv  Detail & Related papers  (2026-04-06T04:53:59Z)
• Know What You Know: Metacognitive Entropy Calibration for Verifiable RL Reasoning [31.629261193485053]
  Large reasoning models (LRMs) have emerged as a powerful paradigm for solving complex real-world tasks.<n>Most existing outcome-only RLVR pipelines rely almost exclusively on a binary correctness signal and largely ignore the model's intrinsic uncertainty.<n>We propose EGPO, a metacognitive entropy calibration framework that explicitly integrates intrinsic uncertainty into RLVR for enhancing LRMs.
  arXiv  Detail & Related papers  (2026-02-26T08:40:06Z)
• On Calibration of Large Language Models: From Response To Capability [66.59139960234326]
  Large language models (LLMs) are widely deployed as general-purpose problem solvers.<n>We introduce capability calibration, which targets the model's expected accuracy on a query.<n>Our results demonstrate that capability-calibrated confidence improves pass@$k$ prediction and inference budget allocation.
  arXiv  Detail & Related papers  (2026-02-14T01:07:45Z)
• From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generation [52.62655622099456]
  We propose reinforcement learning with verifiable reference-based rewards (RLVRR)<n>Instead of checking the final answer, RLVRR extracts an ordered linguistic signal from high-quality references (i.e., reward chain)<n>In this way, RLVRR decomposes rewards into two dimensions: content, which preserves deterministic core concepts, and style, which evaluates adherence to stylistic properties.
  arXiv  Detail & Related papers  (2026-01-26T14:39:58Z)
• DARL: Encouraging Diverse Answers for General Reasoning without Verifiers [41.35516261603945]
  We propose DARL, a reinforcement learning framework that encourages the generation of diverse answers within a controlled deviation range from the reference.<n>Our framework is fully compatible with existing general reinforcement learning methods and can be seamlessly integrated without additional verifiers.
  arXiv  Detail & Related papers  (2026-01-21T06:23:55Z)
• From Faithfulness to Correctness: Generative Reward Models that Think Critically [40.07140704454647]
  We propose the Thinking-supervised Reward Model (TRM) to endow reward models with critical thinking abilities.<n>Given a query, answer, and supporting documents, TRM first assesses the faithfulness of each answer sentence to the supporting documents, and then applies a reasoning step to evaluate sentence-level correctness.
  arXiv  Detail & Related papers  (2025-09-29T19:06:56Z)
• RLPR: Extrapolating RLVR to General Domains without Verifiers [103.14103272635893]
  We propose RLPR, a simple verifier-free framework that extrapolates RLVR to broader general domains.<n>We find that addressing the high variance of this noisy probability reward is crucial to make it work.<n>RLPR consistently improves reasoning capabilities in both areas for Gemma, Llama, and Qwen based models.
  arXiv  Detail & Related papers  (2025-06-23T02:56:36Z)
• Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains [92.36624674516553]
  Reinforcement learning with verifiable rewards (RLVR) has demonstrated significant success in enhancing mathematical reasoning and coding performance of large language models (LLMs)<n>We investigate the effectiveness and scalability of RLVR across diverse real-world domains including medicine, chemistry, psychology, economics, and education.<n>We utilize a generative scoring technique that yields soft, model-based reward signals to overcome limitations posed by binary verifications.
  arXiv  Detail & Related papers  (2025-03-31T08:22:49Z)
• Unsupervised Domain Generalization for Person Re-identification: A Domain-specific Adaptive Framework [50.88463458896428]
  Domain generalization (DG) has attracted much attention in person re-identification (ReID) recently. Existing methods usually need the source domains to be labeled, which could be a significant burden for practical ReID tasks. We propose a simple and efficient domain-specific adaptive framework, and realize it with an adaptive normalization module.
  arXiv  Detail & Related papers  (2021-11-30T02:35:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.