Prioritizing the Best: Incentivizing Reliable Multimodal Reasoning by Rewarding Beyond Answer Correctness

• URL: http://arxiv.org/abs/2604.18892v1
• Date: Mon, 20 Apr 2026 22:28:07 GMT
• Title: Prioritizing the Best: Incentivizing Reliable Multimodal Reasoning by Rewarding Beyond Answer Correctness
• Abstract summary: Reinforcement Learning with Verifiable Rewards improves multimodal reasoning by rewarding verifiable final answers.<n>This gap between answer correctness and reasoning validity motivates trajectory supervision in multimodal RL.<n>Groupwise Ranking Reward ranks verifier-passed trajectories for the same prompt in one pass and redistributes reward accordingly.
• Score: 26.85256114041302
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) improves multimodal reasoning by rewarding verifiable final answers. Yet answer-correct trajectories may still rely on incomplete derivations, weak evidence, or statements that contradict their conclusions. This gap between answer correctness and reasoning validity, which we call reasoning-answer inconsistency, motivates trajectory supervision in multimodal RL. We compare two main approaches: reward models (RMs), and Generative Rewards (GRs). RMs are efficient and help early in training, but their gains weaken as the policy distribution shifts; GRs improve performance, but may give unstable rewards and computationally expensive. We therefore propose Groupwise Ranking Reward, which ranks verifier-passed trajectories for the same prompt in one pass and redistributes reward accordingly. Groupwise comparison better separates stronger and weaker correct trajectories with lower judge overhead than GRs. Experiments show that RLVR aggravates reasoning-answer inconsistency, while trajectory supervision alleviates it. Groupwise Ranking Reward performs best overall, improving reliability-conditioned accuracy from 47.4% to 54.7% over RLVR.

Related papers

• Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL [55.05743310621117]
  Co-RL is a framework in which multiple decoupled models, sharing no parameters, are simultaneously optimized through RL using rewards derived from their peers.<n>We show that increasing cohort diversity, through heterogeneous model families, sizes, and rephrased training samples, reduces the correlated errors that drive self-reinforcing feedback loops.
  arXiv  Detail & Related papers  (2026-08-18T01:16:02Z)
• Reasoning Arena: Trace Tournaments When Verifiable Rewards Fall Short [51.667769734342635]
  Reinforcement learning with verifiable rewards (RLVR) has become a leading paradigm for improving the reasoning ability of large language models.<n>We propose Reasoning Arena, an adaptive training framework that routes such non-diverse reward groups to a judge system instead of discarding them.<n>We show that Reasoning Arena consistently outperforms the RLVR baseline by 7.6% on average in competition mathematics and coding benchmarks.
  arXiv  Detail & Related papers  (2026-06-08T11:57:17Z)
• ARBOR: Online Process Rewards via a Reusable Rubric Buffer for Search Agents [48.80766702702854]
  LLM-based search agents are trained predominantly with outcome-only reward, leaving the search process itself unsupervised.<n>This signal degenerates on outcome-homogeneous groups where all sampled trajectories share the same correctness, yielding zero within-group advantage and no gradient.<n>We propose ARBOR (Adaptive Buffer for Online Reward), a reusable process-reward framework that maintains a rubric memory shared across queries.
  arXiv  Detail & Related papers  (2026-06-02T06:58:54Z)
• Seeing with You: Perception-Reasoning Coevolution for Multimodal Reasoning [30.60184048111503]
  We introduce PRCO (Perception-Reasoning Coevolution), a dual-role RLVR framework with a shared policy.<n>We show PRCO yields consistent improvements across model scales by over 7 points on average accuracy compared to the base model.
  arXiv  Detail & Related papers  (2026-03-30T16:03:56Z)
• Learn Hard Problems During RL with Reference Guided Fine-tuning [56.56461712665904]
  Reinforcement learning (RL) for mathematical reasoning can suffer from reward sparsity.<n>We introduce Reference-Guided Fine-Tuning (ReGFT) to synthesize positive trajectories on hard problems and train on them before RL.<n>Our results show that ReGFT effectively overcomes reward sparsity and unlocks stronger RL-based mathematical reasoning.
  arXiv  Detail & Related papers  (2026-03-01T18:41:28Z)
• Beyond Correctness: Learning Robust Reasoning via Transfer [51.403609251508904]
  We adopt a simple philosophical view, robust reasoning should remain useful beyond the mind that produced it.<n>We introduce Reinforcement Learning with Transferable Reward, which operationalizes robustness via transfer reward.<n>Our approach improves sampling consistency while improving final answer accuracy, and it reaches comparable performance in substantially fewer training steps.
  arXiv  Detail & Related papers  (2026-02-09T10:41:44Z)
• From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generation [52.62655622099456]
  We propose reinforcement learning with verifiable reference-based rewards (RLVRR)<n>Instead of checking the final answer, RLVRR extracts an ordered linguistic signal from high-quality references (i.e., reward chain)<n>In this way, RLVRR decomposes rewards into two dimensions: content, which preserves deterministic core concepts, and style, which evaluates adherence to stylistic properties.
  arXiv  Detail & Related papers  (2026-01-26T14:39:58Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• PACR: Progressively Ascending Confidence Reward for LLM Reasoning [55.06373646059141]
  We propose Progressively Ascending Confidence Reward (PACR)<n>PACR is a dense, model-intrinsic reward computed directly from the model's evolving belief in the correct answer.<n>Our results suggest that dense, model-intrinsic shaping signals can make RLVR training more effective and reliable.
  arXiv  Detail & Related papers  (2025-10-25T11:25:35Z)
• Self-Aligned Reward: Towards Effective and Efficient Reasoners [16.740993145927774]
  Self-aligned reward (SAR) is a self-guided signal that complements verifiable rewards to encourage both reasoning accuracy and efficiency.<n>We show that SAR reliably distinguishes answer quality: concise, correct answers score higher than redundant ones, and partially correct answers score higher than entirely incorrect ones.
  arXiv  Detail & Related papers  (2025-09-05T20:39:43Z)
• Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains [9.917318870162365]
  Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for complex reasoning tasks with clear correctness signals such as math and coding.<n>Rubrics have recently been used in evaluation benchmarks to capture such judgments, but their potential as reward signals for on-policy post-training remains underexplored.<n>We introduce RaR, an on-policy reinforcement learning method that extends RLVR beyond verifiable domains by using rubric-based feedback.
  arXiv  Detail & Related papers  (2025-07-23T17:57:55Z)
• RLPR: Extrapolating RLVR to General Domains without Verifiers [103.14103272635893]
  We propose RLPR, a simple verifier-free framework that extrapolates RLVR to broader general domains.<n>We find that addressing the high variance of this noisy probability reward is crucial to make it work.<n>RLPR consistently improves reasoning capabilities in both areas for Gemma, Llama, and Qwen based models.
  arXiv  Detail & Related papers  (2025-06-23T02:56:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.