SVR-R1: Bootstrapping Multi-modal Reasoning with Self-verification in Reinforcement Learning

• URL: http://arxiv.org/abs/2607.10966v1
• Date: Mon, 13 Jul 2026 00:21:30 GMT
• Title: SVR-R1: Bootstrapping Multi-modal Reasoning with Self-verification in Reinforcement Learning
• Abstract summary: We introduce Self-Verified Reasoner (SVR-R1), a multi-turn RL framework that turns a model's own verification into a learning signal for multimodal reasoning.<n>For each query, the model proposes an answer using the same weights, and issues a binary self-verdict (Yes/No)<n>A 'No' triggers a second-chance rethink; a 'Yes,' or a turn cap, finalizes the output for computing the outcome-based reward.
• Score: 47.4480595602701
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We introduce Self-Verified Reasoner (SVR-R1), a multi-turn RL framework that turns a model's own verification into a learning signal for multimodal reasoning. For each query, the model proposes an answer using the same weights, and issues a binary self-verdict (Yes/No). A 'No' triggers a second-chance rethink; a 'Yes,' or a turn cap, finalizes the output for computing the outcome-based reward. SVR-R1 is implemented with GRPO and an asynchronous multi-turn rollout framework and needs no external supervision or auxiliary critics. We evaluate SVR-R1 on vision-language reasoning benchmarks and show that it improves accuracy by a large margin over strong standard GRPO baselines. Training dynamics show decreasing reliance on verification-fewer verification turns, yet higher test accuracy-indicating that the gap between verification and generation narrows as the policy internalizes self-correction and chooses the most confident answer via our framework. SVR-R1 bridges the less explored intersection of inference-time self-refinement and RL training for VLMs, offering a simple yet effective recipe for bootstrapping multimodal reasoning. We will open-source \textbf{SVR-R1} to facilitate future research in VLMs.

Related papers

• SVR: Self-Verifying Refinement via Joint Verdict-Confidence Reinforcement Learning for Adaptive Test-Time Compute [62.3458279176813]
  Self-Verifying Refinement is an oracle-free multi-turn reinforcement learning framework.<n>It learns to use self-verification as a compute-control policy.<n>It achieves a macro-average accuracy of 0.563 with only 2.99 inference turns on average.
  arXiv  Detail & Related papers  (2026-07-30T16:20:58Z)
• From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement [56.26277769790695]
  Reinforcement Learning with Verifiable Rewards (RLVR) has driven recent progress in reasoning-oriented large language models (LLMs)<n>We propose Reinforcement Learning with Self-Verifiable Rewards (RLSVR), a task-transformation-based training paradigm for extending RLVR to open-ended tasks.<n>We instantiate RLSVR with SpyRL, a Self-PlaY Reinforcement Learning method inspired by social deduction game Who Is the Spy?
  arXiv  Detail & Related papers  (2026-07-26T19:04:33Z)
• Reasoning Portability: Guiding Continual Learning for MLLMs in the RLVR Era [44.08086834976093]
  Vision-Language Models in Continual Learning (VLM-CL) aim to continuously adapt to new multimodal tasks while retaining prior knowledge.<n>We propose Reasoning-based Dynamic Balance Continual Learning (RDB-CL), which formalizes the per-sample Kullback-Leibler regularization in RLVR according to Reasoning Portability (RP)<n> Experiments show that RDB-CL consistently outperforms baselines, improving Last accuracy by +12.0% over the vanilla RLVR baseline.
  arXiv  Detail & Related papers  (2026-05-17T13:26:09Z)
• SAGE: Shaping Anchors for Guided Exploration in RLVR of LLMs [55.46289074417954]
  reinforcement learning with verifiable rewards (RLVR) reliably improves pass@1 on reasoning tasks, yet often fails to yield comparable gains in pass@k.<n>We argue that a central structural constraint arises from reverse-KL regularization, which stabilizes training but inherently anchors the policy to the reference distribution.<n>We propose SAGE, a principled framework that enables controllable empirical support expansion by reshaping the reverse-KL anchor distribution itself.
  arXiv  Detail & Related papers  (2026-05-15T07:42:21Z)
• Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoning [42.998601712623525]
  We show that RLVR does not always lead models to rely on reasoning in the way that is commonly thought.<n>This issue can be remedied with simple modifications to the post-training procedure.
  arXiv  Detail & Related papers  (2026-04-23T21:01:34Z)
• Recursive Think-Answer Process for LLMs and VLMs [54.52289112197118]
  We propose an efficient Recursive Think-Answer Process (R-TAP)<n>R-TAP enables models to engage in iterative reasoning cycles and generate more accurate answers.<n>We show that R-TAP-enhanced models consistently outperform conventional single-pass methods.
  arXiv  Detail & Related papers  (2026-03-02T17:20:10Z)
• Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewards [69.74686029941881]
  Reinforcement Learning with Verifiable Rewards (RLVR) is an effective paradigm for improving the reasoning capabilities of large language models.<n>We propose a unified neural scheduling framework that adaptively selects high-value rollouts throughout training.<n>Experiments on six mathematical reasoning benchmarks demonstrate consistent gains in performance and training efficiency across multiple RLVR optimization methods.
  arXiv  Detail & Related papers  (2026-02-09T10:51:58Z)
• AutoRubric-R1V: Rubric-Based Generative Rewards for Faithful Multimodal Reasoning [25.984031524564188]
  AutoRubric-R1V is a framework that integrates RLVR with process-level supervision through automatically collected rubric-based generative rewards.<n>By jointly leveraging rubric-based and outcome rewards, AutoRubric-R1V achieves state-of-the-art performance on six multimodal reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-10-16T14:40:02Z)
• CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models [85.315711639214]
  We introduce Curiosity-Driven Exploration (CDE), a framework that leverages the model's own intrinsic sense of curiosity to guide exploration.<n>For the actor, we use perplexity over its generated response, and for the critic, we use the variance of value estimates from a multi-head architecture.<n>Our theoretical analysis shows that the actor-wise bonus inherently penalizes overconfident errors and promotes diversity among correct responses.
  arXiv  Detail & Related papers  (2025-09-11T17:59:17Z)
• Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewards [67.86091419220816]
  Large Language Models (LLMs) show great promise in complex reasoning.<n>A prevalent issue is superficial self-reflection'', where models fail to robustly verify their own outputs.<n>We introduce RISE (Reinforcing Reasoning with Self-Verification), a novel online RL framework designed to tackle this.
  arXiv  Detail & Related papers  (2025-05-19T17:59:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.