CORA: Analyzing and bridging thinking-answer gap in Multimodal RLVR via Consistency-Oriented Reasoning Alignment

• URL: http://arxiv.org/abs/2606.14691v1
• Date: Fri, 12 Jun 2026 17:54:59 GMT
• Title: CORA: Analyzing and bridging thinking-answer gap in Multimodal RLVR via Consistency-Oriented Reasoning Alignment
• Abstract summary: Reinforcement learning with verifiable rewards (RLVR) has successfully elicited the reasoning capabilities of large language models.<n>Existing methods primarily focus on improving the visual coverage of reasoning traces and mitigating visual hallucinations, but underestimate the semantic inconsistency between the reasoning process and the final answer.<n>We propose Consistency-Oriented Reasoning Alignment (CORA), which introduces thinking-answer semantic consistency into RLVR through a lightweight plug-and-play consistency reward model.
• Score: 32.29376299423774
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning with verifiable rewards (RLVR) has successfully elicited the reasoning capabilities of large language models, motivating its extension to multimodal scenarios. Existing methods primarily focus on improving the visual coverage of reasoning traces and mitigating visual hallucinations, but underestimate the semantic inconsistency between the reasoning process and the final answer. In this paper, we delve into thinking-answer inconsistency in RLVR for large vision-language models (LVLMs), showing thorough analyses of rollouts collected throughout Group Relative Policy Optimization (GRPO) training process and post-RLVR evaluation outputs that this issue persists during training and remains present during inference. Motivated by the analysis, we propose Consistency-Oriented Reasoning Alignment (CORA), which introduces thinking-answer semantic consistency into RLVR through a lightweight plug-and-play consistency reward model, and further incorporates Hybrid Reward Advantage Splitting (HRAS) to stably coordinate task and consistency optimization. Extensive experiments across representative multimodal reasoning benchmarks and mainstream LVLMs show that CORA improves task performance while effectively mitigating thinking-answer inconsistency, leading to more faithful reasoning traces.

Related papers

• DyCo-RL: Dynamic Cross-Modal Coordination for Visual Reasoning [49.985643703496514]
  We propose DyCo-RL, which integrates dynamic cross-modal coordination into RLVR optimization.<n>Experiments demonstrate that DyCo-RL, when applied to Qwen2.5-VL-3B/7B, consistently improves four representative RLVR algorithms.
  arXiv  Detail & Related papers  (2026-06-06T07:56:25Z)
• Beyond Where to Look: Trajectory-Guided Reinforcement Learning for Multimodal RLVR [30.94808389410323]
  Trajectory-Guided Reinforcement Learning guides the policy model to integrate visual evidence into fine-grained reasoning processes.<n>Experiments on multiple multimodal reasoning benchmarks demonstrate that TGRL consistently improves reasoning performance.
  arXiv  Detail & Related papers  (2026-03-27T07:18:18Z)
• Bridging Perception and Reasoning: Token Reweighting for RLVR in Multimodal LLMs [51.60575965819268]
  We propose a plug-and-play Token-Reweighting (ToR) strategy that explicitly models this interdependence.<n>ToR delivers consistent performance gains across multiple multi-modal reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-03-26T06:25:27Z)
• Rethinking Token-Level Policy Optimization for Multimodal Chain-of-Thought [73.39221516441624]
  Multimodal Chain-of-Thought (CoT) reasoning requires large vision-language models to construct reasoning trajectories.<n>Existing Reinforcement Learning with Verifiable Rewards (RLVR) methods treat CoT uniformly without distinguishing varying degrees of visual grounding.<n>We propose Perception-Exploration Policy Optimization (PEPO), which derives a perception prior to hidden state similarity and integrates it with token entropy.
  arXiv  Detail & Related papers  (2026-03-24T06:38:00Z)
• Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewards [69.74686029941881]
  Reinforcement Learning with Verifiable Rewards (RLVR) is an effective paradigm for improving the reasoning capabilities of large language models.<n>We propose a unified neural scheduling framework that adaptively selects high-value rollouts throughout training.<n>Experiments on six mathematical reasoning benchmarks demonstrate consistent gains in performance and training efficiency across multiple RLVR optimization methods.
  arXiv  Detail & Related papers  (2026-02-09T10:51:58Z)
• Latent Reasoning VLA: Latent Thinking and Prediction for Vision-Language-Action Models [69.58413440457828]
  Vision-Language-Action (VLA) models benefit from chain-of-thought (CoT) reasoning, but existing approaches incur high inference overhead.<n>We propose Latent Reasoning VLA (textbfLaRA-VLA), a unified VLA framework that internalizes multi-modal CoT reasoning into continuous latent representations for embodied action.
  arXiv  Detail & Related papers  (2026-02-01T11:34:37Z)
• Rectifying LLM Thought from Lens of Optimization [48.98086817378953]
  Long chain-of-thought (CoT) prompting enables thorough exploration and deliberation.<n>Despite advances, long-CoT LLMs often exhibit suboptimal reasoning behaviors.<n>We introduce RePro, a novel approach to refine LLM reasoning during post-training.
  arXiv  Detail & Related papers  (2025-12-01T17:41:08Z)
• Rethinking RL Scaling for Vision Language Models: A Transparent, From-Scratch Framework and Comprehensive Evaluation Scheme [36.34443944082215]
  This work introduces a transparent, from-scratch framework forReinforcement learning (RL) in vision-based models (VLMs)<n>It offers a minimal yet functional four-step pipeline validated across multiple models and datasets.<n>In addition, a standardized evaluation scheme is proposed to assess training dynamics and reflective behaviors.
  arXiv  Detail & Related papers  (2025-04-03T13:53:28Z)
• OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cycles [91.88062410741833]
  We introduce OpenVLThinker, one of the first open-source large vision-language models (LVLMs) to exhibit sophisticated chain-of-thought reasoning.<n>We show that OpenVLThinker-7B consistently advances performance across six benchmarks demanding mathematical and general reasoning.
  arXiv  Detail & Related papers  (2025-03-21T17:52:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.