Be Faithful When Response: Returning Fluent and Grounded Answers for Vision-Language Models Reinforcement Learning

• URL: http://arxiv.org/abs/2606.29984v1
• Date: Mon, 29 Jun 2026 08:58:33 GMT
• Title: Be Faithful When Response: Returning Fluent and Grounded Answers for Vision-Language Models Reinforcement Learning
• Abstract summary: Reinforcement Learning (RL) is an important paradigm for improving the reasoning capabilities of Vision-Language Models (VLMs)<n>We propose a Faithful Warm-Start (FWS) strategy that first curates samples with explicit vision-language causal relationships from six general VQA benchmarks.<n>A VLM-based judge is employed to further purify the dataset, ensuring strong causal consistency and visual faithfulness.<n> Experimental results show that such faithful supervision improves answer accuracy, stabilizes RL training, and reduces visually unsupported reasoning.
• Score: 12.622982203412478
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Reinforcement Learning (RL) is an important paradigm for improving the reasoning capabilities of Vision-Language Models (VLMs). However, directly applying RL to rollout multimodal reasoning can lead to instability, due to the exploitation of language priors, the neglect of visual evidence, and the generation of reasoning traces that are fluent yet not visually grounded. The question arises: Can initially steer the policy toward visually faithful reasoning regime before applying reinforcement learning? To this end, we propose a Faithful Warm-Start (FWS) strategy that first curates samples with explicit vision-language causal relationships from six general VQA benchmarks to construct the FaithfulQA dataset, where each of the image-question pairs gains a certain degree of visual observations, question requirements, commonsense knowledge, domain knowledge, and the final answer. Subsequently, a VLM-based judge is employed to further purify the dataset, ensuring strong causal consistency and visual faithfulness. This warm-start stage equips the model with the capability to understand causally grounded vision-language patterns before subsequent RL optimization under sparse answer-level rewards. Experimental results show that such faithful supervision improves answer accuracy, stabilizes RL training, and reduces visually unsupported reasoning.

Related papers

• Harnessing Image Question Dependence for Better VLM Test-time Reinforcement Learning [66.13367818892503]
  We propose TTIQ, a test-time reinforcement learning framework that harnesses image-question dependence for better vlm adaptation.<n> TTIQ teacher-forces each sampled response under the original image-question pair and its image- and question-ablated variants.<n>It combines image and question dependence with confidence to construct a response-level reward that favors jointly grounded responses.
  arXiv  Detail & Related papers  (2026-09-09T22:54:15Z)
• PDCR: Perception-Decomposed Confidence Reward for Vision-Language Reasoning [80.94559742826083]
  Reinforcement Learning with Verifiable Rewards (RLVR) traditionally relies on a sparse, outcome-based signal.<n>Recent work shows that providing a fine-grained, model-intrinsic signal effectively improves language reasoning training by providing step-level guidance without costly external models.<n>While effective for unimodal text, we find that naively applying this global reward to vision-language (V-L) reasoning is a suboptimal strategy.<n>We propose Perception-Decomposed Confidence Reward (PDCR), a framework that solves this by aligning the reward structure with the task's heterogeneous nature.
  arXiv  Detail & Related papers  (2026-05-13T12:55:18Z)
• Beyond Where to Look: Trajectory-Guided Reinforcement Learning for Multimodal RLVR [30.94808389410323]
  Trajectory-Guided Reinforcement Learning guides the policy model to integrate visual evidence into fine-grained reasoning processes.<n>Experiments on multiple multimodal reasoning benchmarks demonstrate that TGRL consistently improves reasoning performance.
  arXiv  Detail & Related papers  (2026-03-27T07:18:18Z)
• Saliency-Aware Multi-Route Thinking: Revisiting Vision-Language Reasoning [50.62037276161025]
  Vision-language models (VLMs) aim to reason by jointly leveraging visual and textual modalities.<n>Key obstacle is that visual inputs are typically provided only once at the start of generation.<n>We propose emphSaliency-Aware Principle (SAP) selection.
  arXiv  Detail & Related papers  (2026-02-18T18:49:56Z)
• See, Think, Learn: A Self-Taught Multimodal Reasoner [3.443084677278651]
  We propose a simple yet effective self-training framework called See-Think-Learn.<n>At its core, STL introduces a structured reasoning template that encourages the model to see before thinking.<n>We augment the training data with negative rationales to enhance the model's ability to distinguish between correct and misleading responses.
  arXiv  Detail & Related papers  (2025-12-02T06:30:10Z)
• Perceptual-Evidence Anchored Reinforced Learning for Multimodal Reasoning [29.78411369746505]
  PEARL is a dual-branch, perception-reasoning synergistic that strengthens multimodal reasoning by explicitly anchoring it to verified visual evidence.<n>PEARL achieves substantial gains on multimodal reasoning benchmarks, e.g., a +9.7% improvement over the baseline and +6.6% over GRPO on MathVerse.
  arXiv  Detail & Related papers  (2025-11-23T13:15:58Z)
• Unveiling Chain of Step Reasoning for Vision-Language Models with Fine-grained Rewards [48.55501117313608]
  We present chain of step reasoning for vision-language models, enabling assessing reasoning step quality accurately.<n>We present a simple, effective, and fully transparent framework, including the step-level reasoning data, process reward model (PRM), and reinforcement learning training.<n>We believe this paper serves as a baseline for vision-language models and offers insights into more complex multimodal reasoning.
  arXiv  Detail & Related papers  (2025-09-23T13:47:32Z)
• Semi-off-Policy Reinforcement Learning for Vision-Language Slow-Thinking Reasoning [68.25125496950928]
  This paper proposes SOPHIA, a simple and scalable Semi-Off-Policy RL for vision-language slow-tHInking reAsoning.<n> SOPHIA builds a semi-off-policy behavior model by combining on-policy visual understanding from a trainable LVLM with off-policy slow-thinking reasoning from a language model.<n>Experiments with InternVL2.5 and InternVL3.0 with 8B and 38B sizes show the effectiveness of SOPHIA.
  arXiv  Detail & Related papers  (2025-07-22T17:59:34Z)
• VL-GenRM: Enhancing Vision-Language Verification via Vision Experts and Iterative Training [23.391643634478587]
  Vision-Language Reward Model (VL-RM) is key to aligning VL models by providing structured feedback.<n> bootstrapping dilemma arises as high-quality training data depends on already strong VL models.<n>We propose an iterative training framework leveraging vision experts, Chain-of-Thought rationales, and Margin-based Rejection Sampling.
  arXiv  Detail & Related papers  (2025-06-16T18:10:51Z)
• Exploring the Effect of Reinforcement Learning on Video Understanding: Insights from SEED-Bench-R1 [53.894789613838654]
  We introduce SEED-Bench-R1, a benchmark designed to evaluate post-training methods for MLLMs in video understanding.<n>It includes intricate real-world videos and complex everyday planning tasks in the format of multiple-choice questions.<n>Using Qwen2-VL-Instruct-7B as a base model, we compare RL with supervised fine-tuning (SFT)<n>Our detailed analysis reveals that RL enhances visual perception but often produces less coherent reasoning chains.
  arXiv  Detail & Related papers  (2025-03-31T17:55:23Z)
• OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cycles [91.88062410741833]
  We introduce OpenVLThinker, one of the first open-source large vision-language models (LVLMs) to exhibit sophisticated chain-of-thought reasoning.<n>We show that OpenVLThinker-7B consistently advances performance across six benchmarks demanding mathematical and general reasoning.
  arXiv  Detail & Related papers  (2025-03-21T17:52:43Z)
• Improve Vision Language Model Chain-of-thought Reasoning [86.83335752119741]
  Chain-of-thought (CoT) reasoning in vision language models (VLMs) is crucial for improving interpretability and trustworthiness. We show that training VLM on short answers does not generalize well to reasoning tasks that require more detailed responses.
  arXiv  Detail & Related papers  (2024-10-21T17:00:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.