On the Faithfulness of Visual Thinking: Measurement and Enhancement

• URL: http://arxiv.org/abs/2510.23482v1
• Date: Mon, 27 Oct 2025 16:15:54 GMT
• Title: On the Faithfulness of Visual Thinking: Measurement and Enhancement
• Abstract summary: Recent vision-language models can generate vision-text multimodal chain-of-thought traces after reinforcement fine-tuning.<n>We observe that the visual information incorporated in MCoT is often inaccurate, though still yield correct answers.<n>We propose a novel MCoT learning strategy termed Sufficient-Component Cause Model (SCCM) learning.
• Score: 37.52991654147004
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent large vision-language models (LVLMs) can generate vision-text multimodal chain-of-thought (MCoT) traces after reinforcement fine-tuning (RFT). However, we observe that the visual information incorporated in MCoT is often inaccurate, though still yield correct answers, indicating a lack of faithfulness in the MCoT reasoning process. We attribute this unfaithfulness to the RL reward in RFT, which solely incentivizes the format of interleaved vision-text cues, ie, it encourages the model to incorporate visual information into its text reasoning steps without considering the correctness of the visual information. In this paper, we first probe the faithfulness of MCoT by measuring how much the prediction changes when its visual and textual thoughts are intervened. Surprisingly, the model's predictions remain nearly unchanged under visual intervention but change significantly under textual intervention, indicating that the visual evidence is largely ignored. To further analyze visual information, we introduce an automated LVLM-based evaluation metric that quantifies the faithfulness of visual cues from two perspectives: reliability and sufficiency. Our evaluation reveals that the visual information in current MCoT traces is simultaneously unreliable and insufficient. To address this issue, we propose a novel MCoT learning strategy termed Sufficient-Component Cause Model (SCCM) learning. This approach encourages the MCoT to generate sufficient yet minimal visual components that are independently capable of leading to correct answers. We note that the proposed SCCM is annotation-free and compatible with various RFT for MCoT in a plug-and-play manner. Empirical results demonstrate that SCCM consistently improves the visual faithfulness across a suite of fine-grained perception and reasoning benchmarks. Code is available at https://github.com/EugeneLiu01/Faithful_Thinking_with_Image.

Related papers

• EVL-MCoT: Enhanced Vision-Language Multi-CoT for Harmful Meme Detection [13.261187830239797]
  MEMEs are widely used on the internet and often carry strong elements of sarcasm or irony.<n>Existing methods focus on the dual-stream vision-language model to extract the visual and text simultaneously.<n>We propose an enhanced vision-language multi-CoT (EVL-MCoT) approach to address these limitations.
  arXiv  Detail & Related papers  (2026-07-24T06:28:57Z)
• Diagnosing Visual Ignorance in Vision-Language Models [29.2851901986069]
  Vision-Language Models (VLMs) frequently rely on language priors, producing confident answers that are weakly grounded in visual evidence.<n>In this work, we study language-prior reliance from both mechanistic and behavioral perspectives.<n>We find that language-prior reliance is a systematic routing failure affecting both model internals and benchmark validity.
  arXiv  Detail & Related papers  (2026-06-05T04:16:01Z)
• Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Models [34.388508959416725]
  We analyze reasoning dynamics in vision language models (VLMs)<n>We find that models are prone to answer inertia, in which early commitments to a prediction are reinforced.<n> Reasoning-trained models are more likely to explicitly refer to the cues, but their longer and fluent CoTs can still appear visually grounded.
  arXiv  Detail & Related papers  (2026-04-16T11:28:53Z)
• Reflect to Inform: Boosting Multimodal Reasoning via Information-Gain-Driven Verification [55.357038267439684]
  Visual Re-Examination (VRE) is a self-evolving training framework that enables MLLMs to autonomously perform visual introspection during reasoning without additional visual inputs.<n>VRE consistently improves reasoning accuracy and perceptual reliability, while substantially reducing hallucinations, especially in long-chain settings.
  arXiv  Detail & Related papers  (2026-03-27T12:22:13Z)
• DIVER: Dynamic Iterative Visual Evidence Reasoning for Multimodal Fake News Detection [6.225860651499494]
  Multimodal fake news detection is crucial for mitigating adversarial misinformation.<n>We propose DIVER (Dynamic Iterative Visual Evidence Reasoning), a framework grounded in a progressive, evidence-driven reasoning paradigm.<n>Experiments on Weibo, Weibo21, and GossipCop demonstrate that DIVER outperforms state-of-the-art baselines by an average of 2.72%.
  arXiv  Detail & Related papers  (2026-01-12T04:01:33Z)
• Draft and Refine with Visual Experts [18.983324604452118]
  Large Vision-Language Models (LVLMs) produce ungrounded or hallucinated responses because they rely too heavily on linguistic priors instead of visual evidence.<n>We propose Draft and Refine (DnR), an agent framework driven by a question-conditioned utilization metric.
  arXiv  Detail & Related papers  (2025-11-14T06:50:14Z)
• When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thought [118.71264263478083]
  We propose MIRA, a new benchmark designed to evaluate models in scenarios where generating intermediate visual images is essential for successful reasoning.<n>We include 546 multimodal problems, annotated with intermediate visual images and final answers.
  arXiv  Detail & Related papers  (2025-11-04T18:00:51Z)
• Rethinking the Text-Vision Reasoning Imbalance in MLLMs through the Lens of Training Recipes [54.374410871041164]
  Multimodal large language models (MLLMs) have demonstrated strong capabilities on vision-and-language tasks.<n>Recent findings reveal an imbalance in their reasoning capabilities across visual and textual modalities.<n>We refer to this phenomenon as the textitmodality gap, defined as the performance disparity between text-centric and vision-centric inputs.
  arXiv  Detail & Related papers  (2025-10-26T21:06:13Z)
• Unleashing Perception-Time Scaling to Multimodal Reasoning Models [60.578179197783754]
  Recent advances in inference-time scaling have substantially enhanced the reasoning capabilities of Large Vision-Language Models (LVLMs)<n>Inspired by this success, similar strategies have been applied to multimodal reasoning, yet their impact on visual perception remains unclear.<n>We propose Perception-Time Scaling (PTS), a novel paradigm that encourages token-rich perception and decomposes complex perception problems into intermediate tractable sub-problems.
  arXiv  Detail & Related papers  (2025-10-10T03:17:52Z)
• Visual CoT Makes VLMs Smarter but More Fragile [79.32638667101817]
  Chain-of-Thought (CoT) techniques have significantly enhanced reasoning in Vision-Language Models (VLMs)<n>Visual CoT integrates explicit visual edits, such as cropping or annotating regions of interest, into the reasoning process.<n>We present the first systematic evaluation of Visual CoT robustness under visual perturbations.
  arXiv  Detail & Related papers  (2025-09-28T10:19:59Z)
• Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought [72.93910800095757]
  multimodal chain-of-thought (MCoT) improves performance and interpretability of large vision-language models (LVLMs)<n>We show that MCoT boosts LVLMs by incorporating visual thoughts, which convey image information to the reasoning process regardless of the MCoT format.<n>We also explore the internal nature of visual thoughts, finding that visual thoughts serve as intermediaries between the input image and reasoning to deeper transformer layers.
  arXiv  Detail & Related papers  (2025-05-21T13:29:58Z)
• Exploring Implicit Visual Misunderstandings in Multimodal Large Language Models through Attention Analysis [21.869968563545736]
  We define implicit visual misunderstanding (IVM), where MLLMs provide correct answers without fully comprehending the visual input.<n>We introduce a scale-agnostic metric, textitattention accuracy, and a novel benchmark for quantifying IVMs.<n>We extend our approach to finer granularities and demonstrate its effectiveness in unimodal scenarios.
  arXiv  Detail & Related papers  (2025-05-15T17:52:40Z)
• Vision Matters When It Should: Sanity Checking Multimodal Machine Translation Models [25.920891392933058]
  Multimodal machine translation (MMT) systems have been shown to outperform their text-only neural machine translation (NMT) counterparts when visual context is available. Recent studies have also shown that the performance of MMT models is only marginally impacted when the associated image is replaced with an unrelated image or noise.
  arXiv  Detail & Related papers  (2021-09-08T03:32:48Z)
• What Makes for Good Views for Contrastive Learning? [90.49736973404046]
  We argue that we should reduce the mutual information (MI) between views while keeping task-relevant information intact. We devise unsupervised and semi-supervised frameworks that learn effective views by aiming to reduce their MI. As a by-product, we achieve a new state-of-the-art accuracy on unsupervised pre-training for ImageNet classification.
  arXiv  Detail & Related papers  (2020-05-20T17:59:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.