Improving Reasoning in Vision-Language Models via Perception Verified Self-Training

• URL: http://arxiv.org/abs/2606.22158v2
• Date: Thu, 25 Jun 2026 05:42:39 GMT
• Title: Improving Reasoning in Vision-Language Models via Perception Verified Self-Training
• Abstract summary: We propose a perception-verified self-training framework that enforces visually grounded reasoning.<n>First, our method employs a CoT template that disentangles perception from reasoning, enabling independent verification of visual understanding.<n>Using caption verification together with answer correctness, we partition the data into three subsets: easy (correct caption and conclusion), medium (correct caption but incorrect conclusion), and hard (incorrect caption)
• Score: 3.443084677278651
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Achieving human-like reasoning in Vision-Language Models (VLMs) remains a long-standing challenge. Recent approaches leverage Chain-of-Thought (CoT) rationales generated by human annotators or proprietary models to improve reasoning, which is costly and difficult to scale. Self-training offers a promising alternative by using models own outputs as supervision. However, existing methods often suffer from visual hallucinations -- where rationales describe non-existent visual content, and language shortcuts -- where predictions rely on textual priors rather than true visual grounding, as rationales are typically filtered only by answer correctness without verifying visual perception. To address this limitation, we propose a perception-verified self-training framework that enforces visually grounded reasoning. First, our method employs a CoT template (caption-reasoning-conclusion) that disentangles perception from reasoning, enabling independent verification of visual understanding. To compensate for the absence of ground-truth captions, we propose PerceptEval, an unsupervised method that evaluates caption quality based on its alignment with visual and textual elements present in the image. Using caption verification together with answer correctness, we partition the data into three subsets: easy (correct caption and conclusion), medium (correct caption but incorrect conclusion), and hard (incorrect caption). Building on this partitioning, we design a two-stage curriculum learning strategy. In Stage 1, the model is trained on easy examples and subsequently in Stage 2, medium samples are incorporated through a caption-guided reasoning enhancement procedure that regenerates reasoning conditioned on verified captions. Only regenerated samples with the correct conclusions are retained.

Related papers

• Disentangling Visual and Factual Correctness in LVLMs' Visualization Literacy [27.323597665236644]
  Large Vision-Language Models (LVLMs) show strong visualization interpretation.<n>It is unclear whether their responses reflect genuine reasoning over visual evidence or factual priors learned during training.<n>We present a framework that isolates visual correctness from factual correctness.
  arXiv  Detail & Related papers  (2026-06-02T04:36:32Z)
• Look on Demand: A Cognitive Scheduling Framework for Visual Evidence Acquisition in Multimodal Reasoning [82.86343313807158]
  We argue that a central challenge is how and when visual evidence is introduced into the reasoning process.<n>Motivated by this insight, we propose CSMR, a multimodal reasoning framework in which a language model controls the reasoning process.
  arXiv  Detail & Related papers  (2026-05-27T08:43:13Z)
• Beyond Shortcuts: Mitigating Visual Illusions in Frozen VLMs via Qualitative Reasoning [14.700305370711973]
  We propose a training-free, data-centric framework designed to fortify visual grounding in frozen Vision-Language Models.<n>Our framework was evaluated on the DataCV 2026 Challenge (Task I: Classic Illusion Understanding), where it ranked 2nd place overall.<n>Our success underscores the potential of structured qualitative grounding as a robust paradigm for developing next-generation, illusion-resistant vision-language systems.
  arXiv  Detail & Related papers  (2026-04-29T03:12:29Z)
• Think in Strokes, Not Pixels: Process-Driven Image Generation via Interleaved Reasoning [59.262311672150055]
  We introduce process-driven image generation, a multi-step paradigm that decomposes synthesis into an interleaved reasoning trajectory.<n>A core challenge of process-driven generation stems from the ambiguity of intermediate states.<n>We address this through dense, step-wise supervision that maintains two complementary constraints.
  arXiv  Detail & Related papers  (2026-04-06T15:11:57Z)
• Seeing the Evidence, Missing the Answer: Tool-Guided Vision-Language Models on Visual Illusions [2.9521041993295607]
  Vision-language models (VLMs) exhibit a systematic bias when confronted with classic optical illusions.<n>We present a tool-guided inference framework that addresses this failure mode without any model training.<n>We report three empirical observations that we believe warrant additional investigation.
  arXiv  Detail & Related papers  (2026-03-31T08:35:10Z)
• Can Unified Generation and Understanding Models Maintain Semantic Equivalence Across Different Output Modalities? [61.533560295383786]
  Unified Multimodal Large Language Models (U-MLLMs) integrate understanding and generation within a single architecture.<n>We observe that U-MLLMs fail to maintain semantic equivalence when required to render the same results in the image modality.<n>We introduce VGUBench, a framework to decouple reasoning logic from generation fidelity.
  arXiv  Detail & Related papers  (2026-02-27T06:23:56Z)
• Analyzing Reasoning Consistency in Large Multimodal Models under Cross-Modal Conflicts [74.47786985522762]
  We identify a critical failure mode termed textual inertia, where models tend to blindly adhere to the erroneous text while neglecting conflicting visual evidence.<n>We propose the LogicGraph Perturbation Protocol that structurally injects perturbations into the reasoning chains of diverse LMMs.<n>Results reveal that models successfully self-correct in less than 10% of cases and predominantly succumb to blind textual error propagation.
  arXiv  Detail & Related papers  (2026-01-07T16:39:34Z)
• Self-Rewarding Vision-Language Model via Reasoning Decomposition [49.784411666601905]
  Vision-Language Models (VLMs) often suffer from visual hallucinations, saying things that are not actually in the image, and language shortcuts.<n>We introduce Vision-SR1, a self-rewarding method that improves visual reasoning without relying on external visual supervisions.<n>Our experiments demonstrate that Vision-SR1 improves visual reasoning, mitigates visual hallucinations, and reduces reliance on language shortcuts.
  arXiv  Detail & Related papers  (2025-08-27T08:01:03Z)
• Grounded and Controllable Image Completion by Incorporating Lexical Semantics [111.47374576372813]
  Lexical Semantic Image Completion (LSIC) may have potential applications in art, design, and heritage conservation. We advocate generating results faithful to both visual and lexical semantic context. One major challenge for LSIC comes from modeling and aligning the structure of visual-semantic context.
  arXiv  Detail & Related papers  (2020-02-29T16:54:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.