Unveiling the Response of Large Vision-Language Models to Visually Absent Tokens

• URL: http://arxiv.org/abs/2509.03025v2
• Date: Fri, 05 Sep 2025 07:49:47 GMT
• Title: Unveiling the Response of Large Vision-Language Models to Visually Absent Tokens
• Abstract summary: Large Vision-Language Models (LVLMs) generate contextually relevant responses by jointly interpreting visual and textual inputs.<n>We probe whether LVLMs possess an internal capability to determine if textual concepts are grounded in the image.<n>We develop a detection module that systematically classifies whether an input token is visually grounded.
• Score: 43.43888632864549
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Vision-Language Models (LVLMs) generate contextually relevant responses by jointly interpreting visual and textual inputs. However, our finding reveals they often mistakenly perceive text inputs lacking visual evidence as being part of the image, leading to erroneous responses. In light of this finding, we probe whether LVLMs possess an internal capability to determine if textual concepts are grounded in the image, and discover a specific subset of Feed-Forward Network (FFN) neurons, termed Visual Absence-aware (VA) neurons, that consistently signal the visual absence through a distinctive activation pattern. Leveraging these patterns, we develop a detection module that systematically classifies whether an input token is visually grounded. Guided by its prediction, we propose a method to refine the outputs by reinterpreting question prompts or replacing the detected absent tokens during generation. Extensive experiments show that our method effectively mitigates the models' tendency to falsely presume the visual presence of text input and its generality across various LVLMs.

Related papers

• Diagnosing Visual Ignorance in Vision-Language Models [29.2851901986069]
  Vision-Language Models (VLMs) frequently rely on language priors, producing confident answers that are weakly grounded in visual evidence.<n>In this work, we study language-prior reliance from both mechanistic and behavioral perspectives.<n>We find that language-prior reliance is a systematic routing failure affecting both model internals and benchmark validity.
  arXiv  Detail & Related papers  (2026-06-05T04:16:01Z)
• Aligning What Vision-Language Models See and Perceive with Adaptive Information Flow [32.21470980856891]
  Vision-Language Models (VLMs) have demonstrated strong capability in a wide range of tasks such as visual recognition, document parsing, and visual grounding.<n>Recent work shows that while VLMs often manage to capture the correct image region corresponding to the question, they do not necessarily produce the correct answers.<n>We show that this misalignment could be attributed to suboptimal information flow within VLMs, where text tokens distribute too much attention to irrelevant visual tokens.
  arXiv  Detail & Related papers  (2026-04-17T08:07:22Z)
• Hidden Meanings in Plain Sight: RebusBench for Evaluating Cognitive Visual Reasoning [14.135916464098317]
  We identify that current models struggle with the complex, multi-step reasoning required to solve problems where information is not explicitly depicted.<n>We introduce RebusBench, a benchmark of 1,164 puzzles designed to test this specific integration of perception and knowledge.<n>Our evaluation of state-of-the-art models shows a severe deficiency: performance saturates below 10% Exact Match and 20% semantic accuracy.
  arXiv  Detail & Related papers  (2026-04-02T08:33:13Z)
• Selective Training for Large Vision Language Models via Visual Information Gain [7.834991119179473]
  We introduce Visual Information Gain (VIG), a perplexity-based metric.<n>VIG measures the reduction in prediction uncertainty provided by visual input.<n>We propose a VIG-guided selective training scheme that prioritizes high-VIG samples and tokens.
  arXiv  Detail & Related papers  (2026-02-19T09:12:21Z)
• Visual Representations inside the Language Model [36.35124375782294]
  We study flow of visual information through the language model, finding that image value tokens encode sufficient information to perform several perception-heavy tasks.<n>We find that while the language model does augment the visual information received from the projection of input visual encodings, it contains less visual information on several tasks than the equivalent visual encoder (SigLIP)<n>Next, we discuss controlling visual information in the language model, showing that adding a text prefix to the image input improves perception capabilities of visual representations.
  arXiv  Detail & Related papers  (2025-10-06T14:01:39Z)
• GLIMPSE: Holistic Cross-Modal Explainability for Large Vision-Language Models [0.0]
  We introduce GLIMPSE, a model-agnostic framework that jointly attributes LVLM outputs to the most relevant visual evidence and textual signals.<n>GLIMPSE fuses gradient-weighted attention, adaptive layer propagation, and relevance-weighted token aggregation to produce holistic response-level heat maps.<n>We demonstrate an analytic approach to uncover fine-grained insights into LVLM cross-modal attribution, trace reasoning dynamics, analyze systematic misalignment, diagnose hallucination and bias, and ensure transparency.
  arXiv  Detail & Related papers  (2025-06-23T18:00:04Z)
• Revisit What You See: Disclose Language Prior in Vision Tokens for LVLM Decoding [6.612630497074871]
  Large Vision-Language Models (LVLMs) achieve strong performance across multimodal tasks by integrating visual perception with language understanding.<n>We propose ReVisiT, a training-free decoding method that references vision tokens to guide text generation.
  arXiv  Detail & Related papers  (2025-06-11T08:46:55Z)
• Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs [42.871396640891334]
  Large Vision-Language Models (LVLMs) are susceptible to hallucinations.<n>We introduce a novel Conditional Pointwise Mutual Information (C-PMI) calibrated decoding strategy.<n>We show that the proposed method significantly reduces hallucinations in LVLMs while preserving decoding efficiency.
  arXiv  Detail & Related papers  (2025-05-26T08:36:10Z)
• Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models [66.71616369573715]
  Large Vision-Language Models (LVLMs) are prone to generating hallucinatory text responses that do not align with the given visual input.<n>We introduce self-correcting Decoding with Generative Feedback (DeGF), a novel training-free algorithm that incorporates feedback from text-to-image generative models into the decoding process.
  arXiv  Detail & Related papers  (2025-02-10T03:43:55Z)
• Retrieval-Based Interleaved Visual Chain-of-Thought in Real-World Driving Scenarios [69.00444996464662]
  We propose RIV-CoT, a Retrieval-Based Interleaved Visual Chain-of-Thought method that enables vision-language models to reason using visual crops corresponding to relevant entities.<n>Our experiments demonstrate that RIV-CoT improves answer accuracy by 3.1% and reasoning accuracy by 4.6% over vanilla CoT prompting.
  arXiv  Detail & Related papers  (2025-01-08T18:31:16Z)
• Debiasing Multimodal Large Language Models via Penalization of Language Priors [38.97645845493758]
  Multimodal Large Language Models (MLLMs) have become indispensable tools in computer vision and natural language processing.<n>Despite their advancements, our investigation reveals a noteworthy bias: the generated content is often driven more by the inherent priors of the underlying Large Language Models (LLMs) than by the input image.<n>We propose two simple, training-free strategies to rectify these biases and redirect the model's focus toward visual information.
  arXiv  Detail & Related papers  (2024-03-08T12:35:07Z)
• Visual In-Context Learning for Large Vision-Language Models [62.5507897575317]
  In Large Visual Language Models (LVLMs) the efficacy of In-Context Learning (ICL) remains limited by challenges in cross-modal interactions and representation disparities. We introduce a novel Visual In-Context Learning (VICL) method comprising Visual Demonstration Retrieval, Intent-Oriented Image Summarization, and Intent-Oriented Demonstration Composition. Our approach retrieves images via ''Retrieval & Rerank'' paradigm, summarises images with task intent and task-specific visual parsing, and composes language-based demonstrations.
  arXiv  Detail & Related papers  (2024-02-18T12:43:38Z)
• GeoVLN: Learning Geometry-Enhanced Visual Representation with Slot Attention for Vision-and-Language Navigation [52.65506307440127]
  We propose GeoVLN, which learns Geometry-enhanced visual representation based on slot attention for robust Visual-and-Language Navigation. We employ V&L BERT to learn a cross-modal representation that incorporate both language and vision informations.
  arXiv  Detail & Related papers  (2023-05-26T17:15:22Z)
• Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models [103.61066310897928]
  Recent text-to-image generative models have demonstrated an unparalleled ability to generate diverse and creative imagery guided by a target text prompt. While revolutionary, current state-of-the-art diffusion models may still fail in generating images that fully convey the semantics in the given text prompt. We analyze the publicly available Stable Diffusion model and assess the existence of catastrophic neglect, where the model fails to generate one or more of the subjects from the input prompt. We introduce the concept of Generative Semantic Nursing (GSN), where we seek to intervene in the generative process on the fly during inference time to improve the faithfulness
  arXiv  Detail & Related papers  (2023-01-31T18:10:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.