Listening makes Vision Clear for VLMs

• URL: http://arxiv.org/abs/2606.23763v1
• Date: Mon, 22 Jun 2026 14:14:46 GMT
• Title: Listening makes Vision Clear for VLMs
• Abstract summary: We adopt prompt-side semantics and propose Prompt-Vision Token Activation Map (PV-TAM)<n>PV-TAM incorporates a filter to remove systematic bias induced by modality boundary markers.<n>It consistently improves both attention-based and IoU-style localization metrics over answer-side baselines on various datasets.
• Score: 10.154962172624954
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent work typically assesses vision--language consistency using attention distributions of answer-side tokens. However, we observe that highest attention regions are not always consistent with the intended semantic token. This probably stems from decoding drift, where language priors from previously generated answer tokens accumulate and mismatch with visual attention. Besides the priors from previous answer tokens, we find that structural tokens, e.g., modality boundary markers, may encompass the entire context and generate high attention to areas unrelated to the target. To avoid these distortions and provide consistency evaluation for large VLMs, we adopt prompt-side semantics and propose Prompt-Vision Token Activation Map (PV-TAM). PV-TAM further incorporates a filter to remove systematic bias induced by modality boundary markers. Unlike traditional methods that evaluate overlap solely through masks while ignoring activation intensity, our metrics leverage the peak distribution of attention to measure the alignment between prompts and visual regions. In experiments, PV-TAM consistently improves both attention-based and IoU-style localization metrics over answer-side baselines on various datasets.

Related papers

• Disentangling Semantic Attention from Structural Bias in the Attention Manifold [61.68044165974505]
  Multimodal Large Language Models (MLLMs) exhibit disproportionate attention toward certain semantically uninformative visual tokens.<n>We introduce Saliency-guided Purification and Adaptive Redistribution (SPAR), a training-free, plug-and-play intervention.
  arXiv  Detail & Related papers  (2026-07-27T05:30:19Z)
• Token-Based Affordance Grounding with Large Vision-Language Models [35.91429072589231]
  Affordance grounding aims to localize image regions that support a specific action.<n>TokAG exploits the token-level semantic-spatial signals in LVLMs to localize action-relevant regions without external supervision.
  arXiv  Detail & Related papers  (2026-07-03T20:34:42Z)
• See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs [81.71384150441163]
  We propose Context-aware Attention Intervention (CAI) for large vision-language models (LVLMs)<n>CAI enforces a see only when needed principle via two-axis selectivity: where to look and when to intervene.<n> experiments across multiple LVLM backbones and benchmarks show that CAI achieves state-of-the-art hallucination mitigation.
  arXiv  Detail & Related papers  (2026-06-29T06:35:47Z)
• Learning to See What You Need: Gaze Attention for Multimodal Large Language Models [96.20985292033465]
  We introduce Gaze Attention, a novel mechanism that enables MLLMs to selectively attend to task-relevant visual regions during generation.<n>We show that Gaze Attention matches or surpasses dense-attention baselines, while using up to 90% fewer visual KV entries in the attention computation.
  arXiv  Detail & Related papers  (2026-05-13T06:54:09Z)
• Aligning What Vision-Language Models See and Perceive with Adaptive Information Flow [32.21470980856891]
  Vision-Language Models (VLMs) have demonstrated strong capability in a wide range of tasks such as visual recognition, document parsing, and visual grounding.<n>Recent work shows that while VLMs often manage to capture the correct image region corresponding to the question, they do not necessarily produce the correct answers.<n>We show that this misalignment could be attributed to suboptimal information flow within VLMs, where text tokens distribute too much attention to irrelevant visual tokens.
  arXiv  Detail & Related papers  (2026-04-17T08:07:22Z)
• From Drop-off to Recovery: A Mechanistic Analysis of Segmentation in MLLMs [50.185593677108436]
  Multimodal Large Language Models (MLLMs) are increasingly applied to pixel-level vision tasks, yet their intrinsic capacity for spatial understanding remains poorly understood.<n>We investigate segmentation capacity through a layerwise linear probing evaluation across the entire MLLM pipeline: vision encoder, adapter, and LLM.
  arXiv  Detail & Related papers  (2026-03-18T00:22:15Z)
• ITSELF: Attention Guided Fine-Grained Alignment for Vision-Language Retrieval [7.809330758164896]
  We introduce ITSELF, an attention-guided framework for implicit local alignment.<n>At its core, Guided Representation with Attentive Bank (GRAB) converts the model's own attention into an Attentive Bank of high-saliency tokens.<n>We also introduce Multi-Layer Attention for Robust Selection (MARS), which aggregates attention across layers and performs diversity-aware top-k selection.
  arXiv  Detail & Related papers  (2026-01-03T01:19:36Z)
• Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigation [8.805397340243557]
  Vision language models (VLMs) often generate hallucination, i.e., content that cannot be substantiated by visual inputs.<n>We propose a method called Gaze Shift-Guided Cross-modal Fusion Enhancement (GIFT) to mitigate hallucination.
  arXiv  Detail & Related papers  (2025-10-24T23:04:26Z)
• Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas [69.56484419619919]
  We study the spatial reasoning challenge from the lens of mechanistic interpretability.<n>We observe that successful spatial reasoning correlates strongly with the model's ability to align its attention with actual object locations.<n>Motivated by these findings, we propose ADAPTVIS to sharpen the attention on highly relevant regions when confident.
  arXiv  Detail & Related papers  (2025-03-03T17:57:03Z)
• Advancing General Multimodal Capability of Vision-language Models with Pyramid-descent Visual Position Encoding [64.29499221878746]
  Vision-language Models (VLMs) have shown remarkable capabilities in advancing general artificial intelligence.<n>PyPE is a novel approach designed to enhance the perception of visual tokens withinVLMs.<n>Our method reduces the relative distance between interrelated visual elements and instruction tokens.
  arXiv  Detail & Related papers  (2025-01-19T07:00:46Z)
• Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Models [16.185253476874006]
  Large Vision Language Models (LVLMs) demonstrate strong capabilities in visual understanding and description, yet often suffer from hallucinations.<n>We propose Attentional Vision (AvisC), a test-time approach that recalibrates the influence of blind tokens without modifying the underlying attention mechanism.<n>Experiments on standard benchmarks, including POPE, MME, and AMBER, demonstrate that AvisC effectively reduces hallucinations in LVLMs.
  arXiv  Detail & Related papers  (2024-05-28T04:40:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.