Learning to See What You Need: Gaze Attention for Multimodal Large Language Models

• URL: http://arxiv.org/abs/2605.13080v1
• Date: Wed, 13 May 2026 06:54:09 GMT
• Title: Learning to See What You Need: Gaze Attention for Multimodal Large Language Models
• Abstract summary: We introduce Gaze Attention, a novel mechanism that enables MLLMs to selectively attend to task-relevant visual regions during generation.<n>We show that Gaze Attention matches or surpasses dense-attention baselines, while using up to 90% fewer visual KV entries in the attention computation.
• Score: 96.20985292033465
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: When humans describe a visual scene, they do not process the entire image uniformly; instead, they selectively fixate on regions relevant to their intended description. In contrast, current multimodal large language models (MLLMs) attend to all visual tokens at each generation step, leading to diluted focus and unnecessary computational overhead. In this work, we introduce Gaze Attention, a novel mechanism that enables MLLMs to selectively attend to task-relevant visual regions during generation. Specifically, we spatially group visual embeddings-stored as key-value caches-into compact gaze regions, each represented by a lightweight descriptor. At each decoding step, the model dynamically selects the most relevant regions and restricts attention to them, reducing redundant computation while enhancing focus. To mitigate the loss of global context caused by localized attention, we further propose learnable context tokens appended to each image or frame, allowing the model to maintain holistic visual awareness. Extensive experiments on image and video understanding benchmarks demonstrate that Gaze Attention matches or surpasses dense-attention baselines, while using up to 90% fewer visual KV entries in the attention computation.

Related papers

• Token-Based Affordance Grounding with Large Vision-Language Models [35.91429072589231]
  Affordance grounding aims to localize image regions that support a specific action.<n>TokAG exploits the token-level semantic-spatial signals in LVLMs to localize action-relevant regions without external supervision.
  arXiv  Detail & Related papers  (2026-07-03T20:34:42Z)
• OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following [59.53720386342017]
  We propose a unified vision-language framework that adapts foundational Vision-Language Models for scalable multi-person gaze reasoning.<n>By modeling all individuals, OmniGF seamlessly integrates precise spatial gaze target estimation, semantic gaze prediction, and complex social gaze reasoning.
  arXiv  Detail & Related papers  (2026-05-26T00:08:06Z)
• Steerable Visual Representations [72.39044430620977]
  We introduce Steerable Visual Representations, whose global and local features can be steered with natural language.<n>Our method also matches or outperforms dedicated approaches on anomaly detection and personalized object discrimination.
  arXiv  Detail & Related papers  (2026-04-02T17:59:49Z)
• VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactions [51.41587958253802]
  We introduce VISion On Request (VISOR), a method that reduces inference cost without discarding visual information.<n>VISOR improves efficiency by sparsifying the interaction between image and text tokens.<n>Experiments show that VISOR drastically reduces computational cost while matching or exceeding state-of-the-art results.
  arXiv  Detail & Related papers  (2026-03-24T17:58:17Z)
• Tinted Frames: Question Framing Blinds Vision-Language Models [29.78944164519993]
  Vision-Language Models (VLMs) have been shown to be blind, often underutilizing their visual inputs even on tasks that require visual reasoning.<n>We quantify how framing alters both the amount and distribution of attention over the image.<n>We introduce a lightweight prompt-tuning method using learnable tokens that encourages the robust, visually grounded attention patterns observed in open-ended settings.
  arXiv  Detail & Related papers  (2026-03-19T17:53:09Z)
• Attention Guided Alignment in Efficient Vision-Language Models [56.20286899428444]
  Large Vision-Language Models (VLMs) rely on effective multimodal alignment between pre-trained vision encoders and Large Language Models (LLMs)<n>This paper presents a comprehensive analysis of attention patterns in efficient VLMs.<n>We introduce Attention-Guided Efficient Vision-Language Models (AGE-VLM), a novel framework that enhances visual grounding through interleaved cross-attention layers.
  arXiv  Detail & Related papers  (2025-11-21T21:36:48Z)
• Zoomer: Adaptive Image Focus Optimization for Black-box MLLM [45.40963536739482]
  SysName is a novel visual prompting mechanism designed to enhance MLLM performance while preserving essential visual details within token limits.<n>SysName consistently outperforms baseline methods, achieving up to a $26.9%$ improvement in accuracy while significantly reducing token consumption.
  arXiv  Detail & Related papers  (2025-04-30T02:51:10Z)
• See What You Are Told: Visual Attention Sink in Large Multimodal Models [4.024850952459758]
  Large multimodal models (LMMs) "see" images by leveraging the attention mechanism between text and visual tokens in the transformer decoder.<n>Recent findings indicate that LMMs have an extraordinary tendency to consistently allocate high attention weights to specific visual tokens.<n>We introduce Visual Attention Redistribution ( VAR), a method that redistributes attention in image-centric heads.
  arXiv  Detail & Related papers  (2025-03-05T09:55:07Z)
• PAINT: Paying Attention to INformed Tokens to Mitigate Hallucination in Large Vision-Language Model [0.0]
  hallucinations often arise from the progressive weakening of attention weight to visual tokens.<n>textbfPAINT (textbfPaying textbfAttention to textbfINformed textbfTokens) is a plug-and-play framework that intervenes in the self-attention mechanism of the Large Vision Language Models.
  arXiv  Detail & Related papers  (2025-01-21T15:22:31Z)
• What does CLIP know about a red circle? Visual prompt engineering for VLMs [116.8806079598019]
  We explore the idea of visual prompt engineering for solving computer vision tasks beyond classification by editing in image space instead of text. We show the power of this simple approach by achieving state-of-the-art in zero-shot referring expressions comprehension and strong performance in keypoint localization tasks.
  arXiv  Detail & Related papers  (2023-04-13T17:58:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.