GazeVLM: Active Vision via Internal Attention Control for Multimodal Reasoning

• URL: http://arxiv.org/abs/2605.07817v1
• Date: Fri, 08 May 2026 14:49:10 GMT
• Title: GazeVLM: Active Vision via Internal Attention Control for Multimodal Reasoning
• Abstract summary: We propose GazeVLM, a multimodal architecture that internalizes this metacognitive oversight over its deployment of attention resources directly into the reasoning loop.<n>By empowering the VLM to autonomously generate gaze tokens, GazeVLM establishes a top-down control mechanism over its own causal attention mask.<n>This architecture enables the model to fluidly transition between global spatial awareness and localized focal reasoning without relying on external agentic contraptions.
• Score: 12.050224516337098
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Human visual reasoning is governed by active vision, a process where metacognitive control drives top-down goal-directed attention, dynamically routing foveal focus toward task-relevant details while maintaining peripheral awareness of the global scene. In contrast, modern Vision-Language Models (VLMs) process visual information passively, relying on the static accumulation of massive token contexts that dilute spatial reasoning and induce linguistic hallucinations. Here we propose the following paradigm shift: GazeVLM, a multimodal architecture that internalizes this metacognitive oversight over its deployment of attention resources directly into the reasoning loop. By empowering the VLM to autonomously generate gaze tokens ($\texttt{<LOOK>}$), GazeVLM establishes a top-down control mechanism over its own causal attention mask. The model dynamically dictates its focal intent, triggering a continuous suppression bias that dampens irrelevant visual features, implementing spatial selective attention and simulating foveal fixation. Once local reasoning concludes, the bias lifts, seamlessly restoring the global view. This architecture enables the model to fluidly transition between global spatial awareness and localized focal reasoning without relying on external agentic contraptions like cropping tools, or inflating the context window with additional visual tokens derived from localized visual patches. Trained with a bespoke Group Relative Policy Optimization (GRPO) procedure that rewards valid grounding, our 4B-parameter GazeVLM delivers strong high-resolution multimodal reasoning performance, surpassing state-of-the-art VLMs in its parameter class by nearly 4% and agentic multimodal pipelines built around thinking with images by more than 5% on HRBench-4k and HRBench-8k.

Related papers

• OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following [59.53720386342017]
  We propose a unified vision-language framework that adapts foundational Vision-Language Models for scalable multi-person gaze reasoning.<n>By modeling all individuals, OmniGF seamlessly integrates precise spatial gaze target estimation, semantic gaze prediction, and complex social gaze reasoning.
  arXiv  Detail & Related papers  (2026-05-26T00:08:06Z)
• Learning to See What You Need: Gaze Attention for Multimodal Large Language Models [96.20985292033465]
  We introduce Gaze Attention, a novel mechanism that enables MLLMs to selectively attend to task-relevant visual regions during generation.<n>We show that Gaze Attention matches or surpasses dense-attention baselines, while using up to 90% fewer visual KV entries in the attention computation.
  arXiv  Detail & Related papers  (2026-05-13T06:54:09Z)
• Cognitive Pivot Points and Visual Anchoring: Unveiling and Rectifying Hallucinations in Multimodal Reasoning Models [65.57985131861399]
  hallucinations are strongly correlated with cognitive bifurcation points that often exhibit high entropy states.<n>We propose V-STAR, a lightweight, holistic training paradigm designed to internalize visually aware reasoning capabilities.
  arXiv  Detail & Related papers  (2026-04-11T13:59:05Z)
• V-Reflection: Transforming MLLMs from Passive Observers to Active Interrogators [43.642375673675566]
  V-Reflection is a framework that transforms the MLLM into an active interrogator through a "think-then-look" visual reflection mechanism.<n>During reasoning, latent states function as dynamic probes that actively interrogate the visual feature space.<n>V-Reflection internalizes the ability to localize task-critical evidence.
  arXiv  Detail & Related papers  (2026-03-31T03:57:56Z)
• Video-Only ToM: Enhancing Theory of Mind in Multimodal Large Language Models [40.30801020339839]
  VisionToM is a vision-oriented intervention framework designed to strengthen task-aware reasoning.<n>It computes intervention vectors that align visual representations with the correct semantic targets.<n>This guidance reduces the model's reliance on spurious linguistic priors.
  arXiv  Detail & Related papers  (2026-03-25T16:24:50Z)
• Knowledge-Guided Manipulation Using Multi-Task Reinforcement Learning [43.715148652244864]
  KG-M3PO is a framework for multi-task robotic manipulation in partially observable settings that unifies Perception, Knowledge, and Policy.<n>The method augments egocentric vision with an online 3D scene graph that grounds open-vocabulary detections into a metric, relational representation.
  arXiv  Detail & Related papers  (2026-03-25T08:41:32Z)
• Perception-Aware Multimodal Spatial Reasoning from Monocular Images [57.42071289037214]
  spatial reasoning from monocular images is essential for autonomous driving.<n>Current Vision-Language Models (VLMs) still struggle with fine-grained geometric perception.<n>We propose a perception-aware multimodal reasoning framework that equipsVLMs with explicit object-centric grounding ability.
  arXiv  Detail & Related papers  (2026-03-07T02:05:12Z)
• TagaVLM: Topology-Aware Global Action Reasoning for Vision-Language Navigation [70.23578202012048]
  Vision-Language Navigation (VLN) presents a unique challenge for Large Vision-Language Models (VLMs) due to their inherent architectural mismatch.<n>We propose TagaVLM (Topology-Aware Global Action reasoning), an end-to-end framework that explicitly injects topological structures into the VLM backbone.<n>To enhance topological node information, an Interleaved Navigation Prompt strengthens node-level visual-text alignment.<n>With the embedded topological graph, the model is capable of global action reasoning, allowing for robust path correction.
  arXiv  Detail & Related papers  (2026-03-03T13:28:07Z)
• VLM4D: Towards Spatiotemporal Awareness in Vision Language Models [66.833085504228]
  We introduce V4DLM, the first benchmark specifically designed to evaluate visual language models (VLMs)<n>Our benchmark comprises diverse real-world and synthetic videos accompanied by carefully curated question-answer pairs.<n>We identify significant performance gaps compared to human baselines, highlighting fundamental deficiencies in existing models.
  arXiv  Detail & Related papers  (2025-08-04T06:06:06Z)
• DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge [41.030494146004806]
  We propose DreamVLA, a novel VLA framework that integrates comprehensive world knowledge forecasting to enable inverse dynamics modeling.<n>DreamVLA introduces a dynamic-region-guided world knowledge prediction, integrated with the spatial and semantic cues, which provide compact yet comprehensive representations for action planning.<n>Experiments on both real-world and simulation environments demonstrate that DreamVLA achieves 76.7% success rate on real robot tasks.
  arXiv  Detail & Related papers  (2025-07-06T16:14:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.