Emergent Goal-Directed Attention in Large Vision-Language Models

• URL: http://arxiv.org/abs/2609.05517v1
• Date: Mon, 31 Aug 2026 15:00:02 GMT
• Title: Emergent Goal-Directed Attention in Large Vision-Language Models
• Abstract summary: General-purpose vision-language models (VLMs) can generate human-aligned, goal-directed spatial priorities without gaze-specific training.<n>These findings show that general-purpose VLMs can generate human-aligned, goal-directed spatial priorities without gaze-specific training.
• Score: 3.2171716419695335
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Human observers prioritize visual information according to task goals. Most computational models of naturalistic viewing are gaze-trained for free viewing, leaving open whether goal-directed attention can emerge in systems without gaze supervision. We tested two off-the-shelf vision-language models (VLMs), Qwen3-VL-32B-Thinking and Gemma-4-26B-A4B-it, on 4,887 naturalistic scenes under visual-search and free-viewing instructions. Model predictions were compared with human fixations on the same images under corresponding tasks. Both models aligned more closely with human fixations under matching goals than under mismatched goals. This crossover persisted in target-absent scenes, where alignment could not be explained by simple visual grounding, and appeared in decoder-layer readouts. Furthermore, model-thinking traces were grounded in target semantics during search and in visual prominence during free viewing. These findings show that general-purpose VLMs can generate human-aligned, goal-directed spatial priorities without gaze-specific training, informing theories of goal-directed attention and offering scalable tools for predicting where people look across tasks.

Related papers

• TDSal: Task-Based Top-Down Saliency Prediction Model [0.0]
  We propose a model that conditions visual attention on natural-language task descriptions.<n>The model produces task-dependent saliency maps that reflect how attention shifts under different viewing intents.<n>We show that incorporating explicit task semantics enables more faithful modeling of goal-directed visual attention.
  arXiv  Detail & Related papers  (2026-07-10T11:47:55Z)
• Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models [20.954224027029625]
  We present EyeVLM, a systematic evaluation framework for gaze understanding in vision-language models (VLMs)<n>To assess gaze understanding capabilities, we focus on two core tasks. The first, gaze following, predicts the 2D location where a person is looking.<n>The second, social gaze prediction, requires social and relational reasoning over multi-person interactions.
  arXiv  Detail & Related papers  (2026-05-19T13:50:40Z)
• Envision: Embodied Visual Planning via Goal-Imagery Video Diffusion [61.63215708592008]
  Embodied visual planning aims to enable manipulation tasks by imagining how a scene evolves toward a desired goal.<n>Video diffusion models provide a promising foundation for such visual imagination.<n>We propose Envision, a diffusion-based framework that performs visual planning for embodied agents.
  arXiv  Detail & Related papers  (2025-12-27T15:46:41Z)
• GoViG: Goal-Conditioned Visual Navigation Instruction Generation [69.79110149746506]
  We introduce Goal-Conditioned Visual Navigation Instruction Generation (GoViG), a new task that aims to autonomously generate precise and contextually coherent navigation instructions.<n>GoViG exclusively leverages raw egocentric visual data, substantially improving its adaptability to unseen and unstructured environments.
  arXiv  Detail & Related papers  (2025-08-13T07:05:17Z)
• From Sight to Insight: Unleashing Eye-Tracking in Weakly Supervised Video Salient Object Detection [60.11169426478452]
  This paper aims to introduce fixation information to assist the detection of salient objects under weak supervision.<n>We propose a Position and Semantic Embedding (PSE) module to provide location and semantic guidance during the feature learning process.<n>An Intra-Inter Mixed Contrastive (MCII) model improves thetemporal modeling capabilities under weak supervision.
  arXiv  Detail & Related papers  (2025-06-30T05:01:40Z)
• Imagine, Verify, Execute: Memory-guided Agentic Exploration with Vision-Language Models [81.08295968057453]
  We present IVE, an agentic exploration framework inspired by human curiosity.<n>We evaluate IVE in both simulated and real-world tabletop environments.
  arXiv  Detail & Related papers  (2025-05-12T17:59:11Z)
• VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMs [102.36953558562436]
  Vision language models (VLMs) are an exciting emerging class of language models (LMs) One understudied capability inVLMs is visual spatial planning. Our study introduces a benchmark that evaluates the spatial planning capability in these models in general.
  arXiv  Detail & Related papers  (2024-07-02T00:24:01Z)
• Top-Down Visual Attention from Analysis by Synthesis [87.47527557366593]
  We consider top-down attention from a classic Analysis-by-Synthesis (AbS) perspective of vision. We propose Analysis-by-Synthesis Vision Transformer (AbSViT), which is a top-down modulated ViT model that variationally approximates AbS, and controllable achieves top-down attention.
  arXiv  Detail & Related papers  (2023-03-23T05:17:05Z)
• Unifying Top-down and Bottom-up Scanpath Prediction Using Transformers [40.27531644565077]
  We propose the Human Attention Transformer (HAT), a single model that predicts both forms of attention control. HAT sets a new standard in computational attention, which emphasizes effectiveness, generality, and interpretability.
  arXiv  Detail & Related papers  (2023-03-16T15:13:09Z)
• Simulating Human Gaze with Neural Visual Attention [44.65733084492857]
  We propose the Neural Visual Attention (NeVA) algorithm to integrate guidance of any downstream visual task into attention modeling. We observe that biologically constrained neural networks generate human-like scanpaths without being trained for this objective.
  arXiv  Detail & Related papers  (2022-11-22T09:02:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.