DeepGaze3.5-VL: Modeling Scanpaths via Autoregressive Token Prediction

• URL: http://arxiv.org/abs/2607.02083v1
• Date: Thu, 02 Jul 2026 12:25:45 GMT
• Title: DeepGaze3.5-VL: Modeling Scanpaths via Autoregressive Token Prediction
• Abstract summary: DeepGaze3.5-VL establishes a new state-of-the-art across multiple datasets, achieving 2.18 bits of Information Gain (IG) on MIT1003, a 46% improvement over DeepGaze III.<n>Our generative framework serves as a powerful computational tool for direct behavioral interventions, allowing for controlled in-silico simulations that would be experimentally difficult or impossible to conduct in vivo.
• Score: 21.818459524136376
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding human visual attention on a scene over time has applications in domains such as interface design and inferring cognitive states. Modeling visual scanpaths has historically relied on specialized architectures with hand-crafted priors. While these architectures can model fixation sequences, their rigid structural biases restrict easy extendability and flexible conditioning. For instance, integrating task-specific instructions or adapting to distinct viewer identities requires custom, disjoint architectural additions. We frame scanpath prediction purely as a discrete sequence modeling task. By mapping coordinates into a text vocabulary, we leverage the pretrained representations of Vision-Language Models. This framing absorbs diverse factors of variation: simple prompting allows for global conditioning, such as providing viewer identities to capture personalized biases, or task-specific objectives like visual search. The framework can also integrate per-fixation attributes, such as individual fixation durations, alongside spatial locations. The autoregressive alignment enables the scalable, exact computation of per-fixation log-likelihoods, directly equivalent to the commonly used Information Gain (IG) metric. Our model, DeepGaze3.5-VL, establishes a new state-of-the-art across multiple datasets, achieving 2.18 bits of IG on MIT1003, a 46% improvement over DeepGaze III. This advantage persists even when baselines use identical high-capacity vision encoders. Beyond predictive performance, our generative framework serves as a powerful computational tool for direct behavioral interventions, allowing for controlled in-silico simulations that would be experimentally difficult or impossible to conduct in vivo. We demonstrate this ability by performing controlled interventions on the durations of pre-saccadic fixations, recovering known oculomotor phenomena purely from data.

Related papers

• World Models for Embodied Intelligence: From Plausible to Controllable to Actionable [100.13060753774657]
  World models connect perception and decision-making in embodied intelligence by maintaining hidden state, anticipating consequences, comparing interventions, and adapting when execution departs from expectations.<n>This raises a central question: which predictive capabilities improve behavior?<n>We introduce three progressively stronger capability levels: Plausible models preserve task-relevant temporal, geometric, or physical structure; Controllable models additionally predict how interventions alter that structure; and Actionable models translate predictions into measurable gains in planning, action, learning, evaluation, verification, recovery, or data selection.
  arXiv  Detail & Related papers  (2026-09-15T06:22:42Z)
• SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning [99.49739831030388]
  We study how the design of this interface shapes the agent's capacity for open-ended spatial reasoning.<n>We propose SpatialClaw, a training-free framework for spatial reasoning that adopts code as the action interface.
  arXiv  Detail & Related papers  (2026-06-11T17:59:36Z)
• AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding [55.535374902204296]
  AffordanceVLA is a unified framework that introduces structured affordance forecasting as a task-oriented intermediate representation.<n>We show that AffordanceVLA achieves strong performance across diverse manipulation scenarios.
  arXiv  Detail & Related papers  (2026-06-04T13:28:51Z)
• From Scene to Object: Text-Guided Dual-Gaze Prediction [17.32439183328327]
  Interpretable driver attention prediction is crucial for human-like autonomous driving.<n>Existing datasets provide only scene-level global gaze rather than fine-grained object-level annotations.<n>This paper proposes a novel dual-branch gaze prediction framework, establishing a complete paradigm from data construction to model architecture.
  arXiv  Detail & Related papers  (2026-04-22T05:11:59Z)
• BEVPredFormer: Spatio-temporal Attention for BEV Instance Prediction in Autonomous Driving [1.5327485092552822]
  We introduce BEVPredFormer, a novel camera-only architecture for BEV prediction.<n>It uses attention-based temporal processing to improve temporal and spatial comprehension of the scene.<n>It was evaluated on the nuScenes dataset and was on par or surpassed State-Of-The-Art methods.
  arXiv  Detail & Related papers  (2026-04-03T09:58:42Z)
• A Scene Graph Backed Approach to Open Set Semantic Mapping [1.8266902367595235]
  Open Set Semantic Mapping and 3D Scene Graphs (3DSSGs) are established paradigms in robotic perception.<n>We propose a mapping architecture where the 3DSSG serves as the foundational backend, acting as the primary knowledge representation.<n>This ensures that the map remains topologically consistent and computationally efficient, even during extended operations in large-scale settings.
  arXiv  Detail & Related papers  (2026-02-03T17:41:51Z)
• ArtiSG: Functional 3D Scene Graph Construction via Human-demonstrated Articulated Objects Manipulation [51.54082859171464]
  ArtiSG is a framework that constructs functional 3D scene graphs by encoding human demonstrations into structured robotic memory.<n>We show that ArtiSG significantly outperforms baselines in functional element recall and articulation estimation precision.
  arXiv  Detail & Related papers  (2025-12-31T13:10:40Z)
• Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policy [73.75271615101754]
  We present Dita, a scalable framework that leverages Transformer architectures to directly denoise continuous action sequences.<n>Dita employs in-context conditioning -- enabling fine-grained alignment between denoised actions and raw visual tokens from historical observations.<n>Dita effectively integrates cross-embodiment datasets across diverse camera perspectives, observation scenes, tasks, and action spaces.
  arXiv  Detail & Related papers  (2025-03-25T15:19:56Z)
• Diffusion Transformer Policy [48.50988753948537]
  We propose a large multi-modal diffusion transformer, dubbed as Diffusion Transformer Policy, to model continuous end-effector actions.<n>By leveraging the scaling capability of transformers, the proposed approach can effectively model continuous end-effector actions across large diverse robot datasets.
  arXiv  Detail & Related papers  (2024-10-21T12:43:54Z)
• Conditioned Human Trajectory Prediction using Iterative Attention Blocks [70.36888514074022]
  We present a simple yet effective pedestrian trajectory prediction model aimed at pedestrians positions prediction in urban-like environments. Our model is a neural-based architecture that can run several layers of attention blocks and transformers in an iterative sequential fashion. We show that without explicit introduction of social masks, dynamical models, social pooling layers, or complicated graph-like structures, it is possible to produce on par results with SoTA models.
  arXiv  Detail & Related papers  (2022-06-29T07:49:48Z)
• Deep Neural Dynamic Bayesian Networks applied to EEG sleep spindles modeling [0.0]
  We propose a generative model for single-channel EEG that incorporates the constraints experts actively enforce during visual scoring. We derive algorithms for exact, tractable inference as a special case of Generalized Expectation Maximization. We validate the model on three public datasets and provide support that more complex models are able to surpass state-of-the-art detectors.
  arXiv  Detail & Related papers  (2020-10-16T21:48:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.