Beyond Scanpaths: Graph-Based Gaze Simulation in Dynamic Scenes

• URL: http://arxiv.org/abs/2603.28319v1
• Date: Mon, 30 Mar 2026 11:41:11 GMT
• Title: Beyond Scanpaths: Graph-Based Gaze Simulation in Dynamic Scenes
• Abstract summary: We formulate gaze modelling as an autoregressive dynamical system and explicitly unroll raw gaze trajectories over time.<n>We also release Focus100, a new dataset of raw gaze data from 30 participants viewing egocentric driving footage.
• Score: 0.2039123720459736
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Accurately modelling human attention is essential for numerous computer vision applications, particularly in the domain of automotive safety. Existing methods typically collapse gaze into saliency maps or scanpaths, treating gaze dynamics only implicitly. We instead formulate gaze modelling as an autoregressive dynamical system and explicitly unroll raw gaze trajectories over time, conditioned on both gaze history and the evolving environment. Driving scenes are represented as gaze-centric graphs processed by the Affinity Relation Transformer (ART), a heterogeneous graph transformer that models interactions between driver gaze, traffic objects, and road structure. We further introduce the Object Density Network (ODN) to predict next-step gaze distributions, capturing the stochastic and object-centric nature of attentional shifts in complex environments. We also release Focus100, a new dataset of raw gaze data from 30 participants viewing egocentric driving footage. Trained directly on raw gaze, without fixation filtering, our unified approach produces more natural gaze trajectories, scanpath dynamics, and saliency maps than existing attention models, offering valuable insights for the temporal modelling of human attention in dynamic environments.

Related papers

• DyG$^2$T: Modeling Object Dynamics with 3D Gaussian Temporal-Spatial Particle Graph Transformer [86.4618122245946]
  DyG$2$T is a dynamics modeling framework that infers object motion trajectories by spatially completing and temporally discriminating Key Point representations.<n>Experiments on both synthetic and real-world datasets demonstrate that DyG$2$T achieves accurate dynamics modeling and reasoning.
  arXiv  Detail & Related papers  (2026-08-19T03:44:28Z)
• Cortical Policy: A Dual-Stream View Transformer for Robotic Manipulation [57.28703268044067]
  We propose Cortical Policy, a novel dual-stream view transformer for robotic manipulation.<n>Our framework offers a fresh perspective for robotic manipulation and holds potential for broader application in vision-based robot control.
  arXiv  Detail & Related papers  (2026-03-22T04:18:54Z)
• Eye Gaze-Informed and Context-Aware Pedestrian Trajectory Prediction in Shared Spaces with Automated Shuttles: A Virtual Reality Study [12.640255397914018]
  This paper presents a Virtual Reality (VR) study that captures how pedestrians interact with automated shuttles across diverse scenarios.<n>We identify critical behavior patterns present in pedestrians' decision-making in shared spaces, including hesitation, evasive maneuvers, gaze allocation, and proxemic adjustments.<n>We propose GazeX-LSTM, a multimodal eye gaze-informed and context-aware prediction model that integrates pedestrians' trajectories, fine-grained eye gaze dynamics, and contextual factors.
  arXiv  Detail & Related papers  (2026-03-20T09:57:23Z)
• ARGaze: Autoregressive Transformers for Online Egocentric Gaze Estimation [46.30718574969354]
  egocentric gaze estimation predicts where a camera wearer is looking from first-person video using only past and current frames.<n>We propose ARGaze, which reformulates gaze estimation as sequential prediction.<n>We achieve state-of-the-art performance across multiple egocentric benchmarks under online evaluation.
  arXiv  Detail & Related papers  (2026-02-04T23:33:16Z)
• Foraging with the Eyes: Dynamics in Human Visual Gaze and Deep Predictive Modeling [0.0]
  Animals often forage via Levy walks with heavy tailed step lengths optimized for sparse resource environments.<n>We show that human visual gaze follows similar dynamics when images.<n>Our findings present new evidence that human visual exploration obeys statistical laws to natural foraging and open avenues for modeling gaze through generative and predictive frameworks.
  arXiv  Detail & Related papers  (2025-10-10T11:45:51Z)
• DiffEye: Diffusion-Based Continuous Eye-Tracking Data Generation Conditioned on Natural Images [24.810828226931605]
  DiffEye is a diffusion-based training framework designed to model continuous and diverse eye movement trajectories during free viewing of natural images.<n>By leveraging raw eye-tracking trajectories rather than relying on scanpaths, DiffEye captures the inherent variability in human gaze behavior.<n>The generated trajectories can also be converted into scanpaths and saliency maps, resulting in outputs that more accurately reflect the distribution of human visual attention.
  arXiv  Detail & Related papers  (2025-09-20T18:20:51Z)
• Modeling Human Gaze Behavior with Diffusion Models for Unified Scanpath Prediction [66.71402249062777]
  We present ScanDiff, a novel architecture that combines diffusion models with Vision Transformers to generate diverse and realistic scanpaths.<n>Our method explicitly models scanpath variability by leveraging the nature of diffusion models, producing a wide range of plausible gaze trajectories.<n>Experiments on benchmark datasets show that ScanDiff surpasses state-of-the-art methods in both free-viewing and task-driven scenarios.
  arXiv  Detail & Related papers  (2025-07-30T18:36:09Z)
• Street-View Image Generation from a Bird's-Eye View Layout [95.36869800896335]
  Bird's-Eye View (BEV) Perception has received increasing attention in recent years. Data-driven simulation for autonomous driving has been a focal point of recent research. We propose BEVGen, a conditional generative model that synthesizes realistic and spatially consistent surrounding images.
  arXiv  Detail & Related papers  (2023-01-11T18:39:34Z)
• Predicting Future Occupancy Grids in Dynamic Environment with Spatio-Temporal Learning [63.25627328308978]
  We propose a-temporal prediction network pipeline to generate future occupancy predictions. Compared to current SOTA, our approach predicts occupancy for a longer horizon of 3 seconds. We publicly release our grid occupancy dataset based on nulis to support further research.
  arXiv  Detail & Related papers  (2022-05-06T13:45:32Z)
• TRiPOD: Human Trajectory and Pose Dynamics Forecasting in the Wild [77.59069361196404]
  TRiPOD is a novel method for predicting body dynamics based on graph attentional networks. To incorporate a real-world challenge, we learn an indicator representing whether an estimated body joint is visible/invisible at each frame. Our evaluation shows that TRiPOD outperforms all prior work and state-of-the-art specifically designed for each of the trajectory and pose forecasting tasks.
  arXiv  Detail & Related papers  (2021-04-08T20:01:00Z)
• SCOUT: Socially-COnsistent and UndersTandable Graph Attention Network for Trajectory Prediction of Vehicles and VRUs [0.0]
  SCOUT is a novel Attention-based Graph Neural Network that uses a flexible and generic representation of the scene as a graph. We explore three different attention mechanisms and test our scheme with both bird-eye-view and on-vehicle urban data. We evaluate our model's flexibility and transferability by testing it under completely new scenarios on RounD dataset.
  arXiv  Detail & Related papers  (2021-02-12T06:29:28Z)
• Implicit Latent Variable Model for Scene-Consistent Motion Forecasting [78.74510891099395]
  In this paper, we aim to learn scene-consistent motion forecasts of complex urban traffic directly from sensor data. We model the scene as an interaction graph and employ powerful graph neural networks to learn a distributed latent representation of the scene.
  arXiv  Detail & Related papers  (2020-07-23T14:31:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.