iTrace: Click-Based Gaze Visualization on the Apple Vision Pro

• URL: http://arxiv.org/abs/2508.12268v1
• Date: Sun, 17 Aug 2025 07:34:37 GMT
• Title: iTrace: Click-Based Gaze Visualization on the Apple Vision Pro
• Abstract summary: The Apple Vision Pro is equipped with accurate eye-tracking capabilities, yet the privacy restrictions on the device prevent direct access to continuous user gaze data.<n>This study introduces iTrace, a novel application that overcomes these limitations through click-based gaze extraction techniques.<n>We developed a system with a client-server architecture that captures the gaze coordinates and transforms them into dynamic heatmaps for video and spatial eye tracking.
• Score: 2.4150871564195007
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The Apple Vision Pro is equipped with accurate eye-tracking capabilities, yet the privacy restrictions on the device prevent direct access to continuous user gaze data. This study introduces iTrace, a novel application that overcomes these limitations through click-based gaze extraction techniques, including manual methods like a pinch gesture, and automatic approaches utilizing dwell control or a gaming controller. We developed a system with a client-server architecture that captures the gaze coordinates and transforms them into dynamic heatmaps for video and spatial eye tracking. The system can generate individual and averaged heatmaps, enabling analysis of personal and collective attention patterns. To demonstrate its effectiveness and evaluate the usability and performance, a study was conducted with two groups of 10 participants, each testing different clicking methods. The 8BitDo controller achieved higher average data collection rates at 14.22 clicks/s compared to 0.45 clicks/s with dwell control, enabling significantly denser heatmap visualizations. The resulting heatmaps reveal distinct attention patterns, including concentrated focus in lecture videos and broader scanning during problem-solving tasks. By allowing dynamic attention visualization while maintaining a high gaze precision of 91 %, iTrace demonstrates strong potential for a wide range of applications in educational content engagement, environmental design evaluation, marketing analysis, and clinical cognitive assessment. Despite the current gaze data restrictions on the Apple Vision Pro, we encourage developers to use iTrace only in research settings.

Related papers

• EyeTheia: A Lightweight and Accessible Eye-Tracking Toolbox [0.0]
  EyeTheia is a lightweight and open deep learning pipeline for webcam-based gaze estimation.<n>It enables real-time gaze tracking using only a standard laptop webcam.<n>It combines MediaPipe-based landmark extraction with a convolutional neural network inspired by iTracker and optional user-specific fine-tuning.
  arXiv  Detail & Related papers  (2026-01-09T19:49:01Z)
• Video Diffusion Models Excel at Tracking Similar-Looking Objects Without Supervision [80.2787127430925]
  Distinguishing visually similar objects by their motion remains a critical challenge in computer vision.<n>We find that pre-trained video diffusion models inherently learn motion representations suitable for tracking without task-specific training.<n>Our method achieves up to a 6-point improvement over recent self-supervised approaches on established benchmarks.
  arXiv  Detail & Related papers  (2025-12-02T02:17:34Z)
• GazeProphetV2: Head-Movement-Based Gaze Prediction Enabling Efficient Foveated Rendering on Mobile VR [0.0]
  This paper introduces a multimodal approach to VR gaze prediction that combines temporal gaze patterns, head movement data, and visual scene information.<n> Evaluations using a dataset spanning 22 VR scenes with 5.3M gaze samples show improvements in predictive accuracy when combining modalities.<n>Cross-scene generalization testing shows consistent performance with 93.1% validation accuracy and temporal consistency in predicted gaze trajectories.
  arXiv  Detail & Related papers  (2025-11-25T06:55:39Z)
• Top-Down Compression: Revisit Efficient Vision Token Projection for Visual Instruction Tuning [70.57180215148125]
  Visual instruction tuning aims to enable large language models to comprehend the visual world.<n>Existing methods often grapple with the intractable trade-off between accuracy and efficiency.<n>We present LLaVA-Meteor, a novel approach that strategically compresses visual tokens without compromising core information.
  arXiv  Detail & Related papers  (2025-05-17T10:22:29Z)
• A Cross-Scene Benchmark for Open-World Drone Active Tracking [54.235808061746525]
  Drone Visual Active Tracking aims to autonomously follow a target object by controlling the motion system based on visual observations.<n>We propose a unified cross-scene cross-domain benchmark for open-world drone active tracking called DAT.<n>We also propose a reinforcement learning-based drone tracking method called R-VAT.
  arXiv  Detail & Related papers  (2024-12-01T09:37:46Z)
• Realtime Dynamic Gaze Target Tracking and Depth-Level Estimation [6.435984242701043]
  Transparent Displays (TD) in various applications, such as Heads-Up Displays (HUDs) in vehicles, is a burgeoning field, poised to revolutionize user experiences. This innovation brings forth significant challenges in realtime human-device interaction, particularly in accurately identifying and tracking a user's gaze on dynamically changing TDs. We present a two-fold robust and efficient systematic solution for realtime gaze monitoring, comprised of: (1) a tree-based algorithm for identifying and dynamically tracking gaze targets; and (2) a multi-stream self-attention architecture to estimate the depth-level of human gaze from eye tracking data.
  arXiv  Detail & Related papers  (2024-06-09T20:52:47Z)
• Spatio-Temporal Attention and Gaussian Processes for Personalized Video Gaze Estimation [7.545077734926115]
  We propose a simple and novel deep learning model designed to estimate gaze from videos. Our method employs a spatial attention mechanism that tracks spatial dynamics within videos. Experimental results confirm the efficacy of the proposed approach, demonstrating its success in both within-dataset and cross-dataset settings.
  arXiv  Detail & Related papers  (2024-04-08T06:07:32Z)
• Decoding Attention from Gaze: A Benchmark Dataset and End-to-End Models [6.642042615005632]
  Eye-tracking has potential to provide rich behavioral data about human cognition in ecologically valid environments. This paper studies using computer vision tools for "attention decoding", the task of assessing the locus of a participant's overt visual attention over time.
  arXiv  Detail & Related papers  (2022-11-20T12:24:57Z)
• Video Annotation for Visual Tracking via Selection and Refinement [74.08109740917122]
  We present a new framework to facilitate bounding box annotations for video sequences. A temporal assessment network is proposed which is able to capture the temporal coherence of target locations. A visual-geometry refinement network is also designed to further enhance the selected tracking results.
  arXiv  Detail & Related papers  (2021-08-09T05:56:47Z)
• Visual Distant Supervision for Scene Graph Generation [66.10579690929623]
  Scene graph models usually require supervised learning on large quantities of labeled data with intensive human annotation. We propose visual distant supervision, a novel paradigm of visual relation learning, which can train scene graph models without any human-labeled data. Comprehensive experimental results show that our distantly supervised model outperforms strong weakly supervised and semi-supervised baselines.
  arXiv  Detail & Related papers  (2021-03-29T06:35:24Z)
• Towards Hardware-Agnostic Gaze-Trackers [0.5512295869673146]
  We present a deep neural network architecture as an appearance-based method for constrained gaze-tracking. Our system achieved an error of 1.8073cm on GazeCapture dataset without any calibration or device specific fine-tuning.
  arXiv  Detail & Related papers  (2020-10-11T00:53:57Z)
• Towards End-to-end Video-based Eye-Tracking [50.0630362419371]
  Estimating eye-gaze from images alone is a challenging task due to un-observable person-specific factors. We propose a novel dataset and accompanying method which aims to explicitly learn these semantic and temporal relationships. We demonstrate that the fusion of information from visual stimuli as well as eye images can lead towards achieving performance similar to literature-reported figures.
  arXiv  Detail & Related papers  (2020-07-26T12:39:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.