GazePrior: Zero-Shot AR/VR Eye Tracking via Learned 3D Gaze Reconstruction

• URL: http://arxiv.org/abs/2605.22359v1
• Date: Thu, 21 May 2026 11:54:01 GMT
• Title: GazePrior: Zero-Shot AR/VR Eye Tracking via Learned 3D Gaze Reconstruction
• Abstract summary: We introduce a data-driven 3D prior that models the distribution of human eyes across diverse identities, gaze directions, and light settings.<n>This model then enables sparse-input 3D reconstruction of annotated data collected with previous ET devices.<n>Our approach synthesizes data with the realism, diversity and ground-truth accuracy of real data collection without its prohibitive costs.
• Score: 60.68732444876243
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Eye tracking (ET) is a foundational technology for advanced AR/VR applications. However, training ET models for every new ET device is challenging: real data collection is costly and time-consuming, while existing synthetic data generation methods lack realism. To remove the need for additional data collection while maintaining data quality, we introduce a data-driven 3D prior that models the distribution of human eyes across diverse identities, gaze directions, and light settings. This model, which we coin GazePrior, then enables sparse-input 3D reconstruction of annotated data collected with previous ET devices, which can in turn be rendered from the cameras of any target ET device. Our approach synthesizes data with the realism, diversity and ground-truth accuracy of real data collection without its prohibitive costs. Our experiments demonstrate that ET models trained with our synthesized data outperform previous zero-shot methods, achieving higher accuracy and robustness.

Related papers

• OASIS: From Simulation Data Collection to Real-World Humanoid Loco-Manipulation [51.37436817948304]
  OASIS is a simulation-data-driven framework for humanoid loco-manipulation.<n>It reconstructs realistic object assets from real-world images using a 3D generative model.<n>Under zero-shot deployment, it achieves higher success rates on most tasks than that trained on real-robot teleoperation data.
  arXiv  Detail & Related papers  (2026-06-07T10:01:53Z)
• Mirage2Matter: A Physically Grounded Gaussian World Model from Video [87.9732484393686]
  We present Simulate Anything, a graphics-driven world modeling and simulation framework.<n>Our approach reconstructs real-world environments into a photorealistic scene representation using 3D Gaussian Splatting (3DGS)<n>We then leverage generative models to recover a physically realistic representation and integrate it into a simulation environment via a precision calibration target.
  arXiv  Detail & Related papers  (2026-01-24T07:43:57Z)
• R3ST: A Synthetic 3D Dataset With Realistic Trajectories [7.7401133539188365]
  We introduce R3ST (Realistic 3D Synthetic Trajectories), a synthetic dataset that overcomes the lack of realistic vehicle motion.<n>The proposed dataset closes the gap between synthetic data and realistic trajectories.
  arXiv  Detail & Related papers  (2025-12-18T17:18:45Z)
• The Comparability of Model Fusion to Measured Data in Confuser Rejection [0.24578723416255746]
  No dataset can account for every slight deviation we might see in live usage.<n>Simulators have been developed utilizing the shooting and bouncing ray method to allow for the generation of synthetic SAR data on 3D models.<n>We aim to use computational power as a substitution for this lack of quality measured data, by ensembling many models trained on synthetic data.
  arXiv  Detail & Related papers  (2025-05-01T19:51:30Z)
• Digitally Prototype Your Eye Tracker: Simulating Hardware Performance using 3D Synthetic Data [38.620399293339034]
  We propose a method for end-to-end evaluation of how hardware changes impact machine learning-based ET performance.<n>We use a dataset of real 3D eyes, reconstructed from light dome data using neural radiance fields (NeRF), to synthesize captured eyes from novel viewpoints.<n>We also present a first-of-its-kind analysis in which we vary ET camera positions, evaluating ET performance ranging from on-axis direct views of the eye to peripheral views on the frame.
  arXiv  Detail & Related papers  (2025-03-20T23:09:15Z)
• Drive-1-to-3: Enriching Diffusion Priors for Novel View Synthesis of Real Vehicles [81.29018359825872]
  This paper consolidates a set of good practices to finetune large pretrained models for a real-world task.<n>Specifically, we develop several strategies to account for discrepancies between the synthetic data and real driving data.<n>Our insights lead to effective finetuning that results in a $68.8%$ reduction in FID for novel view synthesis over prior arts.
  arXiv  Detail & Related papers  (2024-12-19T03:39:13Z)
• Enhancing Generalizability of Representation Learning for Data-Efficient 3D Scene Understanding [50.448520056844885]
  We propose a generative Bayesian network to produce diverse synthetic scenes with real-world patterns. A series of experiments robustly display our method's consistent superiority over existing state-of-the-art pre-training approaches.
  arXiv  Detail & Related papers  (2024-06-17T07:43:53Z)
• Multi-Modal Dataset Acquisition for Photometrically Challenging Object [56.30027922063559]
  This paper addresses the limitations of current datasets for 3D vision tasks in terms of accuracy, size, realism, and suitable imaging modalities for photometrically challenging objects. We propose a novel annotation and acquisition pipeline that enhances existing 3D perception and 6D object pose datasets.
  arXiv  Detail & Related papers  (2023-08-21T10:38:32Z)
• A New Benchmark: On the Utility of Synthetic Data with Blender for Bare Supervised Learning and Downstream Domain Adaptation [42.2398858786125]
  Deep learning in computer vision has achieved great success with the price of large-scale labeled training data. The uncontrollable data collection process produces non-IID training and test data, where undesired duplication may exist. To circumvent them, an alternative is to generate synthetic data via 3D rendering with domain randomization.
  arXiv  Detail & Related papers  (2023-03-16T09:03:52Z)
• Hands-Up: Leveraging Synthetic Data for Hands-On-Wheel Detection [0.38233569758620045]
  This work demonstrates the use of synthetic photo-realistic in-cabin data to train a Driver Monitoring System. We show how performing error analysis and generating the missing edge-cases in our platform boosts performance. This showcases the ability of human-centric synthetic data to generalize well to the real world.
  arXiv  Detail & Related papers  (2022-05-31T23:34:12Z)
• SimAug: Learning Robust Representations from Simulation for Trajectory Prediction [78.91518036949918]
  We propose a novel approach to learn robust representation through augmenting the simulation training data. We show that SimAug achieves promising results on three real-world benchmarks using zero real training data.
  arXiv  Detail & Related papers  (2020-04-04T21:22:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.