Self-Supervised Multi-View 3D Gaze Target Estimation via Probabilistic Ray Marching

• URL: http://arxiv.org/abs/2609.07415v1
• Date: Mon, 07 Sep 2026 12:26:14 GMT
• Title: Self-Supervised Multi-View 3D Gaze Target Estimation via Probabilistic Ray Marching
• Abstract summary: We present a self-supervised approach, Self-MVGTE, for estimating 3D gaze targets from multiple camera views.<n>Unlike existing methods that independently estimate 2D gaze targets per camera view, Self-MVGTE predicts gaze targets directly in 3D space for the first time.
• Score: 13.206046367696961
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We present a self-supervised approach, Self-MVGTE, for estimating 3D gaze targets from multiple camera views. Unlike existing methods that independently estimate 2D gaze targets per camera view, Self-MVGTE predicts gaze targets directly in 3D space for the first time. Moreover, it does not require any ground-truth annotations from the target scene and uses only the multi-view input images from a calibrated camera setup, pseudo 2D gaze target labels from a monocular gaze target estimation model, and 3D gaze vectors from a monocular 3D gaze estimation model. A key challenge is that these pseudo labels are inherently noisy and multi-view inconsistent. To address this, we propose a probabilistic ray marching framework, which models the uncertainty of these pseudo labels and exploits 3D gaze vectors as geometric priors. Specifically, these gaze vectors are first integrated into the monocular gaze target estimation model to improve its generalization to unseen scenes, producing higher-quality pseudo labels. Then, for 3D gaze target estimation, we construct a 3D gaze cone by casting a bundle of rays from the eye position around the gaze vector to strictly constrain the solution space. Within this cone, we propose a depth-guided feature sampling strategy using off-the-shelf DINOv2 and Depth-Anything-3 models, and estimate a spatial likelihood distribution of the gaze target. Finally, we convert the pseudo gaze target labels into a target distribution and softly optimize the network. Extensive experiments on the MVGT dataset show that Self-MVGTE achieves state-of-the-art performance, surpassing existing fully-supervised baselines.

Related papers

• OmniGaze: Reward-inspired Generalizable Gaze Estimation In The Wild [104.57404324262556]
  Current 3D gaze estimation methods struggle to generalize across diverse data domains.<n>We present OmniGaze, a semi-supervised framework for 3D gaze estimation.<n>We show that OmniGaze achieves state-of-the-art performance on five datasets.
  arXiv  Detail & Related papers  (2025-10-15T15:19:52Z)
• Enhancing 3D Gaze Estimation in the Wild using Weak Supervision with Gaze Following Labels [10.827081942898506]
  We introduce a novel Self-Training Weakly-Supervised Gaze Estimation framework (ST-WSGE)<n>We propose the Gaze Transformer (GaT), a modality-agnostic architecture capable of simultaneously learning static and dynamic gaze information from both image and video datasets.<n>By combining 3D video datasets with 2D gaze target labels from gaze following tasks, our approach achieves the following key contributions.
  arXiv  Detail & Related papers  (2025-02-27T16:35:25Z)
• 3D Prior is All You Need: Cross-Task Few-shot 2D Gaze Estimation [27.51272922798475]
  We introduce a novel cross-task 2D gaze estimation approach, aiming to adapt a pre-trained 3D gaze estimation network for 2D gaze prediction on unseen devices.<n>This task is highly challenging due to the domain gap between 3D and 2D gaze, unknown screen poses, and limited training data.<n>We evaluate our method on MPIIGaze, EVE, and GazeCapture datasets, collected respectively on laptops, desktop computers, and mobile devices.
  arXiv  Detail & Related papers  (2025-02-06T13:37:09Z)
• Towards Pixel-Level Prediction for Gaze Following: Benchmark and Approach [27.84672974344777]
  We propose a novel gaze target prediction solution named GazeSeg.<n>It can fully utilize the spatial visual field of the person as guiding information and lead to a progressively coarse-to-fine gaze target segmentation and recognition process.<n>Our approach achieves the Dice of 0.325 in gaze target segmentation and 71.7% top-5 recognition.
  arXiv  Detail & Related papers  (2024-11-30T01:27:48Z)
• VFMM3D: Releasing the Potential of Image by Vision Foundation Model for Monocular 3D Object Detection [80.62052650370416]
  monocular 3D object detection holds significant importance across various applications, including autonomous driving and robotics. In this paper, we present VFMM3D, an innovative framework that leverages the capabilities of Vision Foundation Models (VFMs) to accurately transform single-view images into LiDAR point cloud representations.
  arXiv  Detail & Related papers  (2024-04-15T03:12:12Z)
• Towards Unified 3D Object Detection via Algorithm and Data Unification [70.27631528933482]
  We build the first unified multi-modal 3D object detection benchmark MM- Omni3D and extend the aforementioned monocular detector to its multi-modal version. We name the designed monocular and multi-modal detectors as UniMODE and MM-UniMODE, respectively.
  arXiv  Detail & Related papers  (2024-02-28T18:59:31Z)
• Model-aware 3D Eye Gaze from Weak and Few-shot Supervisions [60.360919642038]
  We propose to predict 3D eye gaze from weak supervision of eye semantic segmentation masks and direct supervision of a few 3D gaze vectors. Our experiments in diverse settings illustrate the significant benefits of the proposed method, achieving about 5 degrees lower angular gaze error over the baseline.
  arXiv  Detail & Related papers  (2023-11-20T20:22:55Z)
• 3DiffTection: 3D Object Detection with Geometry-Aware Diffusion Features [70.50665869806188]
  3DiffTection is a state-of-the-art method for 3D object detection from single images. We fine-tune a diffusion model to perform novel view synthesis conditioned on a single image. We further train the model on target data with detection supervision.
  arXiv  Detail & Related papers  (2023-11-07T23:46:41Z)
• NeRF-Gaze: A Head-Eye Redirection Parametric Model for Gaze Estimation [37.977032771941715]
  We propose a novel Head-Eye redirection parametric model based on Neural Radiance Field. Our model can decouple the face and eyes for separate neural rendering. It can achieve the purpose of separately controlling the attributes of the face, identity, illumination, and eye gaze direction.
  arXiv  Detail & Related papers  (2022-12-30T13:52:28Z)
• 3DGazeNet: Generalizing Gaze Estimation with Weak-Supervision from Synthetic Views [67.00931529296788]
  We propose to train general gaze estimation models which can be directly employed in novel environments without adaptation. We create a large-scale dataset of diverse faces with gaze pseudo-annotations, which we extract based on the 3D geometry of the scene. We test our method in the task of gaze generalization, in which we demonstrate improvement of up to 30% compared to state-of-the-art when no ground truth data are available.
  arXiv  Detail & Related papers  (2022-12-06T14:15:17Z)
• MonoGRNet: A General Framework for Monocular 3D Object Detection [23.59839921644492]
  We propose MonoGRNet for the amodal 3D object detection from a monocular image via geometric reasoning. MonoGRNet decomposes the monocular 3D object detection task into four sub-tasks including 2D object detection, instance-level depth estimation, projected 3D center estimation and local corner regression. Experiments are conducted on KITTI, Cityscapes and MS COCO datasets.
  arXiv  Detail & Related papers  (2021-04-18T10:07:52Z)
• M3DSSD: Monocular 3D Single Stage Object Detector [82.25793227026443]
  We propose a Monocular 3D Single Stage object Detector (M3DSSD) with feature alignment and asymmetric non-local attention. The proposed M3DSSD achieves significantly better performance than the monocular 3D object detection methods on the KITTI dataset.
  arXiv  Detail & Related papers  (2021-03-24T13:09:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.