Accurate Gaze Estimation using an Active-gaze Morphable Model

• URL: http://arxiv.org/abs/2301.13186v1
• Date: Mon, 30 Jan 2023 18:51:14 GMT
• Title: Accurate Gaze Estimation using an Active-gaze Morphable Model
• Authors: Hao Sun and Nick Pears
• Abstract summary: Rather than regressing gaze direction directly from images, we show that adding a 3D shape model can improve gaze estimation accuracy. We equip this with a geometric vergence model of gaze to give an active-gaze 3DMM' Our method can learn with only the ground truth gaze target point and the camera parameters, without access to the ground truth gaze origin points.
• Score: 9.192482716410511
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Rather than regressing gaze direction directly from images, we show that adding a 3D shape model can: i) improve gaze estimation accuracy, ii) perform well with lower resolution inputs and iii) provide a richer understanding of the eye-region and its constituent gaze system. Specifically, we use an `eyes and nose' 3D morphable model (3DMM) to capture the eye-region 3D facial geometry and appearance and we equip this with a geometric vergence model of gaze to give an `active-gaze 3DMM'. We show that our approach achieves state-of-the-art results on the Eyediap dataset and we present an ablation study. Our method can learn with only the ground truth gaze target point and the camera parameters, without access to the ground truth gaze origin points, thus widening the applicability of our approach compared to other methods.

Related papers

• Direct Learning of Mesh and Appearance via 3D Gaussian Splatting [3.4899193297791054]
  We propose a learnable scene model that incorporates 3DGS with an explicit geometry representation, namely a mesh. Our model learns the mesh and appearance in an end-to-end manner, where we bind 3D Gaussians to the mesh faces and perform differentiable rendering of 3DGS to obtain photometric supervision.
  arXiv  Detail & Related papers  (2024-05-11T07:56:19Z)
• VFMM3D: Releasing the Potential of Image by Vision Foundation Model for Monocular 3D Object Detection [80.62052650370416]
  monocular 3D object detection holds significant importance across various applications, including autonomous driving and robotics. In this paper, we present VFMM3D, an innovative framework that leverages the capabilities of Vision Foundation Models (VFMs) to accurately transform single-view images into LiDAR point cloud representations.
  arXiv  Detail & Related papers  (2024-04-15T03:12:12Z)
• Model-aware 3D Eye Gaze from Weak and Few-shot Supervisions [60.360919642038]
  We propose to predict 3D eye gaze from weak supervision of eye semantic segmentation masks and direct supervision of a few 3D gaze vectors. Our experiments in diverse settings illustrate the significant benefits of the proposed method, achieving about 5 degrees lower angular gaze error over the baseline.
  arXiv  Detail & Related papers  (2023-11-20T20:22:55Z)
• DSFNet: Dual Space Fusion Network for Occlusion-Robust 3D Dense Face Alignment [34.223372986832544]
  The state-of-the-art 3DMM-based method, directly regresses the model's coefficients. We propose a fusion network that combines the advantages of both the image and model space predictions.
  arXiv  Detail & Related papers  (2023-05-19T08:43:37Z)
• EFE: End-to-end Frame-to-Gaze Estimation [42.61379693370926]
  We propose a frame-to-gaze network that directly predicts both 3D gaze origin and 3D gaze direction from the raw frame out of the camera without any face or eye cropping. Our method demonstrates that direct gaze regression from the raw downscaled frame, from FHD/HD to VGA/HVGA resolution, is possible despite the challenges of having very few pixels in the eye region.
  arXiv  Detail & Related papers  (2023-05-09T15:25:45Z)
• NeRF-Gaze: A Head-Eye Redirection Parametric Model for Gaze Estimation [37.977032771941715]
  We propose a novel Head-Eye redirection parametric model based on Neural Radiance Field. Our model can decouple the face and eyes for separate neural rendering. It can achieve the purpose of separately controlling the attributes of the face, identity, illumination, and eye gaze direction.
  arXiv  Detail & Related papers  (2022-12-30T13:52:28Z)
• GazeNeRF: 3D-Aware Gaze Redirection with Neural Radiance Fields [100.53114092627577]
  Existing gaze redirection methods operate on 2D images and struggle to generate 3D consistent results. We build on the intuition that the face region and eyeballs are separate 3D structures that move in a coordinated yet independent fashion.
  arXiv  Detail & Related papers  (2022-12-08T13:19:11Z)
• 3DGazeNet: Generalizing Gaze Estimation with Weak-Supervision from Synthetic Views [67.00931529296788]
  We propose to train general gaze estimation models which can be directly employed in novel environments without adaptation. We create a large-scale dataset of diverse faces with gaze pseudo-annotations, which we extract based on the 3D geometry of the scene. We test our method in the task of gaze generalization, in which we demonstrate improvement of up to 30% compared to state-of-the-art when no ground truth data are available.
  arXiv  Detail & Related papers  (2022-12-06T14:15:17Z)
• A Model for Multi-View Residual Covariances based on Perspective Deformation [88.21738020902411]
  We derive a model for the covariance of the visual residuals in multi-view SfM, odometry and SLAM setups. We validate our model with synthetic and real data and integrate it into photometric and feature-based Bundle Adjustment.
  arXiv  Detail & Related papers  (2022-02-01T21:21:56Z)
• MonoDistill: Learning Spatial Features for Monocular 3D Object Detection [80.74622486604886]
  We propose a simple and effective scheme to introduce the spatial information from LiDAR signals to the monocular 3D detectors. We use the resulting data to train a 3D detector with the same architecture as the baseline model. Experimental results show that the proposed method can significantly boost the performance of the baseline model.
  arXiv  Detail & Related papers  (2022-01-26T09:21:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.