EgoEV-HandPose: Egocentric 3D Hand Pose Estimation and Gesture Recognition with Stereo Event Cameras

• URL: http://arxiv.org/abs/2605.12297v1
• Date: Tue, 12 May 2026 15:51:04 GMT
• Title: EgoEV-HandPose: Egocentric 3D Hand Pose Estimation and Gesture Recognition with Stereo Event Cameras
• Abstract summary: EgoEV-HandPose is an end-to-end framework for joint 3D bimanual pose estimation and gesture recognition from stereo event streams.<n>EgoEVHands is the first large-scale real-world stereo event-camera dataset for egocentric hand perception.
• Score: 27.55657128726312
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Egocentric 3D hand pose estimation and gesture recognition are essential for immersive augmented/virtual reality, human-computer interaction, and robotics. However, conventional frame-based cameras suffer from motion blur and limited dynamic range, while existing event-based methods are hindered by ego-motion interference, monocular depth ambiguity, and the lack of large-scale real-world stereo datasets. To overcome these limitations, we propose EgoEV-HandPose, an end-to-end framework for joint 3D bimanual pose estimation and gesture recognition from stereo event streams. Central to our approach is KeypointBEV, a flexible stereo fusion module that lifts features into a canonical bird's-eye-view space and employs an iterative reprojection-guided refinement loop to progressively resolve depth uncertainty and enforce kinematic consistency. In addition, we introduce EgoEVHands, the first large-scale real-world stereo event-camera dataset for egocentric hand perception, containing 5,419 annotated sequences with dense 3D/2D keypoints across 38 gesture classes under varying illumination. Extensive experiments demonstrate that EgoEV-HandPose achieves state-of-the-art performance with an MPJPE of 30.54mm and 86.87% Top-1 gesture recognition accuracy, significantly outperforming RGB-based stereo and prior event-camera methods, particularly in low-light and bimanual occlusion scenarios, thereby setting a new benchmark for event-based egocentric perception. The established dataset and source code will be publicly released at https://github.com/ZJUWang01/EgoEV-HandPose.

Related papers

• EventEgoHands++: Event-based Egocentric 3D Hand Mesh Reconstruction with Real Dataset [3.8731899020772165]
  We propose a framework for event-based 3D hand mesh reconstruction from an egocentric viewpoint.<n>The proposed method incorporates a Hand Detector that estimates instance-level bounding boxes and masks for both the left and right hands.<n>We also introduce Adaptive Attention, which dynamically gates the attention based on these detection results.
  arXiv  Detail & Related papers  (2026-09-15T13:46:02Z)
• HandsOnWorld: Unconstrained Egocentric Video Generation with Camera-Disentangled Hand Control [51.913695868638634]
  We present HandsOnWorld, a framework for hand-controlled egocentric video generation that forgoes multi-view and marker-based motion capture, learning instead from unconstrained monocular video.<n>We build EgoVid-Pro, a dataset of clean, protagonist-only hand trajectories spanning 103K clips and roughly 12M frames across diverse everyday scenes.<n>We also propose the Plcker Hand Map, a 3D-aware control signal that extends Plcker-ray representations from camera rays to the hand surface, disentangling camera and hand motion at the representation level.
  arXiv  Detail & Related papers  (2026-07-02T12:14:02Z)
• EgoForce: Forearm-Guided Camera-Space 3D Hand Pose from a Monocular Egocentric Camera [40.28396219356337]
  EgoForce is a monocular 3D hand reconstruction framework that recovers robust, absolute 3D hand pose.<n>It operates across fisheye, perspective, and distorted wide-FOV camera models using a single unified network.<n>EgoForce achieves state-of-the-art 3D accuracy, reducing camera-space MPJPE by up to 28%.
  arXiv  Detail & Related papers  (2026-05-12T17:59:56Z)
• SHOW3D: Capturing Scenes of 3D Hands and Objects in the Wild [17.556955295385908]
  Existing hand-object interaction datasets are predominantly captured in controlled studio settings.<n>We introduce a novel marker-less multi-camera system that allows for nearly unconstrained mobility in genuinely in-the-wild conditions.<n>We present SHOW3D, the first large-scale dataset with 3D annotations that show hands interacting with objects in diverse real-world environments.
  arXiv  Detail & Related papers  (2026-03-30T17:58:27Z)
• Controllable Egocentric Video Generation via Occlusion-Aware Sparse 3D Hand Joints [87.13154261503168]
  Motion-controllable video generation is crucial for egocentric applications in virtual reality and embodied AI.<n>Existing methods often struggle to achieve 3D-consistent fine-grained hand articulation.<n>We propose a novel framework that generates egocentric videos from a single reference frame.
  arXiv  Detail & Related papers  (2026-03-12T10:02:23Z)
• Ego-Exo 3D Hand Tracking in the Wild with a Mobile Multi-Camera Rig [14.496137517475743]
  We introduce a novel marker-less multi-camera system designed to capture precise 3D hands and objects.<n>We combine a lightweight, back-mounted capture rig with eight exocentric cameras, and a user-worn Meta Quest 3 headset, which contributes two egocentric views.<n>We collect an annotated dataset featuring synchronized multi-view images and precise 3D hand poses.
  arXiv  Detail & Related papers  (2025-10-02T22:26:03Z)
• FRAME: Floor-aligned Representation for Avatar Motion from Egocentric Video [52.33896173943054]
  Egocentric motion capture with a head-mounted body-facing stereo camera is crucial for VR and AR applications.<n>Existing methods rely on synthetic pretraining and struggle to generate smooth and accurate predictions in real-world settings.<n>We propose FRAME, a simple yet effective architecture that combines device pose and camera feeds for state-of-the-art body pose prediction.
  arXiv  Detail & Related papers  (2025-03-29T14:26:06Z)
• EgoEvGesture: Gesture Recognition Based on Egocentric Event Camera [17.61884467264023]
  We propose a novel network architecture specifically designed for event data processing.<n>We establish the first large-scale dataset for egocentric gesture recognition using event cameras.<n>Our method achieves 62.7% accuracy tested on unseen subjects with only 7M parameters, 3.1% higher than state-of-the-art approaches.
  arXiv  Detail & Related papers  (2025-03-16T09:08:02Z)
• Dyn-HaMR: Recovering 4D Interacting Hand Motion from a Dynamic Camera [49.82535393220003]
  Dyn-HaMR is the first approach to reconstruct 4D global hand motion from monocular videos recorded by dynamic cameras in the wild.<n>We show that our approach significantly outperforms state-of-the-art methods in terms of 4D global mesh recovery.<n>This establishes a new benchmark for hand motion reconstruction from monocular video with moving cameras.
  arXiv  Detail & Related papers  (2024-12-17T12:43:10Z)
• HMP: Hand Motion Priors for Pose and Shape Estimation from Video [52.39020275278984]
  We develop a generative motion prior specific for hands, trained on the AMASS dataset which features diverse and high-quality hand motions. Our integration of a robust motion prior significantly enhances performance, especially in occluded scenarios. We demonstrate our method's efficacy via qualitative and quantitative evaluations on the HO3D and DexYCB datasets.
  arXiv  Detail & Related papers  (2023-12-27T22:35:33Z)
• UnrealEgo: A New Dataset for Robust Egocentric 3D Human Motion Capture [70.59984501516084]
  UnrealEgo is a new large-scale naturalistic dataset for egocentric 3D human pose estimation. It is based on an advanced concept of eyeglasses equipped with two fisheye cameras that can be used in unconstrained environments. We propose a new benchmark method with a simple but effective idea of devising a 2D keypoint estimation module for stereo inputs to improve 3D human pose estimation.
  arXiv  Detail & Related papers  (2022-08-02T17:59:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.