Towards in-the-wild Egocentric 3D Hand-Object Pose Estimation

• URL: http://arxiv.org/abs/2606.30598v1
• Date: Mon, 29 Jun 2026 17:38:12 GMT
• Title: Towards in-the-wild Egocentric 3D Hand-Object Pose Estimation
• Abstract summary: Estimating accurate 3D hand-object pose from in-the-wild egocentric RGB remains challenging.<n>First, we introduce EPIC-Contact, an in-the-wild egocentric dataset of 2.3K clips.<n>Second, we propose HOPformer, an end-to-end transformer that jointly predicts bi-manual hand and object pose.
• Score: 59.968931594801234
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Estimating accurate 3D hand-object pose from in-the-wild egocentric RGB remains challenging due to severe occlusions and ambiguous contact. Existing learning-based methods often struggle to generalise to in-the-wild scenes and are limited by the scarcity of supervision. We address these issues with two contributions. First, we introduce EPIC-Contact, an in-the-wild egocentric dataset of 2.3K clips (62.3K frames) with dense, bijective 3D hand-object contact correspondences and posed meshes. Second, we propose HOPformer, an end-to-end transformer that jointly predicts bi-manual hand and object pose in a single forward pass. A cross-attention decoder conditions object features on hand priors, producing robust pose estimation. We test HOPformer on the in-lab 3D dataset, ARCTIC, as well as our newly introduced EPIC-Contact dataset. HOPformer reaches 82.4% success rate on ARCTIC (+6.2 pts over current SOTA). On EPIC-Contact, it nearly doubles the success rate while reducing contact deviation by 75%. EPIC-Contact, HOPformer code and checkpoints are released: https://sid2697.github.io/epic-contact.

Related papers

• You Only Touch Once: 6-DoF Object Pose Estimation from Single Tactile Contact [42.491016490387466]
  YOTO is a tactile-only pose estimation system that recovers the full 6-DoF object pose from a single pair of simultaneous contacts.<n>YOTO represents each tactile contact as a local 3D point cloud and localizes it on the object surface through a coarse-to-fine network.<n>Experiments on four geometrically diverse objects demonstrate accurate tactile contact localization and pose estimation, outperforming vision-based and geometric baselines.
  arXiv  Detail & Related papers  (2026-06-27T13:12:57Z)
• PandaPose: 3D Human Pose Lifting from a Single Image via Propagating 2D Pose Prior to 3D Anchor Space [62.10630827126755]
  PandaPose is a 3D human pose lifting approach via propagating 2D pose prior to 3D anchor space as the unified intermediate representation.<n>Our 3D anchor space comprises: (1) Joint-wise 3D anchors in the canonical coordinate system, providing accurate and robust priors to mitigate 2D pose estimation inaccuracies.
  arXiv  Detail & Related papers  (2026-02-01T08:20:40Z)
• DecoDINO: 3D Human-Scene Contact Prediction with Semantic Classification [0.0]
  We introduce DecoDINO, a three-branch network based on DECO's framework.<n>It uses two DINOv2 ViT-g/14 encoders, class-balanced loss weighting to reduce bias, and patch-level cross-attention for improved local reasoning.<n>On the DAMON benchmark, DecoDINO raises the binary-contact F1 score by 7$%$, (ii) halves the geodesic error, and (iii) augments predictions with object-level semantic labels.
  arXiv  Detail & Related papers  (2025-10-27T10:46:22Z)
• Monocular 3D Hand Pose Estimation with Implicit Camera Alignment [9.199465050084296]
  Estimating the 3D hand articulation from a single color image is an important problem with applications in Augmented Reality (AR), Virtual Reality (VR), Human-Computer Interaction (HCI)<n>We propose an optimization pipeline for estimating the 3D hand articulation from 2D keypoint input, which includes a keypoint alignment step and a fingertip loss.<n>We evaluate our approach on the EgoDexter and Dexter+Object benchmarks to showcase that it performs competitively with the state-of-the-art.
  arXiv  Detail & Related papers  (2025-06-10T18:45:22Z)
• Learning Explicit Contact for Implicit Reconstruction of Hand-held Objects from Monocular Images [59.49985837246644]
  We show how to model contacts in an explicit way to benefit the implicit reconstruction of hand-held objects. In the first part, we propose a new subtask of directly estimating 3D hand-object contacts from a single image. In the second part, we introduce a novel method to diffuse estimated contact states from the hand mesh surface to nearby 3D space.
  arXiv  Detail & Related papers  (2023-05-31T17:59:26Z)
• ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation [68.80339307258835]
  ARCTIC is a dataset of two hands that dexterously manipulate objects. It contains 2.1M video frames paired with accurate 3D hand meshes and detailed, dynamic contact information.
  arXiv  Detail & Related papers  (2022-04-28T17:23:59Z)
• SO-Pose: Exploiting Self-Occlusion for Direct 6D Pose Estimation [98.83762558394345]
  SO-Pose is a framework for regressing all 6 degrees-of-freedom (6DoF) for the object pose in a cluttered environment from a single RGB image. We introduce a novel reasoning about self-occlusion, in order to establish a two-layer representation for 3D objects. Cross-layer consistencies that align correspondences, self-occlusion and 6D pose, we can further improve accuracy and robustness.
  arXiv  Detail & Related papers  (2021-08-18T19:49:29Z)
• HandsFormer: Keypoint Transformer for Monocular 3D Pose Estimation ofHands and Object in Interaction [33.661745138578596]
  We propose a robust and accurate method for estimating the 3D poses of two hands in close interaction from a single color image. Our method starts by extracting a set of potential 2D locations for the joints of both hands as extrema of a heatmap. We use appearance and spatial encodings of these locations as input to a transformer, and leverage the attention mechanisms to sort out the correct configuration of the joints.
  arXiv  Detail & Related papers  (2021-04-29T20:19:20Z)
• Reconstructing Hand-Object Interactions in the Wild [71.16013096764046]
  We propose an optimization-based procedure which does not require direct 3D supervision. We exploit all available related data (2D bounding boxes, 2D hand keypoints, 2D instance masks, 3D object models, 3D in-the-lab MoCap) to provide constraints for the 3D reconstruction. Our method produces compelling reconstructions on the challenging in-the-wild data from the EPIC Kitchens and the 100 Days of Hands datasets.
  arXiv  Detail & Related papers  (2020-12-17T18:59:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.