AnyHand: A Large-Scale Synthetic Dataset for RGB(-D) Hand Pose Estimation

• URL: http://arxiv.org/abs/2603.25726v2
• Date: Mon, 30 Mar 2026 02:48:06 GMT
• Title: AnyHand: A Large-Scale Synthetic Dataset for RGB(-D) Hand Pose Estimation
• Abstract summary: We present AnyHand, a large-scale synthetic dataset designed to advance the state of the art in 3D hand pose estimation.<n>We show that extending the original training sets of existing baselines with AnyHand yields significant gains on multiple benchmarks.<n>We also contribute a lightweight depth fusion module that can be easily integrated into existing RGB-based models.
• Score: 50.62485061150716
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We present AnyHand, a large-scale synthetic dataset designed to advance the state of the art in 3D hand pose estimation from both RGB-only and RGB-D inputs. While recent works with foundation approaches have shown that an increase in the quantity and diversity of training data can markedly improve performance and robustness in hand pose estimation, existing real-world-collected datasets on this task are limited in coverage, and prior synthetic datasets rarely provide occlusions, arm details, and aligned depth together at scale. To address this bottleneck, our AnyHand contains 2.5M single-hand and 4.1M hand-object interaction RGB-D images, with rich geometric annotations. In the RGB-only setting, we show that extending the original training sets of existing baselines with AnyHand yields significant gains on multiple benchmarks (FreiHAND and HO-3D), even when keeping the architecture and training scheme fixed. More impressively, the model trained with AnyHand shows stronger generalization to the out-of-domain HO-Cap dataset, without any fine-tuning. We also contribute a lightweight depth fusion module that can be easily integrated into existing RGB-based models. Trained with AnyHand, the resulting RGB-D model achieves superior performance on the HO-3D benchmark, showing the benefits of depth integration and the effectiveness of our synthetic data.

Related papers

• Demo-Pose: Depth-Monocular Modality Fusion For Object Pose Estimation [5.466547563815996]
  We address the challenge of category-level 9-DoF pose estimation from RGB-D input, without relying on CAD models during inference.<n>We propose DeMo-Pose, a hybrid architecture that fuses semantic features with depth-based graph convolutional representations.<n>Our approach achieves real-time inference and significantly improves over state-of-the-art methods across object categories.
  arXiv  Detail & Related papers  (2026-03-29T05:58:04Z)
• UniPose: Unified Cross-modality Pose Prior Propagation towards RGB-D data for Weakly Supervised 3D Human Pose Estimation [14.52285662885727]
  We present UniPose, a unified cross-modality pose prior propagation method for weakly supervised 3D human pose estimation.<n>UniPose transfers 2D annotations from large-scale RGB datasets to the 3D domain via self-supervised learning.<n> Experiments on CMU Panoptic and ITOP datasets show that UniPose achieves comparable performance to fully supervised methods.
  arXiv  Detail & Related papers  (2025-09-27T15:49:30Z)
• HOGSA: Bimanual Hand-Object Interaction Understanding with 3D Gaussian Splatting Based Data Augmentation [29.766317710266765]
  We propose a new 3D Gaussian Splatting based data augmentation framework for bimanual hand-object interaction.<n>We use mesh-based 3DGS to model objects and hands, and to deal with the rendering blur problem due to multi-resolution input images used.<n>We extend the single hand grasping pose optimization module for the bimanual hand object to generate various poses of bimanual hand-object interaction.
  arXiv  Detail & Related papers  (2025-01-06T08:48:17Z)
• Towards Human-Level 3D Relative Pose Estimation: Generalizable, Training-Free, with Single Reference [65.42565481489132]
  Humans can easily deduce the relative pose of a previously unseen object, without labeling or training, given only a single query-reference image pair.<n>We propose a novel 3D generalizable relative pose estimation method by elaborating 3D/2.5D shape perception with a 2.5D shape from RGB-D reference.<n>Our differentiable takes the 2.5D rotatable mesh textured by the RGB and the semantic maps (obtained by DINOv2 from the RGB input), then renders new RGB and semantic maps under a novel rotated view.
  arXiv  Detail & Related papers  (2024-06-26T16:01:10Z)
• HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactions [68.28684509445529]
  We present HandBooster, a new approach to uplift the data diversity and boost the 3D hand-mesh reconstruction performance. First, we construct versatile content-aware conditions to guide a diffusion model to produce realistic images with diverse hand appearances, poses, views, and backgrounds. Then, we design a novel condition creator based on our similarity-aware distribution sampling strategies to deliberately find novel and realistic interaction poses that are distinctive from the training set.
  arXiv  Detail & Related papers  (2024-03-27T13:56:08Z)
• Reconstructing Hands in 3D with Transformers [64.15390309553892]
  We present an approach that can reconstruct hands in 3D from monocular input. Our approach for Hand Mesh Recovery, HaMeR, follows a fully transformer-based architecture and can analyze hands with significantly increased accuracy and robustness compared to previous work.
  arXiv  Detail & Related papers  (2023-12-08T18:59:07Z)
• RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video [76.86512780916827]
  We present the first real-time method for motion capture of skeletal pose and 3D surface geometry of hands from a single RGB camera. In order to address the inherent depth ambiguities in RGB data, we propose a novel multi-task CNN. We experimentally verify the individual components of our RGB two-hand tracking and 3D reconstruction pipeline.
  arXiv  Detail & Related papers  (2021-06-22T12:53:56Z)
• Two-hand Global 3D Pose Estimation Using Monocular RGB [0.0]
  We tackle the challenging task of estimating global 3D joint locations for both hands via only monocular RGB input images. We propose a novel multi-stage convolutional neural network based pipeline that accurately segments and locates the hands. We present the first work that achieves accurate global 3D hand tracking on both hands using RGB-only inputs.
  arXiv  Detail & Related papers  (2020-06-01T23:53:52Z)
• Measuring Generalisation to Unseen Viewpoints, Articulations, Shapes and Objects for 3D Hand Pose Estimation under Hand-Object Interaction [137.28465645405655]
  HANDS'19 is a challenge to evaluate the abilities of current 3D hand pose estimators (HPEs) to interpolate and extrapolate the poses of a training set. We show that the accuracy of state-of-the-art methods can drop, and that they fail mostly on poses absent from the training set.
  arXiv  Detail & Related papers  (2020-03-30T19:28:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.