HandDreamer: Zero-Shot Text to 3D Hand Model Generation using Corrective Hand Shape Guidance

• URL: http://arxiv.org/abs/2604.04425v1
• Date: Mon, 06 Apr 2026 05:14:20 GMT
• Title: HandDreamer: Zero-Shot Text to 3D Hand Model Generation using Corrective Hand Shape Guidance
• Abstract summary: We introduce HandDreamer, the first method for zero-shot 3D hand model generation from text prompts.<n>Our findings suggest that view-inconsistencies in SDS is primarily caused due to the ambiguity in the probability landscape.<n>We propose a novel corrective hand shape guidance loss to ensure that all the views of the 3D hand model converges to view-consistent modes.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The emergence of virtual reality has necessitated the generation of detailed and customizable 3D hand models for interaction in the virtual world. However, the current methods for 3D hand model generation are both expensive and cumbersome, offering very little customizability to the users. While recent advancements in zero-shot text-to-3D synthesis have enabled the generation of diverse and customizable 3D models using Score Distillation Sampling (SDS), they do not generalize very well to 3D hand model generation, resulting in unnatural hand structures, view-inconsistencies and loss of details. To address these limitations, we introduce HandDreamer, the first method for zero-shot 3D hand model generation from text prompts. Our findings suggest that view-inconsistencies in SDS is primarily caused due to the ambiguity in the probability landscape described by the text prompt, resulting in similar views converging to different modes of the distribution. This is particularly aggravated for hands due to the large variations in articulations and poses. To alleviate this, we propose to use MANO hand model based initialization and a hand skeleton guided diffusion process to provide a strong prior for the hand structure and to ensure view and pose consistency. Further, we propose a novel corrective hand shape guidance loss to ensure that all the views of the 3D hand model converges to view-consistent modes, without leading to geometric distortions. Extensive evaluations demonstrate the superiority of our method over the state-of-the-art methods, paving a new way forward in 3D hand model generation.

Related papers

• 3D Hand Mesh-Guided AI-Generated Malformed Hand Refinement with Hand Pose Transformation via Diffusion Model [40.20849519857311]
  We propose a 3D mesh-guided refinement framework using a diffusion pipeline.<n>For training, we collect and reannotate a dataset consisting of RGB images and 3D hand mesh.<n>We then design a diffusion inpainting model to generate refined outputs guided by 3D hand meshes.
  arXiv  Detail & Related papers  (2025-06-15T01:30:22Z)
• JGHand: Joint-Driven Animatable Hand Avater via 3D Gaussian Splatting [3.1143479095236892]
  Jointly 3D Gaussian Hand (JGHand) is a novel joint-driven 3D Gaussian Splatting (3DGS)-based hand representation.<n>We show that JGHand achieves real-time rendering speeds with enhanced quality, surpassing state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-01-31T12:33:24Z)
• HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactions [68.28684509445529]
  We present HandBooster, a new approach to uplift the data diversity and boost the 3D hand-mesh reconstruction performance. First, we construct versatile content-aware conditions to guide a diffusion model to produce realistic images with diverse hand appearances, poses, views, and backgrounds. Then, we design a novel condition creator based on our similarity-aware distribution sampling strategies to deliberately find novel and realistic interaction poses that are distinctive from the training set.
  arXiv  Detail & Related papers  (2024-03-27T13:56:08Z)
• HanDiffuser: Text-to-Image Generation With Realistic Hand Appearances [34.50137847908887]
  Text-to-image generative models can generate high-quality humans, but realism is lost when generating hands. Common artifacts include irregular hand poses, shapes, incorrect numbers of fingers, and physically implausible finger orientations. We propose a novel diffusion-based architecture called HanDiffuser that achieves realism by injecting hand embeddings in the generative process.
  arXiv  Detail & Related papers  (2024-03-04T03:00:22Z)
• HumanRef: Single Image to 3D Human Generation via Reference-Guided Diffusion [53.1558345421646]
  We propose HumanRef, a 3D human generation framework from a single-view input. To ensure the generated 3D model is photorealistic and consistent with the input image, HumanRef introduces a novel method called reference-guided score distillation sampling. Experimental results demonstrate that HumanRef outperforms state-of-the-art methods in generating 3D clothed humans.
  arXiv  Detail & Related papers  (2023-11-28T17:06:28Z)
• Denoising Diffusion for 3D Hand Pose Estimation from Images [38.20064386142944]
  This paper addresses the problem of 3D hand pose estimation from monocular images or sequences. We present a novel end-to-end framework for 3D hand regression that employs diffusion models that have shown excellent ability to capture the distribution of data for generative purposes. The proposed model provides state-of-the-art performance when lifting a 2D single-hand image to 3D.
  arXiv  Detail & Related papers  (2023-08-18T12:57:22Z)
• Overcoming the Trade-off Between Accuracy and Plausibility in 3D Hand Shape Reconstruction [62.96478903239799]
  Direct mesh fitting for 3D hand shape reconstruction is highly accurate. However, the reconstructed meshes are prone to artifacts and do not appear as plausible hand shapes. We introduce a novel weakly-supervised hand shape estimation framework that integrates non-parametric mesh fitting with MANO model in an end-to-end fashion.
  arXiv  Detail & Related papers  (2023-05-01T03:38:01Z)
• Embodied Hands: Modeling and Capturing Hands and Bodies Together [61.32931890166915]
  Humans move their hands and bodies together to communicate and solve tasks. Most methods treat the 3D modeling and tracking of bodies and hands separately. We formulate a model of hands and bodies interacting together and fit it to full-body 4D sequences.
  arXiv  Detail & Related papers  (2022-01-07T18:59:32Z)
• A Skeleton-Driven Neural Occupancy Representation for Articulated Hands [49.956892429789775]
  Hand ArticuLated Occupancy (HALO) is a novel representation of articulated hands that bridges the advantages of 3D keypoints and neural implicit surfaces. We demonstrate the applicability of HALO to the task of conditional generation of hands that grasp 3D objects.
  arXiv  Detail & Related papers  (2021-09-23T14:35:19Z)
• HandFoldingNet: A 3D Hand Pose Estimation Network Using Multiscale-Feature Guided Folding of a 2D Hand Skeleton [4.1954750695245835]
  This paper proposes HandFoldingNet, an accurate and efficient hand pose estimator. The proposed model utilizes a folding-based decoder that folds a given 2D hand skeleton into the corresponding joint coordinates. Experimental results show that the proposed model outperforms the existing methods on three hand pose benchmark datasets.
  arXiv  Detail & Related papers  (2021-08-12T05:52:44Z)
• MM-Hand: 3D-Aware Multi-Modal Guided Hand Generative Network for 3D Hand Pose Synthesis [81.40640219844197]
  Estimating the 3D hand pose from a monocular RGB image is important but challenging. A solution is training on large-scale RGB hand images with accurate 3D hand keypoint annotations. We have developed a learning-based approach to synthesize realistic, diverse, and 3D pose-preserving hand images.
  arXiv  Detail & Related papers  (2020-10-02T18:27:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.