The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction

• URL: http://arxiv.org/abs/2606.30308v1
• Date: Mon, 29 Jun 2026 13:53:00 GMT
• Title: The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction
• Abstract summary: ViDiHand is a pipeline for 4D hand motion reconstruction from egocentric video.<n>It is based on a pretrained video diffusion model that specializes its features for hands while preserving its world priors.<n>On ARCTIC, HOT3D, and HOI4D, ViDiHand substantially outperforms prior methods.
• Score: 36.627109956900256
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: 4D hand motion reconstruction from egocentric video is bottlenecked by clear limitations of existing methods: image-based pipelines depend on a detector that fails under heavy occlusion, while video-based methods rely on temporal modules learned only from scarce hand-pose annotations, a narrow signal insufficient to model motion dynamics, occlusion reasoning, and hand-object interaction. These capabilities, however, are exactly what video generative models must implicitly acquire when trained to synthesize coherent video at internet scale. Motivated by this, we present ViDiHand, which leverages the representations of a pretrained video diffusion model to reconstruct 4D two-hand pose. We adapt it via a hand-overlay rendering objective that specializes its features for hands while preserving its world priors. A decoder then recovers metric-scale pose from the adapted features. The whole pipeline operates directly on full frames--no detector, no infiller, and no test-time optimization. On ARCTIC, HOT3D, and HOI4D, ViDiHand substantially outperforms prior methods, establishing video diffusion models as a powerful new foundation for hand motion reconstruction and a promising route to scalable in-the-wild data collection for embodied AI. Project page: https://vidihand.github.io.

Related papers

• DreamHand: Repurposing Video Diffusion Models for Occlusion-Robust Egocentric 3D Hand Motion Recovery [51.92897182932217]
  Egocentric video offers scalable manipulation data for embodied AI.<n>However, recovering metric 3D hand trajectories is challenging due to severe object occlusion and frequent out-of-sight gaps.<n>We introduce DreamHand, an offline clip-level framework that extracts features via a Deterministic Clean-Latent and decodes them with a Bidirectional Spatiotemporal Decoder.
  arXiv  Detail & Related papers  (2026-08-20T17:46:24Z)
• ViPS: Video-informed Pose Spaces for Auto-Rigged Meshes [55.32681167870698]
  Video-informed Pose Spaces (ViPS) is a feed-forward framework that discovers the latent distribution of valid articulations for auto-rigged meshes.<n>ViPS transfers generative video priors into a universal distribution over a given rig parameterization.<n>Our evaluations show that ViPS matches the performance of state-of-the-art methods trained on synthetic artist-created 4D data in both plausibility and diversity.
  arXiv  Detail & Related papers  (2026-04-19T21:21:11Z)
• ForeHOI: Feed-forward 3D Object Reconstruction from Daily Hand-Object Interaction Videos [22.436134664301473]
  We introduce ForeHOI, a novel feed-forward model that directly reconstructs 3D object geometry from monocular hand-object interaction videos.<n>ForeHOI achieves state-of-the-art performance in object reconstruction, significantly outperforming previous methods with around a 100x speedup.
  arXiv  Detail & Related papers  (2026-02-05T22:05:57Z)
• Pixel-to-4D: Camera-Controlled Image-to-Video Generation with Dynamic 3D Gaussians [7.051077403685518]
  Humans excel at forecasting the future dynamics of a scene given just a single image.<n>Video generation models that can mimic this ability are an essential component for intelligent systems.<n>Recent approaches have improved temporal coherence and 3D consistency in single-image-conditioned video generation.
  arXiv  Detail & Related papers  (2026-01-02T13:04:47Z)
• Restage4D: Reanimating Deformable 3D Reconstruction from a Single Video [56.781766315691854]
  We introduce textbfRestage4D, a geometry-preserving pipeline for video-conditioned 4D restaging.<n>We validate Restage4D on DAVIS and PointOdyssey, demonstrating improved geometry consistency, motion quality, and 3D tracking performance.
  arXiv  Detail & Related papers  (2025-08-08T21:31:51Z)
• Diffusion-based 3D Hand Motion Recovery with Intuitive Physics [29.784542628690794]
  We present a novel 3D hand motion recovery framework that enhances image-based reconstructions.<n>Our model captures the distribution of refined motion estimates conditioned on initial ones, generating improved sequences.<n>We identify valuable intuitive physics knowledge during hand-object interactions, including key motion states and their associated motion constraints.
  arXiv  Detail & Related papers  (2025-08-03T16:44:24Z)
• Dyn-HaMR: Recovering 4D Interacting Hand Motion from a Dynamic Camera [49.82535393220003]
  Dyn-HaMR is the first approach to reconstruct 4D global hand motion from monocular videos recorded by dynamic cameras in the wild.<n>We show that our approach significantly outperforms state-of-the-art methods in terms of 4D global mesh recovery.<n>This establishes a new benchmark for hand motion reconstruction from monocular video with moving cameras.
  arXiv  Detail & Related papers  (2024-12-17T12:43:10Z)
• WildVidFit: Video Virtual Try-On in the Wild via Image-Based Controlled Diffusion Models [132.77237314239025]
  Video virtual try-on aims to generate realistic sequences that maintain garment identity and adapt to a person's pose and body shape in source videos. Traditional image-based methods, relying on warping and blending, struggle with complex human movements and occlusions. We reconceptualize video try-on as a process of generating videos conditioned on garment descriptions and human motion. Our solution, WildVidFit, employs image-based controlled diffusion models for a streamlined, one-stage approach.
  arXiv  Detail & Related papers  (2024-07-15T11:21:03Z)
• HMP: Hand Motion Priors for Pose and Shape Estimation from Video [52.39020275278984]
  We develop a generative motion prior specific for hands, trained on the AMASS dataset which features diverse and high-quality hand motions. Our integration of a robust motion prior significantly enhances performance, especially in occluded scenarios. We demonstrate our method's efficacy via qualitative and quantitative evaluations on the HO3D and DexYCB datasets.
  arXiv  Detail & Related papers  (2023-12-27T22:35:33Z)
• Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clips [38.02945794078731]
  We tackle the task of reconstructing hand-object interactions from short video clips. Our approach casts 3D inference as a per-video optimization and recovers a neural 3D representation of the object shape. We empirically evaluate our approach on egocentric videos, and observe significant improvements over prior single-view and multi-view methods.
  arXiv  Detail & Related papers  (2023-09-11T17:58:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.