DreamHand: Repurposing Video Diffusion Models for Occlusion-Robust Egocentric 3D Hand Motion Recovery

• URL: http://arxiv.org/abs/2608.20308v1
• Date: Thu, 20 Aug 2026 17:46:24 GMT
• Title: DreamHand: Repurposing Video Diffusion Models for Occlusion-Robust Egocentric 3D Hand Motion Recovery
• Abstract summary: Egocentric video offers scalable manipulation data for embodied AI.<n>However, recovering metric 3D hand trajectories is challenging due to severe object occlusion and frequent out-of-sight gaps.<n>We introduce DreamHand, an offline clip-level framework that extracts features via a Deterministic Clean-Latent and decodes them with a Bidirectional Spatiotemporal Decoder.
• Score: 51.92897182932217
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Egocentric video offers scalable manipulation data for embodied AI, yet recovering metric 3D hand trajectories remains challenging due to severe object occlusion and frequent out-of-sight gaps. Existing single-frame and windowed temporal regressors fail when hand shortly leaves the frame, while recent video diffusion models (VDMs) rely on heavy, stochastic multi-step sampling as pixel-space renderers. We instead repurpose VDM into a deterministic geometry encoder. A single forward pass over the clean latent exposes scene content beyond current observations, including occluded and out-of-sight hands. We introduce DreamHand, an offline clip-level framework that extracts features via a Deterministic Clean-Latent Encoder and decodes them with a Bidirectional Spatiotemporal Decoder. DreamHand recovers continuous bimanual trajectories with metric placement and no external detector, while a Ray-Based Camera Solver supports a second configuration that needs no test-time camera intrinsics. Across five egocentric benchmarks, DreamHand sets a new state of the art, cutting MPJPE-p by 30% on occlusion-heavy ARCTIC and 40% on HOT3D. These gains reach 46%-61% once out-of-sight hands are included in the evaluation, offering a scalable path from everyday human video to robot manipulation data.

Related papers

• Manifold4D: Denoising on Point Cloud Rendered Manifolds for Video Re-shooting [16.163736801043942]
  Video re-shooting re-renders a monocular video of a dynamic scene along a user-specified camera trajectory.<n>The dominant recipe supplies the target geometry explicitly.<n>We propose MANIFOLD4D, which injects the render directly into the initial noise of flow matching.
  arXiv  Detail & Related papers  (2026-08-28T10:39:52Z)
• KineBench: Benchmarking Embodied World Models via IDM-Free Kinematic Grounding [58.541098216700796]
  evaluating the physical consistency of embodied world models is a critical open challenge.<n>Existing frameworks almost exclusively rely on Inverse Dynamics Models(IDMs) for action extraction.<n>We present KineBench, an IDM-free closed-loop benchmark for EWMs.
  arXiv  Detail & Related papers  (2026-07-22T08:04:17Z)
• ReViV: Reconstructing the Viewer and the View in 4D from Monocular Egocentric Video [51.951128822418774]
  ReViV is a framework for holistic egocentric 4D reconstruction.<n>Powered by a Masked Generative Egocentric Transformer, ReViV operates within a single feed-forward architecture.
  arXiv  Detail & Related papers  (2026-07-20T10:22:53Z)
• KineFuse: Kinematic-Aware Haptic Fusion for In-Hand Occluded-Object Pose Tracking [18.013999575547427]
  In-hand manipulation requires continuous 6D pose tracking, yet the manipulating fingers inevitably occlude the object from the camera.<n>We study how to structure the sparse haptic signals already available on multi-fingered hands, including proprioception, proximal force/torque, and binary contact.<n>We propose a kinematic-aware finger-level encoder and systematically compare it against four alternative designs through three levels of evaluation.
  arXiv  Detail & Related papers  (2026-07-16T11:07:47Z)
• HandsOnWorld: Unconstrained Egocentric Video Generation with Camera-Disentangled Hand Control [51.913695868638634]
  We present HandsOnWorld, a framework for hand-controlled egocentric video generation that forgoes multi-view and marker-based motion capture, learning instead from unconstrained monocular video.<n>We build EgoVid-Pro, a dataset of clean, protagonist-only hand trajectories spanning 103K clips and roughly 12M frames across diverse everyday scenes.<n>We also propose the Plcker Hand Map, a 3D-aware control signal that extends Plcker-ray representations from camera rays to the hand surface, disentangling camera and hand motion at the representation level.
  arXiv  Detail & Related papers  (2026-07-02T12:14:02Z)
• The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction [36.627109956900256]
  ViDiHand is a pipeline for 4D hand motion reconstruction from egocentric video.<n>It is based on a pretrained video diffusion model that specializes its features for hands while preserving its world priors.<n>On ARCTIC, HOT3D, and HOI4D, ViDiHand substantially outperforms prior methods.
  arXiv  Detail & Related papers  (2026-06-29T13:53:00Z)
• Controllable Egocentric Video Generation via Occlusion-Aware Sparse 3D Hand Joints [87.13154261503168]
  Motion-controllable video generation is crucial for egocentric applications in virtual reality and embodied AI.<n>Existing methods often struggle to achieve 3D-consistent fine-grained hand articulation.<n>We propose a novel framework that generates egocentric videos from a single reference frame.
  arXiv  Detail & Related papers  (2026-03-12T10:02:23Z)
• QueryOcc: Query-based Self-Supervision for 3D Semantic Occupancy [9.514718375634345]
  Learning 3D scene geometry and semantics from images is a core challenge in computer vision and a key capability for autonomous driving.<n>Since large-scale 3D annotation is expensive, recent work explores self-supervised learning directly from sensor data without manual labels.<n>We introduce QueryOcc, a query-based self-supervised framework that learns continuous 3D occupancy directly through independent 4D semantic queries.
  arXiv  Detail & Related papers  (2025-11-21T13:05:42Z)
• 3DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generation [83.98251722144195]
  Previous methods on controllable video generation primarily leverage 2D control signals to manipulate object motions.<n>We introduce 3DTrajMaster, a robust controller that regulates multi-entity dynamics in 3D space.<n>We show that 3DTrajMaster sets a new state-of-the-art in both accuracy and generalization for controlling multi-entity 3D motions.
  arXiv  Detail & Related papers  (2024-12-10T18:55:13Z)
• STRIDE: Single-video based Temporally Continuous Occlusion-Robust 3D Pose Estimation [27.854074900345314]
  We propose STRIDE (Single-video based TempoRally contInuous Occlusion-Robust 3D Estimation Pose), a novel Test-Time Training (TTT) approach to fit a human motion prior to each video.<n>Our framework demonstrates flexibility by being model-agnostic, allowing us to use any off-the-shelf 3D pose estimation method for improving robustness and temporal consistency.
  arXiv  Detail & Related papers  (2023-12-24T11:05:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.