HandsOnWorld: Unconstrained Egocentric Video Generation with Camera-Disentangled Hand Control

• URL: http://arxiv.org/abs/2607.02075v1
• Date: Thu, 02 Jul 2026 12:14:02 GMT
• Title: HandsOnWorld: Unconstrained Egocentric Video Generation with Camera-Disentangled Hand Control
• Abstract summary: We present HandsOnWorld, a framework for hand-controlled egocentric video generation that forgoes multi-view and marker-based motion capture, learning instead from unconstrained monocular video.<n>We build EgoVid-Pro, a dataset of clean, protagonist-only hand trajectories spanning 103K clips and roughly 12M frames across diverse everyday scenes.<n>We also propose the Plcker Hand Map, a 3D-aware control signal that extends Plcker-ray representations from camera rays to the hand surface, disentangling camera and hand motion at the representation level.
• Score: 51.913695868638634
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present HandsOnWorld, a framework for hand-controlled egocentric video generation that forgoes multi-view and marker-based motion capture, learning instead from unconstrained monocular video. Such generality is bottlenecked by the scarcity of scalable 3D hand annotations: large egocentric corpora lack finger-level labels, whereas precise hand datasets are confined to narrow, instrumented settings, limiting prior hand-controlled generators to restricted scene distributions. We instead annotate 3D hands directly on in-the-wild egocentric video through monocular reconstruction, introducing a protagonist-centered annotation pipeline that filters the reconstructions at the action-semantic, image-quality, and 3D-geometric levels to build EgoVid-Pro, a dataset of clean, protagonist-only hand trajectories spanning 103K clips and roughly 12M frames across diverse everyday scenes. To resolve the camera-hand entanglement induced by large ego-motion, we further propose the Plücker Hand Map, a 3D-aware control signal that extends Plücker-ray representations from camera rays to the hand surface, disentangling camera and hand motion at the representation level. Experiments show that \method surpasses prior hand-controlled generators in reconstruction fidelity and control accuracy, and generalizes to out-of-distribution everyday scenes beyond the laboratory datasets on which prior methods rely.

Related papers

• EventEgoHands++: Event-based Egocentric 3D Hand Mesh Reconstruction with Real Dataset [3.8731899020772165]
  We propose a framework for event-based 3D hand mesh reconstruction from an egocentric viewpoint.<n>The proposed method incorporates a Hand Detector that estimates instance-level bounding boxes and masks for both the left and right hands.<n>We also introduce Adaptive Attention, which dynamically gates the attention based on these detection results.
  arXiv  Detail & Related papers  (2026-09-15T13:46:02Z)
• DreamHand: Repurposing Video Diffusion Models for Occlusion-Robust Egocentric 3D Hand Motion Recovery [51.92897182932217]
  Egocentric video offers scalable manipulation data for embodied AI.<n>However, recovering metric 3D hand trajectories is challenging due to severe object occlusion and frequent out-of-sight gaps.<n>We introduce DreamHand, an offline clip-level framework that extracts features via a Deterministic Clean-Latent and decodes them with a Bidirectional Spatiotemporal Decoder.
  arXiv  Detail & Related papers  (2026-08-20T17:46:24Z)
• EgoForce: Forearm-Guided Camera-Space 3D Hand Pose from a Monocular Egocentric Camera [40.28396219356337]
  EgoForce is a monocular 3D hand reconstruction framework that recovers robust, absolute 3D hand pose.<n>It operates across fisheye, perspective, and distorted wide-FOV camera models using a single unified network.<n>EgoForce achieves state-of-the-art 3D accuracy, reducing camera-space MPJPE by up to 28%.
  arXiv  Detail & Related papers  (2026-05-12T17:59:56Z)
• SHOW3D: Capturing Scenes of 3D Hands and Objects in the Wild [17.556955295385908]
  Existing hand-object interaction datasets are predominantly captured in controlled studio settings.<n>We introduce a novel marker-less multi-camera system that allows for nearly unconstrained mobility in genuinely in-the-wild conditions.<n>We present SHOW3D, the first large-scale dataset with 3D annotations that show hands interacting with objects in diverse real-world environments.
  arXiv  Detail & Related papers  (2026-03-30T17:58:27Z)
• Controllable Egocentric Video Generation via Occlusion-Aware Sparse 3D Hand Joints [87.13154261503168]
  Motion-controllable video generation is crucial for egocentric applications in virtual reality and embodied AI.<n>Existing methods often struggle to achieve 3D-consistent fine-grained hand articulation.<n>We propose a novel framework that generates egocentric videos from a single reference frame.
  arXiv  Detail & Related papers  (2026-03-12T10:02:23Z)
• Hand2World: Autoregressive Egocentric Interaction Generation via Free-Space Hand Gestures [33.2764643227486]
  Egocentric interactive world models are essential for augmented reality and embodied AI, where visual generation must respond to user input with low latency, geometric consistency, and long-term stability.<n>We study egocentric interaction generation from a single scene image under free-space hand gestures, aiming to synthesize photorealistic videos in which hands enter the scene, interact with objects, and induce plausible world dynamics under head motion.<n>This setting introduces fundamental challenges, including distribution shift between free-space gestures and contact-heavy training data, ambiguity between hand motion and camera motion in monocular views, and the need for arbitrary-length video generation.
  arXiv  Detail & Related papers  (2026-02-10T09:51:07Z)
• Ego-Exo 3D Hand Tracking in the Wild with a Mobile Multi-Camera Rig [14.496137517475743]
  We introduce a novel marker-less multi-camera system designed to capture precise 3D hands and objects.<n>We combine a lightweight, back-mounted capture rig with eight exocentric cameras, and a user-worn Meta Quest 3 headset, which contributes two egocentric views.<n>We collect an annotated dataset featuring synchronized multi-view images and precise 3D hand poses.
  arXiv  Detail & Related papers  (2025-10-02T22:26:03Z)
• 3DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generation [83.98251722144195]
  Previous methods on controllable video generation primarily leverage 2D control signals to manipulate object motions.<n>We introduce 3DTrajMaster, a robust controller that regulates multi-entity dynamics in 3D space.<n>We show that 3DTrajMaster sets a new state-of-the-art in both accuracy and generalization for controlling multi-entity 3D motions.
  arXiv  Detail & Related papers  (2024-12-10T18:55:13Z)
• Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objects [89.95728475983263]
  holistic 3Dunderstanding of such interactions from egocentric views is important for tasks in robotics, AR/VR, action recognition and motion generation. We design the HANDS23 challenge based on the AssemblyHands and ARCTIC datasets with carefully designed training and testing splits. Based on the results of the top submitted methods and more recent baselines on the leaderboards, we perform a thorough analysis on 3D hand(-object) reconstruction tasks.
  arXiv  Detail & Related papers  (2024-03-25T05:12:21Z)
• Consistent 3D Hand Reconstruction in Video via self-supervised Learning [67.55449194046996]
  We present a method for reconstructing accurate and consistent 3D hands from a monocular video. detected 2D hand keypoints and the image texture provide important cues about the geometry and texture of the 3D hand. We propose $rm S2HAND$, a self-supervised 3D hand reconstruction model.
  arXiv  Detail & Related papers  (2022-01-24T09:44:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.