GenVid2Robot: From Video Generation to Robot Manipulation via Rigid-Geometric Consistency

• URL: http://arxiv.org/abs/2607.09191v1
• Date: Fri, 10 Jul 2026 08:32:47 GMT
• Title: GenVid2Robot: From Video Generation to Robot Manipulation via Rigid-Geometric Consistency
• Abstract summary: GenVid2Robot is a framework that converts generated video motion into executable real-robot manipulation trajectories.<n>Experiments demonstrate that GenVid2Robot improves the reliability of generated-video-guided manipulation by grounding visual motion priors with sparse metric geometry.
• Score: 19.566368780107613
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generated videos provide useful visual motion priors for robot manipulation, but their visual plausibility does not imply physical executability. A generated video usually lacks metric geometry, grasp grounding, robot kinematic feasibility, and execution-time feedback, which makes direct trajectory replay unreliable in real-world manipulation. This paper presents GenVid2Robot, a rigid-geometric consistency framework that converts generated video motion into executable real-robot manipulation trajectories. Given an initial RGB-D observation and a task instruction, GenVid2Robot samples task-relevant semantic anchors from the real first frame, tracks these anchors through generated video candidates, and verifies whether the resulting 2D motion can be explained by first-frame RGB-D anchors under a sparse relative $SE(3)$ model. In this way, generated videos are treated as uncertain visual motion hypotheses rather than direct robot demonstrations. Only geometrically consistent motion is transferred to the robot. The accepted relative motion is then applied to the real grasp-time TCP pose selected by mask-constrained grasping, producing a grasp-conditioned execution trajectory that is consistent with both the visual motion prior and the physical grasp configuration. To reduce execution mismatch caused by RGB-D noise, calibration residuals, and small contact-induced displacement, a bounded depth-compensation module corrects local depth-direction errors without assuming full online replanning. Real-robot experiments demonstrate that GenVid2Robot improves the reliability of generated-video-guided manipulation by grounding visual motion priors with sparse metric geometry, grasp constraints, robot feasibility checking, and bounded execution feedback.

Related papers

• RGB-D Video Generation for Improving Human-to-Robot Object Handover Prediction [51.76737991200891]
  Human-to-robot (H2R) object handover is a fundamental capability for human-robot collaboration, yet progress is hindered by the scarcity of large-scale, human-centric datasets.<n>We introduce Hand2Bot, an RGB-D video dataset that provides rich contextual information such as body posture and facial expressions.<n>We propose PassGen, a generative pipeline that synthesizes realistic handover sequences while ensuring hand-object consistency.
  arXiv  Detail & Related papers  (2026-08-13T09:55:59Z)
• Self Supervised Learning from Automatically Generated Demonstrations for Visual Robotic Manipulation [0.11083289076967894]
  This paper presents a self supervised visual manipulation method in which a robot automatically generates demonstrations around a target pose.<n>The proposed pipeline uses ROS2 and Isaac Sim to collect labeled image-pose pairs without requiring explicit camera to robot.<n>The method is evaluated both in simulation and on a real UR5e collaborative robot equipped with a gripper and a monocular camera.
  arXiv  Detail & Related papers  (2026-08-01T21:34:39Z)
• PhysV2A: Reachability-Gated and Semantic-Mask-Constrained Feasibility Completion for Video-to-Robot Manipulation [17.70834422785338]
  Video-based manipulation provides object-centric motion priors from human demonstrations, generated videos, or RGB-D observations.<n>PhysV2A is a framework for converting video-derived 6D object motion into robot-executable manipulation trajectories.
  arXiv  Detail & Related papers  (2026-07-10T12:43:00Z)
• Supervise What Survives: Geometry-Guided VLA Adaptation from Synthetic Robot Videos [43.32573764638152]
  We argue that deriving low-level control from generated visuals is a mismatched abstraction.<n>We propose textbfGRA, which extracts the geometric content as future 2D end-effector waypoints.<n>On real-robot tasks, GRA outperforms pseudo-action baselines under matched data budgets.
  arXiv  Detail & Related papers  (2026-06-23T11:35:13Z)
• GRAIL: Generating Humanoid Loco-Manipulation from 3D Assets and Video Priors [113.71148915419246]
  We present GRAIL, a digital generation pipeline that composes 3D assets, simulator-ready scenes, and priors from video foundation models (VFMs) to synthesize interactions without rebuilding physical environments or teleoperating the robot.<n>GRAIL starts from fully specified 3D configurations in which object geometry, camera parameters, metric scale, environment depth, and a robot-proportioned character are known before video generation and reused during reconstruction.<n>We retarget the recovered motions to a humanoid robot and train complementary task-general morphology trackers.<n>GRAIL produces over 20,000 sequences spanning pick-up, object manipulation, sitting
  arXiv  Detail & Related papers  (2026-06-03T17:57:45Z)
• PointAction: 3D Points as Universal Action Representations for Robot Control [42.12570897523391]
  We present PointAction, a framework that bridges video predictions to robot actions through explicit point-based 4D modeling.<n>PointAction achieves state-of-the-art 4D generation quality on robot scenes, outperforms existing baselines in simulation, and generalizes to two real robot arms unseen during pretraining.
  arXiv  Detail & Related papers  (2026-06-02T17:30:50Z)
• GEM-4D: Geometry-Enhanced Video World Models for Robot Manipulation [72.52773248997929]
  Video world models can generate realistic futures from a single instruction, but they often fail to preserve consistent point-level motion over time.<n>We present GEM-4D, a geometry-grounded video world model that injects dense 4D correspondence supervision into the video generative backbone during training.<n>In inverse dynamics module converts correspondence-consistent video rollouts into executable robot trajectories, enabling direct deployment in both real-world and simulated manipulation.
  arXiv  Detail & Related papers  (2026-05-20T21:36:44Z)
• EVA: Aligning Video World Models with Executable Robot Actions via Inverse Dynamics Rewards [47.255807408091755]
  Video generative models are increasingly used as world models for robotics.<n>Current video world models lack explicit executability constraints.<n>We introduce Executable Video Alignment (EVA), a reinforcement-learning framework for aligning video world models.
  arXiv  Detail & Related papers  (2026-03-18T15:02:19Z)
• DRAW2ACT: Turning Depth-Encoded Trajectories into Robotic Demonstration Videos [24.681248200255975]
  Video models provide powerful real-world simulators for embodied AI but remain limited in controllability for robotic manipulation.<n>We present DRAW2ACT, a trajectory-conditioned video generation framework that extracts multiple representations from the input trajectory.<n>We show that DRAW2ACT achieves superior visual fidelity and consistency while yielding higher manipulation success rates compared to existing baselines.
  arXiv  Detail & Related papers  (2025-12-16T09:11:36Z)
• VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulation [53.63540587160549]
  VidBot is a framework enabling zero-shot robotic manipulation using learned 3D affordance from in-the-wild monocular RGB-only human videos.<n> VidBot paves the way for leveraging everyday human videos to make robot learning more scalable.
  arXiv  Detail & Related papers  (2025-03-10T10:04:58Z)
• Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstruction [51.49400490437258]
  This work develops a method for imitating articulated object manipulation from a single monocular RGB human demonstration. We first propose 4D Differentiable Part Models (4D-DPM), a method for recovering 3D part motion from a monocular video. Given this 4D reconstruction, the robot replicates object trajectories by planning bimanual arm motions that induce the demonstrated object part motion. We evaluate 4D-DPM's 3D tracking accuracy on ground truth annotated 3D part trajectories and RSRD's physical execution performance on 9 objects across 10 trials each on a bimanual YuMi robot.
  arXiv  Detail & Related papers  (2024-09-26T17:57:16Z)
• Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulation [65.46610405509338]
  We seek to learn a generalizable goal-conditioned policy that enables zero-shot robot manipulation. Our framework,Track2Act predicts tracks of how points in an image should move in future time-steps based on a goal. We show that this approach of combining scalably learned track prediction with a residual policy enables diverse generalizable robot manipulation.
  arXiv  Detail & Related papers  (2024-05-02T17:56:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.