Supervise What Survives: Geometry-Guided VLA Adaptation from Synthetic Robot Videos

• URL: http://arxiv.org/abs/2606.24448v1
• Date: Tue, 23 Jun 2026 11:35:13 GMT
• Title: Supervise What Survives: Geometry-Guided VLA Adaptation from Synthetic Robot Videos
• Abstract summary: We argue that deriving low-level control from generated visuals is a mismatched abstraction.<n>We propose textbfGRA, which extracts the geometric content as future 2D end-effector waypoints.<n>On real-robot tasks, GRA outperforms pseudo-action baselines under matched data budgets.
• Score: 43.32573764638152
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-Language-Action (VLA) models require large-scale video-action pairs, yet real teleoperation remains scarce. While generated robot videos offer a scalable alternative, existing methods treat them as real robot data by recovering pseudo-actions from synthesized pixels. We argue that deriving low-level control from generated visuals is a mismatched abstraction. A video captures only \emph{geometry}: the spatial trajectory representing the \emph{where} of a task. A real demonstration captures \emph{control}: the exact motor commands representing the \emph{how}. Human-to-robot video generation preserves these unequally: the visible geometry survives the generation process, while the underlying control signals are lost. This \textbf{Asymmetric Preservation Principle} dictates a clean rule: this surviving geometry should solely supervise visual perception, leaving control to real demonstrations. Following this principle, we propose \textbf{GRA} (\textbf{G}eometry-guided \textbf{R}epresentation \textbf{A}lignment), which extracts the geometric content as future 2D end-effector waypoints, computed from the source human video through pose estimation, retargeting, simulation, and calibrated projection, and routes them to the VLA vision backbone via an auxiliary 2D head. The action head is trained on real demonstrations only. During fine-tuning, the waypoint loss persists as a \textbf{spatial representation anchor} that prevents the backbone from losing its geometric grounding. On real-robot tasks, GRA outperforms pseudo-action baselines under matched data budgets and narrows the gap to policies trained with substantially more real demonstrations, suggesting that correctly routed geometry bridges generated videos to robot policies more reliably than recovered actions.

Related papers

• GeoLAM: Learning Geometry-Grounded Latent Actions from Unlabeled Human Videos [8.414561051327915]
  We present GeoLAM, a framework for learning geometry-grounded latent actions from action-free human videos.<n>GeoLAM combines future-frame reconstruction through a frozen geometric feature hierarchy with motion supervision from a training-only 4D geometry teacher.
  arXiv  Detail & Related papers  (2026-09-15T12:31:34Z)
• GenVid2Robot: From Video Generation to Robot Manipulation via Rigid-Geometric Consistency [19.566368780107613]
  GenVid2Robot is a framework that converts generated video motion into executable real-robot manipulation trajectories.<n>Experiments demonstrate that GenVid2Robot improves the reliability of generated-video-guided manipulation by grounding visual motion priors with sparse metric geometry.
  arXiv  Detail & Related papers  (2026-07-10T08:32:47Z)
• GRAIL: Generating Humanoid Loco-Manipulation from 3D Assets and Video Priors [113.71148915419246]
  We present GRAIL, a digital generation pipeline that composes 3D assets, simulator-ready scenes, and priors from video foundation models (VFMs) to synthesize interactions without rebuilding physical environments or teleoperating the robot.<n>GRAIL starts from fully specified 3D configurations in which object geometry, camera parameters, metric scale, environment depth, and a robot-proportioned character are known before video generation and reused during reconstruction.<n>We retarget the recovered motions to a humanoid robot and train complementary task-general morphology trackers.<n>GRAIL produces over 20,000 sequences spanning pick-up, object manipulation, sitting
  arXiv  Detail & Related papers  (2026-06-03T17:57:45Z)
• Imagine2Real: Towards Zero-shot Humanoid-Object Interaction via Video Generative Priors [51.096845970243855]
  Whole-body Humanoid-Object Interaction (HOI) is bottlenecked by the scarcity of high-fidelity 3D data.<n>We propose Imagine2Real, a zero-shot HOI framework for flexible, geometry-free interaction.
  arXiv  Detail & Related papers  (2026-05-21T10:15:39Z)
• GEM-4D: Geometry-Enhanced Video World Models for Robot Manipulation [72.52773248997929]
  Video world models can generate realistic futures from a single instruction, but they often fail to preserve consistent point-level motion over time.<n>We present GEM-4D, a geometry-grounded video world model that injects dense 4D correspondence supervision into the video generative backbone during training.<n>In inverse dynamics module converts correspondence-consistent video rollouts into executable robot trajectories, enabling direct deployment in both real-world and simulated manipulation.
  arXiv  Detail & Related papers  (2026-05-20T21:36:44Z)
• Robotic Manipulation is Vision-to-Geometry Mapping ($f(v) \rightarrow G$): Vision-Geometry Backbones over Language and Video Models [65.05130114320734]
  We argue that the foundation for generalizable robotic control should be a vision-geometry backbone, rather than the widely adopted vision-language or video models.<n>We propose the Vision-Geometry-Action model, which directly conditions action generation on pretrained native 3D representations.<n> Specifically, VGA replaces conventional language or video backbones with a pretrained 3D world model, establishing a seamless vision-to-geometry mapping.
  arXiv  Detail & Related papers  (2026-04-14T15:57:16Z)
• CRISP: Contact-Guided Real2Sim from Monocular Video with Planar Scene Primitives [65.89192712575797]
  We introduce CRISP, a method that recovers simulatable human motion and scene geometry from monocular video.<n>Our approach reduces motion tracking failure rates from 55.2% to 6.9% on human-centric video benchmarks.<n>This demonstrates CRISP's ability to generate physically-valid human motion and interaction environments at scale.
  arXiv  Detail & Related papers  (2025-12-16T18:59:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.