ContactFlow: A video action conditioning that transfers across embodiments

• URL: http://arxiv.org/abs/2607.26579v1
• Date: Wed, 29 Jul 2026 07:59:47 GMT
• Title: ContactFlow: A video action conditioning that transfers across embodiments
• Abstract summary: Contact Flow is an embodiment-agnostic action representation that encodes manipulation through the trajectory of 3D contact points.<n>We train a large-scale video generative model on both human and robotic object interaction videos conditioned on Contact Flow.<n>We integrate this model into a propose-imagine-verify-act pipeline, where generated rollouts are assessed by a vision-language model before execution.
• Score: 23.81895581513295
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: World models offer a promising route toward robot planning by enabling agents to imagine and verify the consequences of actions before execution. However, current video-based world models often struggle to capture the physical constraints that govern manipulation, particularly contact. Further, their action conditioning is often constrained to specific embodiments such as parallel grippers. We propose \emph{Contact Flow}, an embodiment-agnostic action representation that encodes manipulation through the trajectory of 3D contact points between an actor and a target object. By discarding actor-specific appearance and kinematics, Contact Flow provides a shared conditioning signal for both human demonstrations and robotic execution. Therefore, we can train a large-scale video generative model on both human and robotic object interaction videos conditioned on Contact Flow, yielding a world model that predicts physically plausible manipulation outcomes. We integrate this model into a propose-imagine-verify-act pipeline, where generated rollouts are assessed by a vision-language model before execution. Experiments on the DROID dataset and real-world tabletop manipulation tasks demonstrate that Contact Flow enables transfer between human demonstrations and different robotic embodiments.

Related papers

• AnyWorld: Factorized Egocentric World Models for Cross-Embodiment Generalization [88.25615187835321]
  We propose AnyWorld, a cross-embodiment world modeling framework.<n>Our model factorizes an interaction into action, camera, and embodiment.<n>We train the model with large-scale human interaction pretraining.
  arXiv  Detail & Related papers  (2026-08-29T12:55:15Z)
• H2R-Bench: Benchmarking Human-to-Robot Manipulation Video Generation in World Models [56.60098789696351]
  H2R-Bench is a benchmark for evaluating cross-embodiment human-to-robot manipulation video generation.<n>Each benchmark instance contains a human demonstration video, target embodiment constraints, and source-grounded annotations.<n>We benchmark eleven state-of-the-art video generation models across six manipulation families and two robot embodiments.
  arXiv  Detail & Related papers  (2026-08-13T10:14:33Z)
• Masked Visual Actions for Unified World Modeling [96.12988421978463]
  We introduce Masked Visual Actions, a pixel-space control interface that expresses action as a partially revealed trajectory of an arbitrary entity in a video.<n>Revealing robot motion makes the model act as a forward dynamics model that predicts the scene's response to low-level robot actions, while revealing desired object motion makes the same model recover robot behavior consistent with that outcome.
  arXiv  Detail & Related papers  (2026-07-21T17:59:11Z)
• Hand-centric Human-to-Robot Trajectory Transfer from Video Demonstrations via Open-World Contact Localization [28.23926683554352]
  We present emphHOWTransfer, a hand-centric framework that distills human demonstrations into contact-aware, taxonomy-informed, and diverse robotic trajectories.<n>emphHOWTransfer recovers temporally consistent 3D hand motion and localizes temporal contact intervals by reasoning over observed hand-object interaction cues.<n>Experiments show that emphHOWTransfer enables accurate contact localization and high-quality robot motion with $86%$ success.
  arXiv  Detail & Related papers  (2026-06-09T11:53:29Z)
• From Imagined Futures to Executable Actions: Mixture of Latent Actions for Robot Manipulation [88.39072412680633]
  We propose MoLA, a control-oriented interface that transforms imagined future videos into executable representations.<n>We evaluate our approach on simulated benchmarks and real-world robot manipulation tasks.
  arXiv  Detail & Related papers  (2026-05-12T14:15:16Z)
• EmbodMocap: In-the-Wild 4D Human-Scene Reconstruction for Embodied Agents [85.77432303199176]
  We propose EmbodMocap, a portable and affordable data collection pipeline using two moving iPhones.<n>Our key idea is to jointly calibrate dual RGB-D sequences to reconstruct both humans and scenes.<n>Based on the collected data, we empower three embodied AI tasks: monocular human-scene-reconstruction, where we fine-tune feedforward models that output metric-scale, world-space aligned humans and scenes; physics-based character animation, where we prove our data could be used to scale human-object interaction skills and scene-aware motion tracking; and robot motion control, where we train a humanoid robot via
  arXiv  Detail & Related papers  (2026-02-26T16:53:41Z)
• DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos [110.98100817695307]
  We introduce DreamDojo, a foundation world model that learns diverse interactions and dexterous controls from 44k hours of egocentric human videos.<n>Our work enables several important applications based on generative world models, including live teleoperation, policy evaluation, and model-based planning.
  arXiv  Detail & Related papers  (2026-02-06T18:49:43Z)
• Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flow [21.658558775915267]
  We introduce Dream2Flow, a framework that bridges video generation and robotic control through 3D object flow as an intermediate representation.<n>Our method reconstructs 3D object motions from generated videos and formulates manipulation as object trajectory tracking.<n>Dream2Flow overcomes the embodiment gap and enables zero-shot guidance from pre-trained video models to manipulate objects of diverse categories.
  arXiv  Detail & Related papers  (2025-12-31T10:25:24Z)
• NovaFlow: Zero-Shot Manipulation via Actionable Flow from Generated Videos [13.84832813181084]
  NovaFlow is an autonomous manipulation framework that converts a task description into an actionable plan for a target robot.<n>We validate on rigid, articulated, and deformable object manipulation tasks using a table-top Franka arm and a Spot quadrupedal mobile robot.
  arXiv  Detail & Related papers  (2025-10-09T17:59:55Z)
• Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulation [65.46610405509338]
  We seek to learn a generalizable goal-conditioned policy that enables zero-shot robot manipulation. Our framework,Track2Act predicts tracks of how points in an image should move in future time-steps based on a goal. We show that this approach of combining scalably learned track prediction with a residual policy enables diverse generalizable robot manipulation.
  arXiv  Detail & Related papers  (2024-05-02T17:56:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.