EVA: Aligning Video World Models with Executable Robot Actions via Inverse Dynamics Rewards

• URL: http://arxiv.org/abs/2603.17808v2
• Date: Tue, 24 Mar 2026 17:12:47 GMT
• Title: EVA: Aligning Video World Models with Executable Robot Actions via Inverse Dynamics Rewards
• Abstract summary: Video generative models are increasingly used as world models for robotics.<n>Current video world models lack explicit executability constraints.<n>We introduce Executable Video Alignment (EVA), a reinforcement-learning framework for aligning video world models.
• Score: 47.255807408091755
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video generative models are increasingly used as world models for robotics, where a model generates a future visual rollout conditioned on the current observation and task instruction, and an inverse dynamics model (IDM) converts the generated frames into executable robot actions. However, current video world models lack explicit executability constraints. As a result, visually coherent rollouts may still violate rigid-body and kinematic consistency, producing unstable or infeasible control commands when decoded by an IDM. We refer to this mismatch between visual generation and physically executable control as the executability gap. While this gap can be mitigated at inference time using techniques such as rejection sampling, such approaches are inefficient due to the high cost of video generation. In this paper, we leverage the executability gap as a training signal and introduce Executable Video Alignment (EVA), a reinforcement-learning post-training framework for aligning video world models. EVA trains an inverse dynamics model on real robot trajectories and repurposes it as a reward model that evaluates generated videos through the action sequences they induce, encouraging smooth motions measured by velocity, acceleration, and jerk while penalizing actions that violate embodiment constraints. Importantly, the reward remains informative even when generated videos contain severe visual artifacts, since such artifacts typically translate into unstable or out-of-bound actions. Experiments on the RoboTwin benchmark and a real bimanual robot show that EVA reduces embodiment-specific artifacts in generated rollouts and improves downstream task execution success.

Related papers

• KineBench: Benchmarking Embodied World Models via IDM-Free Kinematic Grounding [58.541098216700796]
  evaluating the physical consistency of embodied world models is a critical open challenge.<n>Existing frameworks almost exclusively rely on Inverse Dynamics Models(IDMs) for action extraction.<n>We present KineBench, an IDM-free closed-loop benchmark for EWMs.
  arXiv  Detail & Related papers  (2026-07-22T08:04:17Z)
• GEM-4D: Geometry-Enhanced Video World Models for Robot Manipulation [72.52773248997929]
  Video world models can generate realistic futures from a single instruction, but they often fail to preserve consistent point-level motion over time.<n>We present GEM-4D, a geometry-grounded video world model that injects dense 4D correspondence supervision into the video generative backbone during training.<n>In inverse dynamics module converts correspondence-consistent video rollouts into executable robot trajectories, enabling direct deployment in both real-world and simulated manipulation.
  arXiv  Detail & Related papers  (2026-05-20T21:36:44Z)
• From Imagined Futures to Executable Actions: Mixture of Latent Actions for Robot Manipulation [88.39072412680633]
  We propose MoLA, a control-oriented interface that transforms imagined future videos into executable representations.<n>We evaluate our approach on simulated benchmarks and real-world robot manipulation tasks.
  arXiv  Detail & Related papers  (2026-05-12T14:15:16Z)
• RoboWM-Bench: A Benchmark for Evaluating World Models in Robotic Manipulation [23.57524297963567]
  RoboWM-Bench is a manipulation-centric benchmark for embodiment-grounded evaluation of video world models.<n>We evaluate state-of-the-art video world models and find that reliably generating physically executable behaviors remains an open challenge.
  arXiv  Detail & Related papers  (2026-04-21T05:09:56Z)
• Learning Physics from Pretrained Video Models: A Multimodal Continuous and Sequential World Interaction Models for Robotic Manipulation [63.04810454548667]
  We introduce PhysGen, a scalable continuous and sequential world interaction framework to solve robotic manipulation tasks.<n>By treating the pretrained video model as a proxy for a physics simulator, PhysGen models the dynamic interplay between the external environment and robot actions.<n>We introduce a multimodal continuous representation that unifies video and action into shared physical tokens, bridging the gap between discrete video generation and continuous robotic control.
  arXiv  Detail & Related papers  (2026-02-18T14:58:18Z)
• Physical Autoregressive Model for Robotic Manipulation without Action Pretraining [65.8971623698511]
  We build upon autoregressive video generation models to propose a Physical Autoregressive Model (PAR)<n>PAR leverages the world knowledge embedded in video pretraining to understand physical dynamics without requiring action pretraining.<n>Experiments on the ManiSkill benchmark show that PAR achieves a 100% success rate on the PushCube task.
  arXiv  Detail & Related papers  (2025-08-13T13:54:51Z)
• Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrations [19.28925489415787]
  RIGVid enables robots to perform complex manipulation tasks by imitating AI-generated videos.<n>A video diffusion model generates potential demonstration videos, and a vision-language model automatically filters out results that do not follow the command.<n>A 6D pose tracker then extracts object trajectories from the video, and the trajectories are retargeted to the robot in an embodiment-agnostic fashion.
  arXiv  Detail & Related papers  (2025-07-01T17:39:59Z)
• ORV: 4D Occupancy-centric Robot Video Generation [33.360345403049685]
  Acquiring real-world robotic simulation data through teleoperation is notoriously time-consuming and labor-intensive.<n>We propose ORV, an Occupancy-centric Robot Video generation framework, which utilizes 4D semantic occupancy sequences as a fine-grained representation.<n>By leveraging occupancy-based representations, ORV enables seamless translation of simulation data into photorealistic robot videos, while ensuring high temporal consistency and precise controllability.
  arXiv  Detail & Related papers  (2025-06-03T17:00:32Z)
• VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulation [53.63540587160549]
  VidBot is a framework enabling zero-shot robotic manipulation using learned 3D affordance from in-the-wild monocular RGB-only human videos.<n> VidBot paves the way for leveraging everyday human videos to make robot learning more scalable.
  arXiv  Detail & Related papers  (2025-03-10T10:04:58Z)
• Learning Real-World Action-Video Dynamics with Heterogeneous Masked Autoregression [23.99292102237088]
  We propose Heterogeneous Masked Autoregression (HMA) for modeling action-video dynamics.<n>After post-training, this model can be used as a video simulator for evaluating policies and generating synthetic data.
  arXiv  Detail & Related papers  (2025-02-06T18:38:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.