IOI: Decoupling Kinematics and Physics for Interactive World Models

• URL: http://arxiv.org/abs/2606.23296v1
• Date: Mon, 22 Jun 2026 13:09:34 GMT
• Title: IOI: Decoupling Kinematics and Physics for Interactive World Models
• Abstract summary: We propose IOI, a hybrid interactive world model integrating kinematics with learned physical dynamics.<n> IOI introduces explicit kinematic guidance, computing kinematics sequences for accurate motion trajectories.<n> experiments on the RoboTwin benchmark validate IOI across kinematic fidelity, out-of-distribution, and policy evaluation.
• Score: 46.3330122411516
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Developing generalist embodied agents requires interactive environments providing visually realistic feedback and accurate action-conditioned dynamics. Interactive world models address this by simulating such complex dynamics. However, purely data-driven methods struggle to ensure precise control alignment and physically plausible visual feedback due to a lack of explicit structural constraints. To address this, we propose IOI, a hybrid interactive world model integrating analytical kinematic priors with learned physical dynamics. Unlike data-driven approaches prone to spatiotemporal drift, IOI introduces explicit kinematic guidance, computing forward kinematics from action sequences for accurate motion trajectories. These trajectories are rendered into synchronized front, side, and top orthographic projections, eliminating the need for extrinsic camera calibration. A Multi-view Kinematic Aggregation and Injection module fuses these geometric cues and injects them into the video generator, providing geometry-consistent guidance. Conditioning video generation on these deterministic trajectories establishes a synergy between the analytical simulator and the world model. Decoupling deterministic motion into the kinematic prior frees the generator to model stochastic physical interactions. Experiments on the RoboTwin benchmark validate IOI across kinematic fidelity, out-of-distribution (OOD) generalization, and policy evaluation. IOI achieves state-of-the-art simulation performance and robust zero-shot generalization to unseen OOD tasks. Furthermore, IOI serves as a reliable policy evaluator, yielding success rates closely aligning with ground-truth physics simulators. On real-world platforms, policies trained on IOI-synthesized data match those trained on teleoperation demonstrations, solidifying its practical value for embodied policy learning.

Related papers

• DyG$^2$T: Modeling Object Dynamics with 3D Gaussian Temporal-Spatial Particle Graph Transformer [86.4618122245946]
  DyG$2$T is a dynamics modeling framework that infers object motion trajectories by spatially completing and temporally discriminating Key Point representations.<n>Experiments on both synthetic and real-world datasets demonstrate that DyG$2$T achieves accurate dynamics modeling and reasoning.
  arXiv  Detail & Related papers  (2026-08-19T03:44:28Z)
• WorldSimProbe: Diagnosing Simulator Faithfulness in Action-Conditioned World Models for Embodied Manipulation [85.15806404962193]
  Action-conditioned world models (ACWMs) promise to provide embodied AI with scalable predictive simulators for planning, policy evaluation, and data generation.<n>We evaluate ACWMs through the observable capabilities expected of physical simulators.<n>WorldSimProbe reveals systematic action-realization degradation across control variation, structured failures in interaction grounding and dynamics.
  arXiv  Detail & Related papers  (2026-08-10T08:48:06Z)
• GAUGE: A Measurement-Grounded Benchmark for Physical Fidelity in Simulation Engines and Video World Models [66.77745044586173]
  GAUGE is a real-world-grounded diagnostic benchmark for evaluating how numerical simulators and generative video world models reproduce or deviate from real-world physics.<n>We benchmark Isaac Sim, Genesis, and Newton on 14 task families using generalized trajectory errors, and evaluate 6 image-to-video models on 5 rigid-body tasks.<n>Our results reveal no uniformly faithful physics engine, with the largest discrepancies arising in impulsive contact, rapid textile motion, and volumetric deformation.
  arXiv  Detail & Related papers  (2026-08-06T12:19:38Z)
• Policy-as-Data: Learning Generalizable HOI Diffusion Models from Simulated Physics [23.400128774539812]
  We propose a novel framework to overcome the data-scarcity bottleneck in HOI generation.<n>Our method demonstrates enhanced generalization to unseen objects and the capability of long-horizon generation.
  arXiv  Detail & Related papers  (2026-06-22T03:32:35Z)
• Direct Dynamic Retargeting for Humanoid Imitation Learning from Videos [6.238518905285812]
  Imitation Learning from monocular video demonstrations provides a scalable approach for teaching complex skills to humanoid robots.<n>We propose Direct Dynamic Retargeting ( DDR), a novel single-stage framework that generates high-fidelity, dynamically feasible trajectories directly from expert videos.<n>Our experiments demonstrate that bypassing the geometric bias allows DDR to outperform state-of-the-art baselines in demonstration tracking accuracy.
  arXiv  Detail & Related papers  (2026-05-22T15:33:40Z)
• ACWM-Phys: Investigating Generalized Physical Interaction in Action-Conditioned Video World Models [30.527810700174488]
  Action-conditioned world models (ACWMs) have shown strong promise for video prediction and decision-making.<n>We introduce ACWM-Phys, a new benchmark for evaluating action-conditioned prediction under diverse physical dynamics.
  arXiv  Detail & Related papers  (2026-05-09T00:00:47Z)
• EA-WM: Event-Aware Generative World Model with Structured Kinematic-to-Visual Action Fields [15.319293934673915]
  We present EA-WM, an EventAware Generative World Model that closes the loop between kinematic control and visual perception.<n>We introduce event-aware bidirectional fusion blocks that modulate cross-branch attention, capturing object state changes and precise interaction dynamics.<n>EA-WM achieves state-of-the-art performance, outperforming existing baselines by a significant margin.
  arXiv  Detail & Related papers  (2026-05-07T13:06:19Z)
• Contact-Aware Neural Dynamics [42.83872574010789]
  We propose an implicit sim-to-real framework that learns to align the simulator's dynamics with contact information.<n>We show that using tactile contact information from robotic hands can effectively model the non-smooth discontinuities inherent in contact-rich tasks.
  arXiv  Detail & Related papers  (2026-01-19T07:56:13Z)
• Spatial-Temporal Graph Diffusion Policy with Kinematic Modeling for Bimanual Robotic Manipulation [88.83749146867665]
  Existing approaches learn a policy to predict a distant next-best end-effector pose.<n>They then compute the corresponding joint rotation angles for motion using inverse kinematics.<n>We propose Kinematics enhanced Spatial-TemporAl gRaph diffuser.
  arXiv  Detail & Related papers  (2025-03-13T17:48:35Z)
• Pre-Trained Video Generative Models as World Simulators [59.546627730477454]
  We propose Dynamic World Simulation (DWS) to transform pre-trained video generative models into controllable world simulators.<n>To achieve precise alignment between conditioned actions and generated visual changes, we introduce a lightweight, universal action-conditioned module.<n> Experiments demonstrate that DWS can be versatilely applied to both diffusion and autoregressive transformer models.
  arXiv  Detail & Related papers  (2025-02-10T14:49:09Z)
• InterDyn: Controllable Interactive Dynamics with Video Diffusion Models [50.38647583839384]
  We propose InterDyn, a framework that generates videos of interactive dynamics given an initial frame and a control signal encoding the motion of a driving object or actor.<n>Our key insight is that large video generation models can act as both neurals and implicit physics simulators'', having learned interactive dynamics from large-scale video data.
  arXiv  Detail & Related papers  (2024-12-16T13:57:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.