WorldSimProbe: Diagnosing Simulator Faithfulness in Action-Conditioned World Models for Embodied Manipulation

• URL: http://arxiv.org/abs/2608.09298v1
• Date: Mon, 10 Aug 2026 08:48:06 GMT
• Title: WorldSimProbe: Diagnosing Simulator Faithfulness in Action-Conditioned World Models for Embodied Manipulation
• Abstract summary: Action-conditioned world models (ACWMs) promise to provide embodied AI with scalable predictive simulators for planning, policy evaluation, and data generation.<n>We evaluate ACWMs through the observable capabilities expected of physical simulators.<n>WorldSimProbe reveals systematic action-realization degradation across control variation, structured failures in interaction grounding and dynamics.
• Score: 85.15806404962193
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Action-conditioned world models (ACWMs) promise to provide embodied AI with scalable predictive simulators for planning, policy evaluation, and data generation. Realizing this promise requires precise action-conditioned transitions rather than merely plausible outputs. Yet their applicability remains difficult to establish because prevailing evaluations emphasize visual quality, task outcomes, or coarse rollout-level responsiveness without directly testing simulator fidelity. To address this gap, we evaluate ACWMs through the observable capabilities expected of physical simulators. Accordingly, we formalize Observable Simulator Contract, a minimal contract that any action-conditioned physical simulator should satisfy: supplied actions must induce corresponding agent motion, and environment responses must be grounded in that realized motion. To operationalize this contract, we introduce WorldSimProbe, comprising five controlled suites spanning local control sensitivity, global trajectory variation, source-diverse actions, interaction grounding, and dynamics. Suite-specific evaluators assess simulator-relative calibration, dense action-to-motion correspondence, false-interaction grounding, and primitive-level dynamics. We evaluate six open-source ACWMs on more than 18,000 instances across RoboTwin, ManiSkill, and LIBERO. World-SimProbe reveals systematic action-realization degradation across control variation, structured failures in interaction grounding and dynamics, and benchmark signals consistent with human judgments and downstream outcomes. Together, this capability-based framework provides a transparent, and standardized paradigm for diagnosing ACWM simulator fidelity beyond coarse, task-directed evaluation.

Related papers

• IOI: Decoupling Kinematics and Physics for Interactive World Models [46.3330122411516]
  We propose IOI, a hybrid interactive world model integrating kinematics with learned physical dynamics.<n> IOI introduces explicit kinematic guidance, computing kinematics sequences for accurate motion trajectories.<n> experiments on the RoboTwin benchmark validate IOI across kinematic fidelity, out-of-distribution, and policy evaluation.
  arXiv  Detail & Related papers  (2026-06-22T13:09:34Z)
• ReactSim-Bench: Benchmarking Reactive Behavior World Model Simulation in Autonomous Driving [56.96697737229895]
  Reactive capability is a key property of data-driven behavior world model simulators for autonomous driving simulation systems.<n>We introduce ReactSim-Bench for evaluating the reactive capability of behavior world model simulation in autonomous driving.
  arXiv  Detail & Related papers  (2026-06-12T03:11:06Z)
• ManipArena: Comprehensive Real-world Evaluation of Reasoning-Oriented Generalist Robot Manipulation [66.02142169323521]
  Vision-Language-Action models and world models have recently emerged as promising paradigms for general-purpose robotic intelligence.<n>Existing benchmarks are largely simulator-centric, which provide controllability but fail to capture the reality gap caused by perception noise.<n>We introduce ManipArena, a standardized evaluation framework designed to bridge simulation and real-world execution.
  arXiv  Detail & Related papers  (2026-03-30T15:06:41Z)
• ARTIS: Agentic Risk-Aware Test-Time Scaling via Iterative Simulation [72.78362530982109]
  ARTIS, Agentic Risk-Aware Test-Time Scaling via Iterative Simulation, is a framework that decouples exploration from commitment.<n>We show that naive LLM-based simulators struggle to capture rare but high-impact failure modes.<n>We introduce a risk-aware tool simulator that emphasizes fidelity on failure-inducing actions.
  arXiv  Detail & Related papers  (2026-02-02T06:33:22Z)
• REALM: A Real-to-Sim Validated Benchmark for Generalization in Robotic Manipulation [46.970355660265284]
  Vision-Language-Action (VLA) models empower robots to understand and execute tasks described by natural language instructions.<n>A key challenge lies in their ability to generalize beyond the specific environments and conditions they were trained on.<n>We present REALM, a new simulation environment and benchmark designed to evaluate the generalization capabilities of VLA models.
  arXiv  Detail & Related papers  (2025-12-22T16:44:23Z)
• PolaRiS: Scalable Real-to-Sim Evaluations for Generalist Robot Policies [88.78188489161028]
  We introduce Policy Evaluation and Environment Reconstruction in Simulation (PolaRiS)<n>PolaRiS is a scalable real-to-sim framework for high-fidelity simulated robot evaluation.<n>We show that PolaRiS evaluations provide a much stronger correlation to real world generalist policy performance than existing simulated benchmarks.
  arXiv  Detail & Related papers  (2025-12-18T18:49:41Z)
• BeSimulator: A Large Language Model Powered Text-based Behavior Simulator [18.318419980796012]
  We propose BeSimulator as an attempt towards behavior simulation in the context of text-based environments.<n>BeSimulator can generalize across scenarios and achieve long-horizon complex simulation.<n>Our experiments show a significant performance improvement in behavior simulation compared to baselines.
  arXiv  Detail & Related papers  (2024-09-24T08:37:04Z)
• INTAGS: Interactive Agent-Guided Simulation [4.04638613278729]
  In many applications involving multi-agent system (MAS), it is imperative to test an experimental (Exp) autonomous agent in a high-fidelity simulator prior to its deployment to production. We propose a metric to distinguish between real and synthetic multi-agent systems, which is evaluated through the live interaction between the Exp and BG agents. We show that using INTAGS to calibrate the simulator can generate more realistic market data compared to the state-of-the-art conditional Wasserstein Generative Adversarial Network approach.
  arXiv  Detail & Related papers  (2023-09-04T19:56:18Z)
• Nonprehensile Riemannian Motion Predictive Control [57.295751294224765]
  We introduce a novel Real-to-Sim reward analysis technique to reliably imagine and predict the outcome of taking possible actions for a real robotic platform. We produce a closed-loop controller to reactively push objects in a continuous action space. We observe that RMPC is robust in cluttered as well as occluded environments and outperforms the baselines.
  arXiv  Detail & Related papers  (2021-11-15T18:50:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.