What-If World: A Causal Benchmark for General World Models in Embodied Scenarios

• URL: http://arxiv.org/abs/2605.27589v1
• Date: Tue, 26 May 2026 19:02:26 GMT
• Title: What-If World: A Causal Benchmark for General World Models in Embodied Scenarios
• Abstract summary: Video generation models are increasingly used as world simulators for tasks like driving and robotic manipulation.<n>What matters in these settings is not whether a single video looks right, but whether the model's output changes when its input changes.<n>We test this by giving a model two prompts describing the same scene with one physical detail varied, and checking whether the two videos diverge the way physics predicts.
• Score: 23.527918480081013
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video generation models are increasingly used as world simulators for tasks like driving and robotic manipulation. What matters in these settings is not whether a single video looks right, but whether the model's output changes when its input changes. We test this by giving a model two prompts describing the same scene with one physical detail varied, and checking whether the two videos diverge the way physics predicts. The wording difference between the prompts is small by design, since only one variable is changed, but the correct physical difference is not. A model that misses this can still produce two videos that each look plausible individually, and existing benchmarks score videos one at a time and cannot detect this failure. We introduce What-If World, 319 such prompt pairs built on real frames from nuScenes and DROID, organized by a taxonomy of six physical variables shared across driving and manipulation. Each pair is scored with APEO, a four-part rubric checking whether each video follows its prompt (Adherence), is physically consistent (Physics), preserves the shared scene (Environment), and ends in the correct difference (Outcome). Across nine state-of-the-art models, no system exceeds 52% on the paired score, and open-source models cluster near 28%. Every model tested fails on a large fraction of causal interventions, indicating substantial room before these models can reliably support action-conditioned simulation or model-based planning. Where models do score well, performance appears to track the visual prominence of the intervention rather than the tractability of its underlying physics. Some visually subtle interventions score as low as 14.2%, while visually pronounced ones reach 40.4%.

Related papers

• TEMPO: Learning Temporal Context for Dynamic Robot Manipulation [12.057683796484232]
  We propose TEMPO, which augments a pretrained VLA with two temporal inputs to resolve motion ambiguity and state aliasing.<n>It improves Bottle Handover success from 44% to 74% and is the only method that solves state aliasing.<n>We further release TEMPO-Bench, a benchmark of over 50k annotated frames for evaluating motion-aware robot perception.
  arXiv  Detail & Related papers  (2026-09-15T08:56:52Z)
• MotionBlind: Probing the Illusion of Motion Understanding in Video-LLMs [4.006324106431453]
  Video large language models (Video-LLMs) are increasingly used as the perceptual front end of world models.<n>We introduce MotionBlind, a contrastive benchmark of self-recorded video for physically grounded motion.<n>A Video-LLM can watch two clips of the same person in the same room, name every object in both, and still fail to say which clip moves faster.
  arXiv  Detail & Related papers  (2026-09-08T23:15:20Z)
• Dancing Stick Figures: An Introductory Dataset for Training Video Generation Models [0.0]
  Dancing Stick Figures is a synthetic video dataset built against three obstacles.<n>For speed, its 64x64, 64-frame reference task is sized for practical repeated training on a single workstation GPU.<n>For accessibility, the release is a 0.79-GB training tier of 4,020 video clips--1,340 six-second source motions.
  arXiv  Detail & Related papers  (2026-08-29T08:03:19Z)
• RigidBench: Evaluating Rigid-Body Physics in Video Generation Models [16.53417077740312]
  We introduce a simulator-grounded benchmark that compares a generated continuation with a reference rollout from the same initial frame and motion description.<n>We evaluate eight models on the same 100 examples with ten measurements that keep these aspects separate.<n>Full fine-tuning reduces 3D trajectory error by about 20% with almost no change in SSIM.
  arXiv  Detail & Related papers  (2026-08-16T06:05:20Z)
• Apple-$π$: Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence [54.58886283991723]
  Apple-PI is the first benchmark that anchors video-model evaluation explicitly in physical laws.<n>Apple-PI comprises three components.<n> Benchmarking 11 models shows that current video models remain far from reliable law-grounded world simulators.
  arXiv  Detail & Related papers  (2026-07-17T18:00:05Z)
• WBench: A Comprehensive Multi-turn Benchmark for Interactive Video World Model Evaluation [56.5415838759151]
  WBench is a comprehensive benchmark for evaluation of interactive world models.<n>It covers video quality, setting adherence, interaction adherence, consistency, and physics compliance.<n>It contains 289 test cases and 1,058 interaction turns, covering diverse scenes, styles, subjects, and both first- and third-person perspectives.
  arXiv  Detail & Related papers  (2026-05-25T14:01:31Z)
• World Action Models are Zero-shot Policies [111.91938055103633]
  We introduce DreamZero, a World Action Model (WAM) built upon a pretrained video diffusion backbone.<n>By jointly modeling video and action, DreamZero learns diverse skills effectively from heterogeneous robot data.<n>We demonstrate two forms of cross-embodiment transfer: video-only demonstrations from other robots or humans yield a relative improvement of over 42% on unseen task performance.
  arXiv  Detail & Related papers  (2026-02-17T15:04:02Z)
• MAD: Motion Appearance Decoupling for efficient Driving World Models [94.40548866741791]
  We propose an efficient adaptation framework that converts generalist video models into controllable driving world models.<n>Key idea is to decouple motion learning from appearance synthesis.<n>Scaling to LTX, our MAD-LTX model outperforms all open-source competitors.
  arXiv  Detail & Related papers  (2026-01-14T12:52:23Z)
• Motion Dreamer: Boundary Conditional Motion Reasoning for Physically Coherent Video Generation [27.690736225683825]
  We introduce Motion Dreamer, a two-stage framework that explicitly separates motion reasoning from visual synthesis.<n>Our approach introduces instance flow, a sparse-to-dense motion representation enabling effective integration of partial user-defined motions.<n>Experiments demonstrate that Motion Dreamer significantly outperforms existing methods, achieving superior motion plausibility and visual realism.
  arXiv  Detail & Related papers  (2024-11-30T17:40:49Z)
• ActionAtlas: A VideoQA Benchmark for Domain-specialized Action Recognition [111.32822459456793]
  ActionAtlas is a video question answering benchmark featuring short videos across various sports. The dataset includes 934 videos showcasing 580 unique actions across 56 sports, with a total of 1896 actions within choices. We evaluate open and proprietary foundation models on this benchmark, finding that the best model, GPT-4o, achieves a maximum accuracy of 45.52%.
  arXiv  Detail & Related papers  (2024-10-08T07:55:09Z)
• Perception Test: A Diagnostic Benchmark for Multimodal Video Models [78.64546291816117]
  We propose a novel multimodal video benchmark to evaluate the perception and reasoning skills of pre-trained multimodal models. The Perception Test focuses on skills (Memory, Abstraction, Physics, Semantics) and types of reasoning (descriptive, explanatory, predictive, counterfactual) across video, audio, and text modalities. The benchmark probes pre-trained models for their transfer capabilities, in a zero-shot / few-shot or limited finetuning regime.
  arXiv  Detail & Related papers  (2023-05-23T07:54:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.