CRONOS: Benchmarking Counterfactual Physical Consistency in Video Models

• URL: http://arxiv.org/abs/2605.23699v1
• Date: Fri, 22 May 2026 14:51:22 GMT
• Title: CRONOS: Benchmarking Counterfactual Physical Consistency in Video Models
• Abstract summary: CRONOS is an intervention-based benchmark designed to evaluate counterfactual physical consistency.<n>Built in an Unreal Engine environment, CRONOS enables controlled, high-fidelity generation of videos across diverse scenes and dynamics.
• Score: 25.976578205889776
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video prediction is increasingly viewed as a path toward generalizable world models, yet it remains unclear whether these systems learn underlying causal structure or merely exploit superficial visual correlations for future prediction. We introduce CRONOS, an intervention-based benchmark designed to evaluate counterfactual physical consistency: whether a model's predictions of physical events respond appropriately to controlled changes in the visual input, such as variations of scene context, viewpoint, object appearance, and object category. Built in a photorealistic Unreal Engine environment, CRONOS enables controlled, high-fidelity generation of videos across diverse scenes and dynamics. In contrast to previous benchmarks, CRONOS systematically intervenes on four key factors - viewpoint, scene, object category, and object appearance - while keeping the underlying physical event type, such as a collision, occlusion, or fall, fixed. Our evaluation of recent open-source video generators reveals substantial failures in counterfactual physical consistency: prediction quality for the same physical event type is affected by appearance, environment, and, particularly by viewpoint changes. CRONOS provides a controlled and reproducible testbed for diagnosing how the quality of generated videos changes for different interventions, establishing a concrete target for developing models that perform consistently across changes of multiple conditions. The dataset and code are available at our project page.

Related papers

• CAER: Causal Action Effect Reweighting for World Model Training [31.737118584204342]
  We introduce Causal Action Effect Reweighting (CAER), a training paradigm that redistributes supervision toward the tokens whose predicted future is causally affected by the action.<n>CAER contrasts the model's own predictions with and without action conditioning to localize these tokens online, then normalizes the resulting effect map into a weight that preserves the total coefficient mass and changes only where it is spent.<n> Experiments across heterogeneous action-conditioned world-model tasks show that CAER converges to better solutions than uniform MSE training, with consistent improvements in the physical consistency, controllability, and visual quality of generated videos
  arXiv  Detail & Related papers  (2026-08-31T14:49:56Z)
• DyG$^2$T: Modeling Object Dynamics with 3D Gaussian Temporal-Spatial Particle Graph Transformer [86.4618122245946]
  DyG$2$T is a dynamics modeling framework that infers object motion trajectories by spatially completing and temporally discriminating Key Point representations.<n>Experiments on both synthetic and real-world datasets demonstrate that DyG$2$T achieves accurate dynamics modeling and reasoning.
  arXiv  Detail & Related papers  (2026-08-19T03:44:28Z)
• Event-Conditioned Diagnostics of Kinematic, Contact, and Object-Permanence Fields in Passive Object-State World Models [6.556150313826966]
  We study event-conditioned latent physical structure in passive object-state world models.<n> Protocol tests whether hidden representations encode event-regime information.<n>Results support event-conditioned organization and fixed-horizon functional sensitivity of latent physical fields.
  arXiv  Detail & Related papers  (2026-06-26T10:11:59Z)
• MemoBench: Benchmarking World Modeling in Dynamically Changing Environments [72.23517478870605]
  We introduce MemoBench, a diagnostic benchmark built around the disappear-and-reappear paradigm in dynamically changing environments.<n>We curate 360 ground-truth clips spanning synthetic and real-world scenes, and design an evaluation suite combining automated metrics with VQA-based assessment.
  arXiv  Detail & Related papers  (2026-06-25T20:37:39Z)
• Trimming the Long-Tail of Visual World Modeling Evaluation [67.17191772079316]
  We introduce Tailor-Bench, a benchmark that challenges world models to simulate irregular physical interactions.<n>Regular scenarios reflect common tool-task pairs, Unconventional scenarios replace conventional tools with attribute-compatible substitutes, and Impossible scenarios introduce attribute-violating tools.<n>Our experimental results reveal a clear long-tail gap in physical world modeling, indicating limited generalization beyond common interactions.
  arXiv  Detail & Related papers  (2026-06-23T07:43:41Z)
• PhyCo: Learning Controllable Physical Priors for Generative Motion [55.59209981836171]
  We present PhyCo, a framework that introduces continuous, interpretable, and physically grounded control into video generation.<n>Our approach integrates three key components: (i) a large-scale dataset of over 100K photorealistic simulation videos where friction, restitution, deformation, and force are systematically varied across diverse scenarios; (ii) physics-supervised fine-tuning of a pretrained diffusion model using a ControlNet conditioned on pixel-aligned physical property maps; and (iii) VLM-guided reward optimization, where a fine-tuned vision-language model evaluates generated videos with targeted physics queries and provides differentiable feedback.
  arXiv  Detail & Related papers  (2026-04-30T17:53:03Z)
• GenHOI: Towards Object-Consistent Hand-Object Interaction with Temporally Balanced and Spatially Selective Object Injection [54.879037588415656]
  GenHOI is a lightweight augmentation to pretrained video generation models.<n>It injects reference-object information in a temporally balanced and spatially selective manner.<n>GenHOI significantly outperforms state-of-the-art HOI reenactment and all-in-one video editing methods.
  arXiv  Detail & Related papers  (2026-03-06T09:01:09Z)
• Counterfactual World Models via Digital Twin-conditioned Video Diffusion [7.761903219193172]
  We introduce CWMDT, a framework to turn standard video diffusion models into effective counterfactual world models.<n>CWMDT constructs digital twins of observed scenes to explicitly encode objects and their relationships, represented as structured text.<n>We show that CWMDT achieves state-of-the-art performance, suggesting that alternative representations of videos, such as the digital twins considered here, offer powerful control signals for video forward simulation-based world models.
  arXiv  Detail & Related papers  (2025-11-21T18:37:23Z)
• On the Benefits of Instance Decomposition in Video Prediction Models [5.653106385738823]
  State-of-the-art video prediction methods typically model the dynamics of a scene jointly and implicitly, without any explicit decomposition into separate objects.<n>This is challenging and potentially sub-optimal, as every object in a dynamic scene has their own pattern of movement, typically somewhat independent of others.<n>In this paper, we investigate the benefit of explicitly modeling the objects in a dynamic scene separately within the context of latent-transformer video prediction models.
  arXiv  Detail & Related papers  (2025-01-17T21:36:06Z)
• BEHAVIOR Vision Suite: Customizable Dataset Generation via Simulation [57.40024206484446]
  We introduce the BEHAVIOR Vision Suite (BVS), a set of tools and assets to generate fully customized synthetic data for systematic evaluation of computer vision models. BVS supports a large number of adjustable parameters at the scene level. We showcase three example application scenarios.
  arXiv  Detail & Related papers  (2024-05-15T17:57:56Z)
• A Control-Centric Benchmark for Video Prediction [69.22614362800692]
  We propose a benchmark for action-conditioned video prediction in the form of a control benchmark. Our benchmark includes simulated environments with 11 task categories and 310 task instance definitions. We then leverage our benchmark to study the effects of scaling model size, quantity of training data, and model ensembling.
  arXiv  Detail & Related papers  (2023-04-26T17:59:45Z)
• STDepthFormer: Predicting Spatio-temporal Depth from Video with a Self-supervised Transformer Model [0.0]
  Self-supervised model simultaneously predicts a sequence of future frames from video-input with a spatial-temporal attention network is proposed. The proposed model leverages prior scene knowledge such as object shape and texture similar to single-image depth inference methods. It is implicitly capable of forecasting the motion of objects in the scene, rather than requiring complex models involving multi-object detection, segmentation and tracking.
  arXiv  Detail & Related papers  (2023-03-02T12:22:51Z)
• COPILOT: Human-Environment Collision Prediction and Localization from Egocentric Videos [62.34712951567793]
  The ability to forecast human-environment collisions from egocentric observations is vital to enable collision avoidance in applications such as VR, AR, and wearable assistive robotics. We introduce the challenging problem of predicting collisions in diverse environments from multi-view egocentric videos captured from body-mounted cameras. We propose a transformer-based model called COPILOT to perform collision prediction and localization simultaneously.
  arXiv  Detail & Related papers  (2022-10-04T17:49:23Z)
• Causal Discovery in Physical Systems from Videos [123.79211190669821]
  Causal discovery is at the core of human cognition. We consider the task of causal discovery from videos in an end-to-end fashion without supervision on the ground-truth graph structure.
  arXiv  Detail & Related papers  (2020-07-01T17:29:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.