Autonomous Video Generation with Counterfactual Controllability for Self-Evolving World Models

• URL: http://arxiv.org/abs/2606.24152v1
• Date: Tue, 23 Jun 2026 05:19:47 GMT
• Title: Autonomous Video Generation with Counterfactual Controllability for Self-Evolving World Models
• Abstract summary: Existing literature claims that video generation essentially is world modelling.<n>In this paper we present a new perspective, i.e., autonomous video generation with counterfactual controllability.
• Score: 40.57509385724164
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing literature claims that video generation essentially is world modelling. On the one hand, the claim is productive because it pushes generative AI beyond static images and toward temporally extended physical scenes. On the other hand, this claim dangerously relies on the belief that scaling visual prediction alone will automatically yield physical agents. We prefer a more accurate statement: video generation models learn a partial, implicit spatiotemporal world model, but not a fully grounded or controllable one. The reason is as follows: a model may generate a plausible video of a drone crossing a forest or a robot arm manipulating a cup, yet still fail to know which variables are controllable, which constraints belong to a particular body and which futures remain valid under intervention. The frontier in essence is not predictive realism alone, instead it emphasizes a self-evolving generative nature that requires the decisive criterion to be counterfactual controllability: the capability of asking what would happen under an action, to test whether the generated future can survive embodiment constraints and to feed the resulting action knowledge back into future imagination (generation). Therefore, in this paper we present a new perspective, i.e., autonomous video generation with counterfactual controllability is one promising way to realize self-evolving world models.

Related papers

• Thinking in Video: Can Video Generators Really Reason About the Real World? [48.57950330221376]
  Thinking in Video is where video is not merely an output artifact but a medium for constructing, extending, and verifying causal thought.<n>We introduce the Causal-Generative Dual-Judge (DJCG), auditing World Consistency Model from two perspectives.<n>We show that open-source models produce plausible dynamics despite near-zero explicit causal perception, whereas advanced closed-source systems show stronger but still limited alignment between reasoning and generation.
  arXiv  Detail & Related papers  (2026-07-20T03:56:43Z)
• World Action Models: A Survey [100.95337034529263]
  World Action Models (WAMs) are embodied predictive-action models that make a forecast of the future available to action.<n>Recent WAMs repurpose large video generation models, and a parallel line relies on language or vision-language backbones without a video-generation core.<n>This survey gives the field a common account. It first clarifies these boundaries, then organizes existing works through two complementary views.
  arXiv  Detail & Related papers  (2026-06-18T17:05:19Z)
• From Imagined Futures to Executable Actions: Mixture of Latent Actions for Robot Manipulation [88.39072412680633]
  We propose MoLA, a control-oriented interface that transforms imagined future videos into executable representations.<n>We evaluate our approach on simulated benchmarks and real-world robot manipulation tasks.
  arXiv  Detail & Related papers  (2026-05-12T14:15:16Z)
• Say, Dream, and Act: Learning Video World Models for Instruction-Driven Robot Manipulation [31.881404705268356]
  We propose a framework for fast and predictive video-conditioned action.<n>Our approach first selects and adapts a robust video generation model to ensure reliable future predictions.<n>Our method produces temporally coherent, spatially accurate video predictions that directly support precise manipulation.
  arXiv  Detail & Related papers  (2026-02-11T10:23:52Z)
• From Generative Engines to Actionable Simulators: The Imperative of Physical Grounding in World Models [4.52033729546524]
  A world model is an AI system that simulates how an environment evolves under actions.<n>Current world models suffer from visual conflation: the mistaken assumption that high-fidelity video generation implies an understanding of physical and causal dynamics.<n>We show that while modern models excel at predicting pixels, they frequently violate invariant constraints, fail under intervention, and break down in safety-critical decision-making.
  arXiv  Detail & Related papers  (2026-01-21T23:35:33Z)
• Walk through Paintings: Egocentric World Models from Internet Priors [65.30611174953958]
  We present the Egocentric World Model (EgoWM), a simple, architecture-agnostic method that transforms any pretrained video diffusion model into an action-conditioned world model.<n>Rather than training from scratch, we repurpose the rich world priors of Internet-scale video models and inject motor commands through lightweight conditioning layers.<n>Our approach scales naturally across embodiments and action spaces, ranging from 3-DoF mobile robots to 25-DoF humanoids.
  arXiv  Detail & Related papers  (2026-01-21T18:59:32Z)
• From Generated Human Videos to Physically Plausible Robot Trajectories [103.28274349461607]
  Video generation models are rapidly improving in their ability to synthesize human actions in novel contexts.<n>To realize this potential, how can a humanoid execute the human actions from generated videos in a zero-shot manner?<n>This challenge arises because generated videos are often noisy and exhibit morphological distortions that make direct imitation difficult compared to real video.<n>We propose GenMimic, a physics-aware reinforcement learning policy conditioned on 3D keypoints, and trained with symmetry regularization and keypoint-weighted tracking rewards.
  arXiv  Detail & Related papers  (2025-12-04T18:56:03Z)
• ProphetDWM: A Driving World Model for Rolling Out Future Actions and Videos [13.630119246378518]
  We argue that driving world models should have two additional abilities: action control and action prediction.<n>We propose ProphetDWM, a novel end-to-end driving world model that jointly predicts future videos and actions.
  arXiv  Detail & Related papers  (2025-05-24T11:35:09Z)
• Future Frame Prediction for Robot-assisted Surgery [57.18185972461453]
  We propose a ternary prior guided variational autoencoder (TPG-VAE) model for future frame prediction in robotic surgical video sequences. Besides content distribution, our model learns motion distribution, which is novel to handle the small movements of surgical tools.
  arXiv  Detail & Related papers  (2021-03-18T15:12:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.