What Makes Video World Model Latents Action-Relevant: Prediction over Reconstruction

• URL: http://arxiv.org/abs/2606.07687v1
• Date: Fri, 05 Jun 2026 04:43:02 GMT
• Title: What Makes Video World Model Latents Action-Relevant: Prediction over Reconstruction
• Abstract summary: We find that action-relevant structure is driven primarily by temporal video pretraining rather than pixel reconstruction fidelity.<n>Our results identify temporal predictive structure -- not reconstruction fidelity -- as the primary ingredient underlying action-relevant video representations.
• Score: 9.020077150911526
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video world models are increasingly used to provide predictive visual representations, yet it remains unclear which pretraining signals induce action-relevant structure in their latent spaces. We study this question through a unified probe-based evaluation across diverse encoder families, including image-only self-supervision, video pretraining with and without latent prediction, reconstruction-based autoencoders, diffusion models, and shortcut-forcing dynamics models. Using a common inverse-dynamics probing objective, we find that action-relevant structure is driven primarily by temporal video pretraining rather than pixel reconstruction fidelity: models with strong pixel decoding quality can exhibit near-zero action recoverability, while video-pretrained self-supervised encoders consistently achieve the best Pareto trade-off between visual fidelity and action prediction. Comparing V-JEPA and VideoMAE further shows that most gains arise from natural-video temporal context, with feature-level latent prediction providing a smaller additional benefit. These trends transfer across robotic benchmarks, though CALVIN reveals that static-environment tasks can partially mask the importance of temporal structure by allowing strong image priors to suffice. Finally, inverse-dynamics supervision substantially improves robustness to visual corruption, suggesting that action-aware objectives regularize latent geometry beyond clean-setting performance. Our results identify temporal predictive structure -- not reconstruction fidelity -- as the primary ingredient underlying action-relevant video representations.

Related papers

• Self-Supervised Learning of Structured Dynamics from Videos [76.23834498074805]
  We study whether structured motion representations can be recovered from frozen features of a pretrained image vision transformer.<n>We propose the Structured Dynamics Model (SDM), which explicitly separates the dominant source of temporal change from residual dynamics.<n>We evaluate SDM on ProbeMotion, a new evaluation suite spanning synthetic and real videos with camera motion, object motion, and combined dynamics.
  arXiv  Detail & Related papers  (2026-07-23T17:55:07Z)
• Native Video-Action Pretraining for Generalizable Robot Control [77.59831077077176]
  LingBot-VA 2.0 is a video-action foundation model built from the ground up for embodiment.<n>To meet the demands of high-frequency inference, our model employs a sparse MoE backbone.<n>Real-world deployment validates LingBot-VA 2.0 as a robust foundation model.
  arXiv  Detail & Related papers  (2026-07-09T16:15:43Z)
• Understanding and Mitigating the Video-Action Generalization Gap via Temporal Ratio [42.378085641934796]
  Generative video foundation models exhibit strong compositional priors.<n>World-action models (WAMs) and video-action models (VAMs) often lose these priors after finetuning on robotic action data.<n>We propose an inference-time adaptive guidance method, which exploits this intrinsic feature attention pattern.
  arXiv  Detail & Related papers  (2026-07-09T05:56:08Z)
• Momentum-Guided Semantic Forecasting (MoFore) for Self-Supervised Video Representation Learning [0.0]
  We introduce a Momentum-Guided Semantic Forecasting framework for self-supervised video representation learning.<n>The framework learns temporally consistent and semantically meaningful video representations without using action labels during training.
  arXiv  Detail & Related papers  (2026-06-08T17:50:01Z)
• Video Generation with Predictive Latents [50.3100375593545]
  Video Autoencoder (Eational) enables latent video generative modeling by mapping the visual world into compact latent spaces.<n>How to enhance the diffusability of video latents remains a critical and unresolved challenge.<n>We introduce a simple and effective predictive reconstruction objective that unifies predictive learning with video reconstruction.
  arXiv  Detail & Related papers  (2026-05-04T01:30:04Z)
• Representations Before Pixels: Semantics-Guided Hierarchical Video Prediction [12.661995278049764]
  We present Re2Pix, a hierarchical video prediction framework.<n>It decomposes forecasting into two stages: semantic representation prediction and representation-guided visual synthesis.<n>A key challenge arises from the train-test mismatch between ground-truth representations available during training and predicted ones used at inference.
  arXiv  Detail & Related papers  (2026-04-13T16:42:46Z)
• Chain of World: World Model Thinking in Latent Motion [24.24061036481793]
  Vision-Language-Action (VLA) models often overlook the predictive and temporal-causal structure underlying visual dynamics.<n>We introduce CoWVLA, a new "Chain of World" paradigm that unifies world-model temporal reasoning with a disentangled latent motion representation.<n>CoWVLA outperforms existing world-model and latent-action approaches and achieves moderate computational efficiency.
  arXiv  Detail & Related papers  (2026-03-03T17:52:06Z)
• Learning from Next-Frame Prediction: Autoregressive Video Modeling Encodes Effective Representations [53.91818843831925]
  We propose NExT-Vid, a novel autoregressive visual generative pretraining framework.<n>We introduce a context-isolated autoregressive predictor to decouple semantic representation from target decoding.<n>Through context-isolated flow-matching pretraining, our approach achieves strong representations.
  arXiv  Detail & Related papers  (2025-12-24T07:07:08Z)
• Generative Pre-trained Autoregressive Diffusion Transformer [74.25668109048418]
  GPDiT is a Generative Pre-trained Autoregressive Diffusion Transformer.<n>It unifies the strengths of diffusion and autoregressive modeling for long-range video synthesis.<n>It autoregressively predicts future latent frames using a diffusion loss, enabling natural modeling of motion dynamics.
  arXiv  Detail & Related papers  (2025-05-12T08:32:39Z)
• VNVC: A Versatile Neural Video Coding Framework for Efficient Human-Machine Vision [59.632286735304156]
  It is more efficient to enhance/analyze the coded representations directly without decoding them into pixels. We propose a versatile neural video coding (VNVC) framework, which targets learning compact representations to support both reconstruction and direct enhancement/analysis.
  arXiv  Detail & Related papers  (2023-06-19T03:04:57Z)
• A new way of video compression via forward-referencing using deep learning [0.0]
  This paper explores a new way of video coding by modelling human pose from the already-encoded frames. It is expected that the proposed approach can overcome the limitations of the traditional backward-referencing frames. Experimental results show that the proposed approach can achieve on average up to 2.83 dB PSNR gain and 25.93% residual savings for high motion video sequences.
  arXiv  Detail & Related papers  (2022-08-13T16:19:11Z)
• Local Frequency Domain Transformer Networks for Video Prediction [24.126513851779936]
  Video prediction is of interest not only in anticipating visual changes in the real world but has, above all, emerged as an unsupervised learning rule. This paper proposes a fully differentiable building block that can perform all of those tasks separately while maintaining interpretability.
  arXiv  Detail & Related papers  (2021-05-10T19:48:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.