Video Generation with Predictive Latents

• URL: http://arxiv.org/abs/2605.02134v1
• Date: Mon, 04 May 2026 01:30:04 GMT
• Title: Video Generation with Predictive Latents
• Abstract summary: Video Autoencoder (Eational) enables latent video generative modeling by mapping the visual world into compact latent spaces.<n>How to enhance the diffusability of video latents remains a critical and unresolved challenge.<n>We introduce a simple and effective predictive reconstruction objective that unifies predictive learning with video reconstruction.
• Score: 50.3100375593545
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video Variational Autoencoder (VAE) enables latent video generative modeling by mapping the visual world into compact spatiotemporal latent spaces, improving training efficiency and stability. While existing video VAEs achieve commendable reconstruction quality, continued optimization of reconstruction does not necessarily translate into improved generative performance. How to enhance the diffusability of video latents remains a critical and unresolved challenge. In this work, inspired by principles of predictive world modeling, we investigate the potential of predictive learning to improve the video generative modeling. To this end, we introduce a simple and effective predictive reconstruction objective that unifies predictive learning with video reconstruction. Specifically, we randomly discard future frames and encode only partial past observations, while training the decoder to reconstruct the observed frames and predict future ones simultaneously. This design encourages the latent space to encode temporally predictive structures and build a more coherent understanding of video dynamics, thereby improving generation quality. Our model, termed Predictive Video VAE (PV-VAE), achieves superior performance on video generation, with 52% faster convergence and a 34.42 FVD improvement over the Wan2.2 VAE on UCF101. Furthermore, comprehensive analyses demonstrate that PV-VAE not only exhibits favorable scalability, with generative performance improving alongside VAE training, but also yields consistent gains in downstream video understanding, underscoring a latent space that effectively captures temporal coherence and motion priors.

Related papers

• V-RAE: Rethinking Video Latent Spaces for Generation [30.145790682827514]
  V-RAE is a video representation autoencoder that builds compact generative latents on top of frozen vision foundation model representations.<n>A lightweight temporal pooling module removes temporal redundancy while preserving semantic structure.<n>V-RAE achieves 2.13 rFVD on K600, outperforming all evaluated large-scale pretrained video VAEs.
  arXiv  Detail & Related papers  (2026-08-13T17:59:43Z)
• What Makes Video World Model Latents Action-Relevant: Prediction over Reconstruction [9.020077150911526]
  We find that action-relevant structure is driven primarily by temporal video pretraining rather than pixel reconstruction fidelity.<n>Our results identify temporal predictive structure -- not reconstruction fidelity -- as the primary ingredient underlying action-relevant video representations.
  arXiv  Detail & Related papers  (2026-06-05T04:43:02Z)
• Learning from Next-Frame Prediction: Autoregressive Video Modeling Encodes Effective Representations [53.91818843831925]
  We propose NExT-Vid, a novel autoregressive visual generative pretraining framework.<n>We introduce a context-isolated autoregressive predictor to decouple semantic representation from target decoding.<n>Through context-isolated flow-matching pretraining, our approach achieves strong representations.
  arXiv  Detail & Related papers  (2025-12-24T07:07:08Z)
• STCDiT: Spatio-Temporally Consistent Diffusion Transformer for High-Quality Video Super-Resolution [60.06664986365803]
  We present STCDiT, a video super-resolution framework built upon a pre-trained video diffusion model.<n>It aims to restore structurally faithful and temporally stable videos from degraded inputs, even under complex camera motions.
  arXiv  Detail & Related papers  (2025-11-24T05:37:23Z)
• CVD-STORM: Cross-View Video Diffusion with Spatial-Temporal Reconstruction Model for Autonomous Driving [26.379817613036597]
  CVD-STORM is a cross-view video diffusion model utilizing a spatial-temporal reconstruction Variational Autoencoder (VAE)<n>Our approach first fine-tunes the VAE with an auxiliary 4D reconstruction task, enhancing its ability to encode 3D structures and temporal dynamics.<n> Experimental results demonstrate that our model achieves substantial improvements in both FID and FVD metrics.
  arXiv  Detail & Related papers  (2025-10-09T08:41:58Z)
• OneVAE: Joint Discrete and Continuous Optimization Helps Discrete Video VAE Train Better [75.24657690640525]
  We show that FSQ could effectively pre-trained continuous VAE priors compared to other quantization methods.<n>We introduce a multi-token quantization mechanism, which achieves nearly a 1 improvement in PSNR dimensions without compromising the token compression ratio.<n>We propose a joint discrete-continuous optimization scheme that unifies the two paradigms.
  arXiv  Detail & Related papers  (2025-08-13T14:49:54Z)
• Generative Pre-trained Autoregressive Diffusion Transformer [74.25668109048418]
  GPDiT is a Generative Pre-trained Autoregressive Diffusion Transformer.<n>It unifies the strengths of diffusion and autoregressive modeling for long-range video synthesis.<n>It autoregressively predicts future latent frames using a diffusion loss, enabling natural modeling of motion dynamics.
  arXiv  Detail & Related papers  (2025-05-12T08:32:39Z)
• LeanVAE: An Ultra-Efficient Reconstruction VAE for Video Diffusion Models [17.29580459404157]
  We propose LeanVAE, a novel and ultra-efficient Video VAE framework.<n>Our model offers up to 50x fewer FLOPs and 44x faster inference speed.<n>Our experiments validate LeanVAE's superiority in video reconstruction and generation.
  arXiv  Detail & Related papers  (2025-03-18T14:58:59Z)
• Autoregressive Video Generation without Vector Quantization [90.87907377618747]
  We reformulate the video generation problem as a non-quantized autoregressive modeling of temporal frame-by-frame prediction.<n>With the proposed approach, we train a novel video autoregressive model without vector quantization, termed NOVA.<n>Our results demonstrate that NOVA surpasses prior autoregressive video models in data efficiency, inference speed, visual fidelity, and video fluency, even with a much smaller model capacity.
  arXiv  Detail & Related papers  (2024-12-18T18:59:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.