V-RAE: Rethinking Video Latent Spaces for Generation

• URL: http://arxiv.org/abs/2608.13556v1
• Date: Thu, 13 Aug 2026 17:59:43 GMT
• Title: V-RAE: Rethinking Video Latent Spaces for Generation
• Abstract summary: V-RAE is a video representation autoencoder that builds compact generative latents on top of frozen vision foundation model representations.<n>A lightweight temporal pooling module removes temporal redundancy while preserving semantic structure.<n>V-RAE achieves 2.13 rFVD on K600, outperforming all evaluated large-scale pretrained video VAEs.
• Score: 30.145790682827514
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Latent video generation relies on autoencoders to define a compact space in which generative models operate. Although video autoencoder architectures have evolved substantially, their latent spaces are still optimized primarily for pixel-level reconstruction and provide limited high-level semantic organization. A reconstruction-optimal latent space, however, need not be well suited to generative modeling. We propose V-RAE, a video representation autoencoder that builds compact generative latents on top of frozen vision foundation model representations. A lightweight temporal pooling module removes temporal redundancy while preserving semantic structure, and a video decoder reconstructs continuous motion from the compressed features. We evaluate V-RAE with four representative frozen encoders on video reconstruction, semantic probing, and class-conditional generation. V-RAE achieves 2.13 rFVD on K600, outperforming all evaluated large-scale pretrained video VAEs. Its latents retain substantially more semantic information than conventional video tokenizer latents. Under matched generation settings, our best variant achieves gFVD scores of 117.86 and 19.16 on UCF101 and K600, respectively, while converging up to 6x faster}. We further show that reconstruction quality alone is insufficient to characterize generative utility and introduce tFVD, a temporal-coherence diagnostic that correlates more reliably with downstream generation quality. Beyond video generation, V-RAE also improves future video prediction on Cityscapes over the Wan 2.2 VAE latent space under matched prediction settings. Taken together, the experiments show that frozen semantic representations can support video reconstruction, generation, and predictive modeling. The project page: https://v-rae.github.io/.

Related papers

• VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoders [14.313320718526414]
  Video Foundation Models (VF-Ms) as VJEPA and VideoMAEv2 show strong video understanding capabilities.<n>VideoRAE is a representation autoencoder that leverages hierarchical features from a frozen video foundation encoder and compresses them with a lightweight 1D self-attention.<n>Experiments show that VideoRAE achieves strong reconstruction in both continuous and discrete regimes.
  arXiv  Detail & Related papers  (2026-07-15T17:59:23Z)
• Video Generation with Predictive Latents [50.3100375593545]
  Video Autoencoder (Eational) enables latent video generative modeling by mapping the visual world into compact latent spaces.<n>How to enhance the diffusability of video latents remains a critical and unresolved challenge.<n>We introduce a simple and effective predictive reconstruction objective that unifies predictive learning with video reconstruction.
  arXiv  Detail & Related papers  (2026-05-04T01:30:04Z)
• SemanticGen: Video Generation in Semantic Space [60.49729308406981]
  State-of-the-art video generative models typically learn the distribution of video latents in the VAE space and map them to pixels using a VAE decoder.<n>We introduce SemanticGen, a novel solution to generate videos in the semantic space.<n>Our method is also effective and computationally efficient when extended to long video generation.
  arXiv  Detail & Related papers  (2025-12-23T18:59:56Z)
• CVD-STORM: Cross-View Video Diffusion with Spatial-Temporal Reconstruction Model for Autonomous Driving [26.379817613036597]
  CVD-STORM is a cross-view video diffusion model utilizing a spatial-temporal reconstruction Variational Autoencoder (VAE)<n>Our approach first fine-tunes the VAE with an auxiliary 4D reconstruction task, enhancing its ability to encode 3D structures and temporal dynamics.<n> Experimental results demonstrate that our model achieves substantial improvements in both FID and FVD metrics.
  arXiv  Detail & Related papers  (2025-10-09T08:41:58Z)
• Hi-VAE: Efficient Video Autoencoding with Global and Detailed Motion [23.80254637449824]
  Hi-VAE formulates an efficient video autoencoding framework that encodes coarse-to-fine motion representations of video dynamics.<n>We show that Hi-VAE exhibits a high compression factor of 1428$times$, almost 30$times$ higher than baseline methods.
  arXiv  Detail & Related papers  (2025-06-08T13:30:11Z)
• FRAME: Pre-Training Video Feature Representations via Anticipation and Memory [55.046881477209695]
  FRAME is a self-supervised video frame encoder tailored for dense video understanding.<n>It learns to predict current and future DINO patch features from past and present RGB frames.<n>It consistently outperforms image encoders and existing self-supervised video models.
  arXiv  Detail & Related papers  (2025-06-05T19:44:47Z)
• LeanVAE: An Ultra-Efficient Reconstruction VAE for Video Diffusion Models [17.29580459404157]
  We propose LeanVAE, a novel and ultra-efficient Video VAE framework.<n>Our model offers up to 50x fewer FLOPs and 44x faster inference speed.<n>Our experiments validate LeanVAE's superiority in video reconstruction and generation.
  arXiv  Detail & Related papers  (2025-03-18T14:58:59Z)
• HiTVideo: Hierarchical Tokenizers for Enhancing Text-to-Video Generation with Autoregressive Large Language Models [63.65066762436074]
  HiTVideo aims to address the potential limitations of existing video tokenizers in text-to-video generation tasks.<n>It utilizes a 3D causal VAE with a multi-layer discrete token framework, encoding video content into hierarchically structured codebooks.
  arXiv  Detail & Related papers  (2025-03-14T15:36:39Z)
• Improved Video VAE for Latent Video Diffusion Model [55.818110540710215]
  Video Autoencoder (VAE) aims to compress pixel data into low-dimensional latent space, playing an important role in OpenAI's Sora. Most of existing VAEs inflate a pretrained image VAE into the 3D causal structure for temporal-spatial compression. We propose a new KTC architecture and a group causal convolution (GCConv) module to further improve video VAE (IV-VAE)
  arXiv  Detail & Related papers  (2024-11-10T12:43:38Z)
• Video Probabilistic Diffusion Models in Projected Latent Space [75.4253202574722]
  We propose a novel generative model for videos, coined projected latent video diffusion models (PVDM) PVDM learns a video distribution in a low-dimensional latent space and thus can be efficiently trained with high-resolution videos under limited resources.
  arXiv  Detail & Related papers  (2023-02-15T14:22:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.