Delving into Latent Spectral Biasing of Video VAEs for Superior Diffusability

• URL: http://arxiv.org/abs/2512.05394v1
• Date: Fri, 05 Dec 2025 03:20:02 GMT
• Title: Delving into Latent Spectral Biasing of Video VAEs for Superior Diffusability
• Abstract summary: We present a statistical analysis of video VAE latent spaces and identify two properties essential for diffusion training.<n>We propose two lightweight, backbone-agnostic regularizers: Local Correlation Regularization and Latent Masked Reconstruction.<n> Experiments show that our Spectral-Structured VAE achieves a $3times$ speedup in text-to-video convergence and a 10% gain in video reward, outperforming strong open-source VAEs.
• Score: 21.927814986478356
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Latent diffusion models pair VAEs with diffusion backbones, and the structure of VAE latents strongly influences the difficulty of diffusion training. However, existing video VAEs typically focus on reconstruction fidelity, overlooking latent structure. We present a statistical analysis of video VAE latent spaces and identify two spectral properties essential for diffusion training: a spatio-temporal frequency spectrum biased toward low frequencies, and a channel-wise eigenspectrum dominated by a few modes. To induce these properties, we propose two lightweight, backbone-agnostic regularizers: Local Correlation Regularization and Latent Masked Reconstruction. Experiments show that our Spectral-Structured VAE (SSVAE) achieves a $3\times$ speedup in text-to-video generation convergence and a 10\% gain in video reward, outperforming strong open-source VAEs. The code is available at https://github.com/zai-org/SSVAE.

Related papers

• Revisiting Spectral Representations in Generative Diffusion Models [98.43552218868837]
  Diffusion models have shown remarkable performance on diverse generation tasks.<n>Recent work finds that imposing representation alignment on the hidden states of diffusion networks can both facilitate training convergence and enhance sampling quality.<n>We investigate the connection between self-supervised spectral representation learning and diffusion generative models through a shared perspective on perturbation kernels.
  arXiv  Detail & Related papers  (2026-09-08T04:54:02Z)
• Learning to Refine: Spectral-Decoupled Iterative Refinement Framework for Precipitation Nowcasting [21.300422393514804]
  We propose a deterministic framework that reformulates nowcasting as progressive frequency-decoupled refinement.<n>SDIR first extracts a stable low-frequency synoptic skeleton, then iteratively refines high-frequency textures under physical constraints.<n> Experiments on three benchmarks show SDIR significantly outperforms SOTA methods in spatial accuracy.
  arXiv  Detail & Related papers  (2026-06-01T07:34:30Z)
• Spectrum Matching: a Unified Perspective for Superior Diffusability in Latent Diffusion [20.731913778080344]
  We show that pixel-space diffusion trained with an MSE objective is inherently biased toward learning low and mid frequencies.<n>Motivated by this result, we propose the emphSpectrum Matching Hypothesis.<n>We show that Spectrum Matching yields superior diffusion generation on CelebA and ImageNet datasets.
  arXiv  Detail & Related papers  (2026-03-15T22:47:43Z)
• Adaptive Spectral Feature Forecasting for Diffusion Sampling Acceleration [58.19554276924402]
  We propose spectral diffusion feature forecaster (Spectrum) to enable global, long-range feature reuse with tightly controlled error.<n>We achieve up to 4.79$times$ speedup on FLUX.1 and 4.67$times$ speedup on Wan2.1-14B, while maintaining much higher sample quality compared with the baselines.
  arXiv  Detail & Related papers  (2026-03-02T08:59:11Z)
• Diffusion-DRF: Differentiable Reward Flow for Video Diffusion Fine-Tuning [72.16213872139748]
  Diffusion-DRF is a differentiable reward flow for fine-tuning video diffusion models.<n>It backpropagates VLM feedback through the diffusion denoising chain.<n>It improves video quality and semantic alignment while mitigating reward hacking and collapse.
  arXiv  Detail & Related papers  (2026-01-07T18:05:08Z)
• Lazy Diffusion: Mitigating spectral collapse in generative diffusion-based stable autoregressive emulation of turbulent flows [0.0]
  We show that standard DDPMs induce a fundamental emphspectral collapse.<n>We introduce power-law schedules that preserve fine-scale structure deeper into diffusion time.<n>These methods are applied to high-Reynolds-number 2D Kolmogorov turbulence and $1/12circ$ Gulf of Mexico ocean reanalysis.
  arXiv  Detail & Related papers  (2025-12-10T12:05:32Z)
• InstantViR: Real-Time Video Inverse Problem Solver with Distilled Diffusion Prior [13.775331675468024]
  We introduce InstantViR, an amortized inference framework for ultra-fast video reconstruction powered by a pre-trained video diffusion prior.<n>We show that InstantViR is compatible with real-time, interactive, editable, streaming scenarios, turning high-quality video restoration into a practical component of modern vision systems.
  arXiv  Detail & Related papers  (2025-11-18T07:40:38Z)
• Frequency Regulation for Exposure Bias Mitigation in Diffusion Models [15.61368627978836]
  We make a key observation: the energy of predicted noisy samples in the reverse process continuously declines compared to perturbed samples in the forward process.<n>We introduce a dynamic frequency regulation mechanism utilizing wavelet transforms, which separately adjusts the low- and high-frequency subbands.<n>We derive the rigorous mathematical form of exposure bias.
  arXiv  Detail & Related papers  (2025-07-14T08:58:38Z)
• FANeRV: Frequency Separation and Augmentation based Neural Representation for Video [32.35716293561769]
  We present a Frequency Separation and Augmentation based Neural Representation for video (FANeRV)<n>FANeRV explicitly separates input frames into high and low-frequency components using discrete wavelet transform.<n>A specially designed gated network effectively fuses these frequency components for optimal reconstruction.
  arXiv  Detail & Related papers  (2025-04-09T10:19:35Z)
• Rethinking Video Tokenization: A Conditioned Diffusion-based Approach [58.164354605550194]
  New tokenizer, Diffusion Conditioned-based Gene Tokenizer, replaces GAN-based decoder with conditional diffusion model.<n>We trained using only a basic MSE diffusion loss for reconstruction, along with KL term and LPIPS perceptual loss from scratch.<n>Even a scaled-down version of CDT (3$times inference speedup) still performs comparably with top baselines.
  arXiv  Detail & Related papers  (2025-03-05T17:59:19Z)
• SNeRV: Spectra-preserving Neural Representation for Video [8.978061470104532]
  We propose spectra-preserving NeRV (SNeRV) as a novel approach to enhance implicit video representations.<n>In this paper, we use 2D discrete wavelet transform (DWT) to decompose video into low-frequency (LF) and high-frequency (HF) features.<n>We demonstrate that SNeRV outperforms existing NeRV models in capturing fine details and achieves enhanced reconstruction.
  arXiv  Detail & Related papers  (2025-01-03T07:57:38Z)
• Time Series Diffusion in the Frequency Domain [54.60573052311487]
  We analyze whether representing time series in the frequency domain is a useful inductive bias for score-based diffusion models. We show that a dual diffusion process occurs in the frequency domain with an important nuance. We show how to adapt the denoising score matching approach to implement diffusion models in the frequency domain.
  arXiv  Detail & Related papers  (2024-02-08T18:59:05Z)
• Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution [65.91317390645163]
  Upscale-A-Video is a text-guided latent diffusion framework for video upscaling. It ensures temporal coherence through two key mechanisms: locally, it integrates temporal layers into U-Net and VAE-Decoder, maintaining consistency within short sequences. It also offers greater flexibility by allowing text prompts to guide texture creation and adjustable noise levels to balance restoration and generation.
  arXiv  Detail & Related papers  (2023-12-11T18:54:52Z)
• APLA: Additional Perturbation for Latent Noise with Adversarial Training Enables Consistency [9.07931905323022]
  We propose a novel text-to-video (T2V) generation network structure based on diffusion models. Our approach only necessitates a single video as input and builds upon pre-trained stable diffusion networks. We leverage a hybrid architecture of transformers and convolutions to compensate for temporal intricacies, enhancing consistency between different frames within the video.
  arXiv  Detail & Related papers  (2023-08-24T07:11:00Z)
• DDS2M: Self-Supervised Denoising Diffusion Spatio-Spectral Model for Hyperspectral Image Restoration [103.79030498369319]
  Self-supervised diffusion model for hyperspectral image restoration is proposed. textttDDS2M enjoys stronger ability to generalization compared to existing diffusion-based methods. Experiments on HSI denoising, noisy HSI completion and super-resolution on a variety of HSIs demonstrate textttDDS2M's superiority over the existing task-specific state-of-the-arts.
  arXiv  Detail & Related papers  (2023-03-12T14:57:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.