H3AE: High Compression, High Speed, and High Quality AutoEncoder for Video Diffusion Models

• URL: http://arxiv.org/abs/2504.10567v1
• Date: Mon, 14 Apr 2025 17:59:06 GMT
• Title: H3AE: High Compression, High Speed, and High Quality AutoEncoder for Video Diffusion Models
• Authors: Yushu Wu, Yanyu Li, Ivan Skorokhodov, Anil Kag, Willi Menapace, Sharath Girish, Aliaksandr Siarohin, Yanzhi Wang, Sergey Tulyakov,
• Abstract summary: Autoencoder (AE) is the key to the success of latent diffusion models for image and video generation.<n>In this work, we examine the architecture design choices and optimize the computation distribution to obtain efficient and high-compression video AEs.<n>Our AE achieves an ultra-high compression ratio and real-time decoding speed on mobile while outperforming prior art in terms of reconstruction metrics.
• Score: 76.1519545010611
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Autoencoder (AE) is the key to the success of latent diffusion models for image and video generation, reducing the denoising resolution and improving efficiency. However, the power of AE has long been underexplored in terms of network design, compression ratio, and training strategy. In this work, we systematically examine the architecture design choices and optimize the computation distribution to obtain a series of efficient and high-compression video AEs that can decode in real time on mobile devices. We also unify the design of plain Autoencoder and image-conditioned I2V VAE, achieving multifunctionality in a single network. In addition, we find that the widely adopted discriminative losses, i.e., GAN, LPIPS, and DWT losses, provide no significant improvements when training AEs at scale. We propose a novel latent consistency loss that does not require complicated discriminator design or hyperparameter tuning, but provides stable improvements in reconstruction quality. Our AE achieves an ultra-high compression ratio and real-time decoding speed on mobile while outperforming prior art in terms of reconstruction metrics by a large margin. We finally validate our AE by training a DiT on its latent space and demonstrate fast, high-quality text-to-video generation capability.

Related papers

• LeanVAE: An Ultra-Efficient Reconstruction VAE for Video Diffusion Models [17.29580459404157]
  We propose LeanVAE, a novel and ultra-efficient Video VAE framework. Our model offers up to 50x fewer FLOPs and 44x faster inference speed. Our experiments validate LeanVAE's superiority in video reconstruction and generation.
  arXiv  Detail & Related papers  (2025-03-18T14:58:59Z)
• Pathology Image Compression with Pre-trained Autoencoders [52.208181380986524]
  Whole Slide Images in digital histopathology pose significant storage, transmission, and computational efficiency challenges.<n>Standard compression methods, such as JPEG, reduce file sizes but fail to preserve fine-grained phenotypic details critical for downstream tasks.<n>In this work, we repurpose autoencoders (AEs) designed for Latent Diffusion Models as an efficient learned compression framework for pathology images.
  arXiv  Detail & Related papers  (2025-03-14T17:01:17Z)
• REGEN: Learning Compact Video Embedding with (Re-)Generative Decoder [52.698595889988766]
  We present a novel perspective on learning video embedders for generative modeling.<n>Rather than requiring an exact reproduction of an input video, an effective embedder should focus on visually plausible reconstructions.<n>We propose replacing the conventional encoder-decoder video embedder with an encoder-generator framework.
  arXiv  Detail & Related papers  (2025-03-11T17:51:07Z)
• Rethinking Video Tokenization: A Conditioned Diffusion-based Approach [58.164354605550194]
  New tokenizer, Diffusion Conditioned-based Gene Tokenizer, replaces GAN-based decoder with conditional diffusion model.<n>We trained using only a basic MSE diffusion loss for reconstruction, along with KL term and LPIPS perceptual loss from scratch.<n>Even a scaled-down version of CDT (3$times inference speedup) still performs comparably with top baselines.
  arXiv  Detail & Related papers  (2025-03-05T17:59:19Z)
• Large Motion Video Autoencoding with Cross-modal Video VAE [52.13379965800485]
  Video Variational Autoencoder (VAE) is essential for reducing video redundancy and facilitating efficient video generation.<n>Existing Video VAEs have begun to address temporal compression; however, they often suffer from inadequate reconstruction performance.<n>We present a novel and powerful video autoencoder capable of high-fidelity video encoding.
  arXiv  Detail & Related papers  (2024-12-23T18:58:24Z)
• Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models [38.84567900296605]
  Deep Compression Autoencoder (DC-AE) is a new family of autoencoder models for accelerating high-resolution diffusion models.<n>Applying our DC-AE to latent diffusion models, we achieve significant speedup without accuracy drop.
  arXiv  Detail & Related papers  (2024-10-14T17:15:07Z)
• Data-Model-Circuit Tri-Design for Ultra-Light Video Intelligence on Edge Devices [90.30316433184414]
  We propose a data-model-hardware tri-design framework for high- throughput, low-cost, and high-accuracy MOT on HD video stream. Compared to the state-of-the-art MOT baseline, our tri-design approach can achieve 12.5x latency reduction, 20.9x effective frame rate improvement, 5.83x lower power, and 9.78x better energy efficiency, without much accuracy drop.
  arXiv  Detail & Related papers  (2022-10-16T16:21:40Z)
• A Unified End-to-End Framework for Efficient Deep Image Compression [35.156677716140635]
  We propose a unified framework called Efficient Deep Image Compression (EDIC) based on three new technologies. Specifically, we design an auto-encoder style network for learning based image compression. Our EDIC method can also be readily incorporated with the Deep Video Compression (DVC) framework to further improve the video compression performance.
  arXiv  Detail & Related papers  (2020-02-09T14:21:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.