DA-VAE: Plug-in Latent Compression for Diffusion via Detail Alignment

• URL: http://arxiv.org/abs/2603.22125v1
• Date: Mon, 23 Mar 2026 15:51:03 GMT
• Title: DA-VAE: Plug-in Latent Compression for Diffusion via Detail Alignment
• Abstract summary: We propose textbfDetail-textbfAligned VAE, which increases the compression ratio of a pretrained VAE with only lightweight adaptation of the pretrained diffusion backbone.<n>With a warm-start fine-tuning strategy, our method enables $1024 times 1024$ image generation with Stable Diffusion 3.5 using only $32 times 32$ tokens, $4times$ fewer than the original model.
• Score: 26.049348476631206
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reducing token count is crucial for efficient training and inference of latent diffusion models, especially at high resolution. A common strategy is to build high-compression image tokenizers with more channels per token. However, when trained only for reconstruction, high-dimensional latent spaces often lose meaningful structure, making diffusion training harder. Existing methods address this with extra objectives such as semantic alignment or selective dropout, but usually require costly diffusion retraining. Pretrained diffusion models, however, already exhibit a structured, lower-dimensional latent space; thus, a simpler idea is to expand the latent dimensionality while preserving this structure. We therefore propose \textbf{D}etail-\textbf{A}ligned VAE, which increases the compression ratio of a pretrained VAE with only lightweight adaptation of the pretrained diffusion backbone. DA-VAE uses an explicit latent layout: the first $C$ channels come directly from the pretrained VAE at a base resolution, while an additional $D$ channels encode higher-resolution details. A simple detail-alignment mechanism encourages the expanded latent space to retain the structure of the original one. With a warm-start fine-tuning strategy, our method enables $1024 \times 1024$ image generation with Stable Diffusion 3.5 using only $32 \times 32$ tokens, $4\times$ fewer than the original model, within 5 H100-days. It further unlocks $2048 \times 2048$ generation with SD3.5, achieving a $6\times$ speedup while preserving image quality. We also validate the method and its design choices quantitatively on ImageNet.

Related papers

• RiT: Vanilla Diffusion Transformers Suffice in Representation Space [12.711808725422108]
  Flow matching with $x$prediction is known to exploit low-dimensional manifold structure effectively in pixel space citeli2025back.<n>We ask whether a pretrained representation space, while containing a low-dimensional data manifold of comparable intrinsic dimensionality, offers a distribution more favorable for flow-matching learning.
  arXiv  Detail & Related papers  (2026-05-21T04:21:43Z)
• DiT-IC: Aligned Diffusion Transformer for Efficient Image Compression [56.68267606568949]
  DiT-IC is an Aligned Diffusion Transformer for Image Compression.<n>It replaces the U-Net with a Diffusion Transformer capable of performing diffusion in latent space entirely at 32x downscaled resolution.<n>It can reconstruct 2048x2048 images on a 16 GB laptop GPU.
  arXiv  Detail & Related papers  (2026-03-13T16:56:00Z)
• CoD: A Diffusion Foundation Model for Image Compression [57.572664625372106]
  Existing diffusion codecs typically build on text-to-image diffusion foundation models like Stable Diffusion.<n>textbfCoD can be trained from scratch to enable end-to-end optimization of both compression and generation.
  arXiv  Detail & Related papers  (2025-11-24T03:00:15Z)
• Diffusion As Self-Distillation: End-to-End Latent Diffusion In One Model [53.77953728335891]
  Latent Diffusion Models rely on a complex, three-part architecture consisting of a separate encoder, decoder, and diffusion network.<n>We propose Diffusion as Self-Distillation (DSD), a new framework with key modifications to the training objective that stabilize the latent space.<n>This approach enables, for the first time, the stable end-to-end training of a single network that simultaneously learns to encode, decode, and perform diffusion.
  arXiv  Detail & Related papers  (2025-11-18T17:58:16Z)
• Fine-structure Preserved Real-world Image Super-resolution via Transfer VAE Training [14.058527210122831]
  We propose a Transfer VAE Training (TVT) strategy to transfer the 8$times$ downsampled VAE into a 4$times$ one while adapting to the pre-trained UNet.<n>TVT strategy aligns the new encoder-decoder pair with the original VAE latent space while enhancing image fine details.
  arXiv  Detail & Related papers  (2025-07-27T14:11:29Z)
• Finding Local Diffusion Schrödinger Bridge using Kolmogorov-Arnold Network [7.651628106346734]
  This paper proposes for the first time to find local Diffusion Schr"odinger Bridges (LDSB) in the diffusion path subspace.<n>The experiment shows that our LDSB significantly improves the quality and efficiency of image generation using the same pre-trained denoising network.
  arXiv  Detail & Related papers  (2025-02-27T04:34:03Z)
• Diffusion Models Need Visual Priors for Image Generation [86.92260591389818]
  Diffusion on Diffusion (DoD) is an innovative multi-stage generation framework that first extracts visual priors from previously generated samples, then provides rich guidance for the diffusion model. We evaluate DoD on the popular ImageNet-$256 times 256$ dataset, reducing 7$times$ training cost compared to SiT and DiT. Our largest model DoD-XL achieves an FID-50K score of 1.83 with only 1 million training steps, which surpasses other state-of-the-art methods without bells and whistles during inference.
  arXiv  Detail & Related papers  (2024-10-11T05:03:56Z)
• PaGoDA: Progressive Growing of a One-Step Generator from a Low-Resolution Diffusion Teacher [55.22994720855957]
  PaGoDA is a novel pipeline that reduces the training costs through three stages: training diffusion on downsampled data, distilling the pretrained diffusion, and progressive super-resolution. With the proposed pipeline, PaGoDA achieves a $64times$ reduced cost in training its diffusion model on 8x downsampled data. PaGoDA's pipeline can be applied directly in the latent space, adding compression alongside the pre-trained autoencoder in Latent Diffusion Models.
  arXiv  Detail & Related papers  (2024-05-23T17:39:09Z)
• Cross-view Masked Diffusion Transformers for Person Image Synthesis [21.242398582282522]
  We present X-MDPT, a novel diffusion model designed for pose-guided human image generation. X-MDPT distinguishes itself by employing masked diffusion transformers that operate on latent patches. Our model outperforms state-of-the-art approaches on the DeepFashion dataset.
  arXiv  Detail & Related papers  (2024-02-02T15:57:13Z)
• Patch Diffusion: Faster and More Data-Efficient Training of Diffusion Models [166.64847903649598]
  We propose Patch Diffusion, a generic patch-wise training framework. Patch Diffusion significantly reduces the training time costs while improving data efficiency. We achieve outstanding FID scores in line with state-of-the-art benchmarks.
  arXiv  Detail & Related papers  (2023-04-25T02:35:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.