Distilling Drifting Transformers with Representation Autoencoders

• URL: http://arxiv.org/abs/2606.15553v1
• Date: Sun, 14 Jun 2026 02:46:50 GMT
• Title: Distilling Drifting Transformers with Representation Autoencoders
• Abstract summary: Representation Autoencoders (RAEs) have improved diffusion and flow models by semantically richer latent space.<n>In this work, we argue that the RAE latent space is compatible with distillation via the newly proposed Drifting Models.
• Score: 23.990618033500695
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Representation Autoencoders (RAEs) have improved diffusion and flow models by semantically richer latent space owing to the strongly label-wise clustered DINO features in the pretrained encoders. Yet in the distillation stage, the severe anisotropy and large curvatures caused by the rich semantic representations would hinder the convergence and performance, making the trajectory-based distillation unstable. In this work, we argue that the RAE latent space is compatible with distillation via the newly proposed Drifting Models. We first quantitatively study the curvatures and isotropy statistics across different autoencoders, and theoretically reveal that Drifting Model itself is highly likely to fail on extremely scattered spaces like reconstruction-based VAEs. These motivate us to apply the drifting paradigm directly to representation autoencoders. Our proposed method, Drift-RAE, distills pretrained flow models in RAE latent spaces using Drifting, together with insightful modifications that improve training stability by thereotically aligning drifting fields with other frameworks. Regarding the experimental evidences, we achieve 1.77 FID on ImageNet 256 dataset using only 10k distillation steps, surpassing state-of-the-art RAE distillation methods and appearing comparative with the original Drifting Model without requiring an auxiliary MAE feature extractor. The code will be made publicly available.

Related papers

• Second Order Drifting Models [9.446439690148713]
  Drifting models evolve the model distribution during training using a predefined sample-based drift field.<n>We propose Second-Order Drifting Models, which lift drifting dynamics into phase space by augmenting generated samples with artificial velocity variables.<n>We derive a practical semi-implicit training algorithm and evaluate it on synthetic distribution matching, sequential data generation, and robotic control.
  arXiv  Detail & Related papers  (2026-08-08T05:02:55Z)
• Curvature-Adaptive Consistency Flow Matching: Autonomous Trajectory Optimization via Reinforcement Learning [8.422673935994451]
  consistency distillation has significantly accelerated the inference of diffusion models.<n>We propose Curvature-Adaptive Consistency Flow Matching to address the limitations of static sampling.<n>Our approach achieves new state-of-the-art results on large-scale models such as FLUX and SDXL.
  arXiv  Detail & Related papers  (2026-06-21T08:49:50Z)
• Score Distillation of Flow Matching Models [67.86066177182046]
  We extend Score identity Distillation (SiD) to pretrained text-to-image flow-matching models.<n>SiD works out of the box across these models, in both data-free and data-aided settings.<n>This provides the first systematic evidence that score distillation applies broadly to text-to-image flow matching models.
  arXiv  Detail & Related papers  (2025-09-29T17:45:48Z)
• Adversarial Distribution Matching for Diffusion Distillation Towards Efficient Image and Video Synthesis [65.77083310980896]
  We propose Adrial Distribution Matching (ADM) to align latent predictions between real and fake score estimators for score distillation.<n>Our proposed method achieves superior one-step performance on SDXL compared to DMD2 while consuming less GPU time.<n>Additional experiments that apply multi-step ADM distillation on SD3-Medium, SD3.5-Large, and CogVideoX set a new benchmark towards efficient image and video synthesis.
  arXiv  Detail & Related papers  (2025-07-24T16:45:05Z)
• Taming Diffusion for Dataset Distillation with High Representativeness [49.3818035378669]
  D3HR is a novel diffusion-based framework to generate distilled datasets with high representativeness.<n>Our experiments demonstrate that D3HR can achieve higher accuracy across different model architectures.
  arXiv  Detail & Related papers  (2025-05-23T22:05:59Z)
• Restoration Score Distillation: From Corrupted Diffusion Pretraining to One-Step High-Quality Generation [82.39763984380625]
  We propose textitRestoration Score Distillation (RSD), a principled generalization of Denoising Score Distillation (DSD)<n>RSD accommodates a broader range of corruption types, such as blurred, incomplete, or low-resolution images.<n>It consistently surpasses its teacher model across diverse restoration tasks on both natural and scientific datasets.
  arXiv  Detail & Related papers  (2025-05-19T17:21:03Z)
• One-Step Offline Distillation of Diffusion-based Models via Koopman Modeling [26.913398550088477]
  We introduce the Koopman Distillation Model (KDM), a novel offline distillation approach grounded in Koopman theory.<n>KDM encodes noisy inputs into an embedded space where a learned linear operator propagates them forward, followed by a decoder that reconstructs clean samples.<n>KDM achieves highly competitive performance across standard offline distillation benchmarks.
  arXiv  Detail & Related papers  (2025-05-19T16:59:47Z)
• Denoising Score Distillation: From Noisy Diffusion Pretraining to One-Step High-Quality Generation [82.39763984380625]
  We introduce denoising score distillation (DSD), a surprisingly effective and novel approach for training high-quality generative models from low-quality data.<n>DSD pretrains a diffusion model exclusively on noisy, corrupted samples and then distills it into a one-step generator capable of producing refined, clean outputs.
  arXiv  Detail & Related papers  (2025-03-10T17:44:46Z)
• Distilling Diffusion Models into Conditional GANs [90.76040478677609]
  We distill a complex multistep diffusion model into a single-step conditional GAN student model. For efficient regression loss, we propose E-LatentLPIPS, a perceptual loss operating directly in diffusion model's latent space. We demonstrate that our one-step generator outperforms cutting-edge one-step diffusion distillation models.
  arXiv  Detail & Related papers  (2024-05-09T17:59:40Z)
• Adaptive Multi-step Refinement Network for Robust Point Cloud Registration [82.64560249066734]
  Point Cloud Registration estimates the relative rigid transformation between two point clouds of the same scene.<n>We propose an adaptive multi-step refinement network that refines the registration quality at each step by leveraging the information from the preceding step.<n>Our method achieves state-of-the-art performance on both the 3DMatch/3DLoMatch and KITTI benchmarks.
  arXiv  Detail & Related papers  (2023-12-05T18:59:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.