Importance-Aware Low-Rank Distillation of Diffusion Transformers

• URL: http://arxiv.org/abs/2609.04646v1
• Date: Fri, 04 Sep 2026 02:25:11 GMT
• Title: Importance-Aware Low-Rank Distillation of Diffusion Transformers
• Abstract summary: Diffusion Transformers (DiTs) have emerged as a dominant architecture for high-quality text-to-image generation.<n>We introduce SVDtrunc, a two-step block-level compression scheme.<n>We find that SVDtrunc produces smooth degradation even under substantial global compression.
• Score: 15.737230365885907
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion Transformers (DiTs) have emerged as a dominant architecture for high-quality text-to-image generation, yet their scale poses challenges for efficient deployment. While truncated singular value decomposition (SVD) is a principled tool for parameter reduction, evidence from large language models (LLMs) suggests that naive low-rank approximation can cause catastrophic failure. In contrast, we find that truncated SVD in DiTs produces smooth degradation even under substantial global compression, with redundancy distributed across projection matrices throughout the whole network rather than concentrated in a few transformer blocks. Building on these insights, we introduce SVDtrunc, a two-step block-level compression scheme, first allocating ranks across blocks and compressing the least important ones via truncated SVD under a global parameter budget, and then fine-tuning all blocks with modular knowledge distillation and a rectified-flow objective. We apply SVDtrunc to FLUX.dev across compression levels ranging from 40-90% of the original parameter count. Across three benchmarks, GenEval, HPSv2, and DPG, we outperform all competing approaches. Notably, and in contrast to prior work, we retain near-full performance at 68% and remain competitive even at 57% of the original parameter budget. Furthermore, we show that SVDtrunc complements step distillation and achieves strong results even without fine-tuning, positioning it as a practical continuation of efficiency improvements beyond diffusion step reduction for large-scale generative models. Project page: https://vislearn.github.io/SVDtrunc/

Related papers

• LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression [4.953709812007795]
  We propose LACE-SVD, a Loss-Aware SVD framework with Cumulative Error correction for LLM compression.<n>LACE-SVD first estimates the calibration negative-log-likelihood increase induced by candidate layer-wise compression ratios.<n>It then refines the compressed model with closed-form local updates and introduces a propagation-aware correction for residual-stream output modules.
  arXiv  Detail & Related papers  (2026-07-03T07:46:13Z)
• AA-SVD : Anchored and Adaptive SVD for Large Language Model Compression [19.41870881350408]
  We introduce a fast low-rank factorization-based framework for compressing large language models without retraining.<n>We refine each transformer block end-to-end, minimizing block-level output distortion.<n>Our method consistently outperforms existing SVD-based baselines across compression ratios.
  arXiv  Detail & Related papers  (2026-04-02T14:55:49Z)
• FlattenGPT: Depth Compression for Transformer with Layer Flattening [89.7587433008809]
  textbfFlattenGPT is a novel way to detect and reduce depth-wise redundancies.<n>Experiments demonstrate that FlattenGPT enhances model efficiency with a decent trade-off to performance.
  arXiv  Detail & Related papers  (2026-02-09T16:22:58Z)
• Low-Rank Prehab: Preparing Neural Networks for SVD Compression [12.618746274492421]
  Inspired by prehabilitation in surgery, we introduce a pre-compression fine-tuning stage, Low-Rank Prehab.<n>We show that Prehab substantially reduces the immediate accuracy drop after compression and consistently improves post-finetuning performance.<n>Our method outperforms state-of-the-art SVD-based techniques such as SVD-LLM.
  arXiv  Detail & Related papers  (2025-12-01T18:37:53Z)
• Quantized Visual Geometry Grounded Transformer [67.15451442018258]
  This paper proposes the first Quantization framework for VGGTs, namely QuantVGGT.<n>We introduce Dual-Smoothed Fine-Grained Quantization, which integrates pre-global Hadamard rotation and post-local channel smoothing.<n>We also design Noise-Filtered Diverse Sampling, which filters outliers via deep-layer statistics.
  arXiv  Detail & Related papers  (2025-09-25T15:17:11Z)
• CALR: Corrective Adaptive Low-Rank Decomposition for Efficient Large Language Model Layer Compression [0.0]
  Large Language Models (LLMs) present significant deployment challenges due to their immense size and computational requirements.<n>We introduce Corrective Adaptive Low-Rank Decomposition (CALR), a two-component compression approach.<n>We show that CALR can reduce parameter counts by 26.93% to 51.77% while retaining 59.45% to 90.42% of the original model's performance.
  arXiv  Detail & Related papers  (2025-08-21T13:16:02Z)
• FlashSVD: Memory-Efficient Inference with Streaming for Low-Rank Models [15.244129138320782]
  FlashSVD is an end-to-end rank-aware streaming inference framework for SVD-compressed large language models.<n>It cuts peak activation memory by up to 70.2% and intermediate transient memory by 75%.<n>It incurs no accuracy loss with upstream encodering compression methods, offering a practical path toward memory-constrained deployment of low-rank LLMs.
  arXiv  Detail & Related papers  (2025-08-02T22:06:46Z)
• AdaSVD: Adaptive Singular Value Decomposition for Large Language Models [75.1196637934987]
  Singular Value Decomposition (SVD) has emerged as a promising compression technique for large language models (LLMs)<n>Existing SVD-based methods often struggle to effectively mitigate the errors introduced by SVD truncation.<n>We propose AdaSVD, an adaptive SVD-based LLM compression approach.
  arXiv  Detail & Related papers  (2025-02-03T14:34:37Z)
• ALoRE: Efficient Visual Adaptation via Aggregating Low Rank Experts [71.91042186338163]
  ALoRE is a novel PETL method that reuses the hypercomplex parameterized space constructed by Kronecker product to Aggregate Low Rank Experts.<n>Thanks to the artful design, ALoRE maintains negligible extra parameters and can be effortlessly merged into the frozen backbone.
  arXiv  Detail & Related papers  (2024-12-11T12:31:30Z)
• TinyFusion: Diffusion Transformers Learned Shallow [52.96232442322824]
  Diffusion Transformers have demonstrated remarkable capabilities in image generation but often come with excessive parameterization.<n>We present TinyFusion, a depth pruning method designed to remove redundant layers from diffusion transformers via end-to-end learning.<n>Experiments with DiT-XL show that TinyFusion can craft a shallow diffusion transformer at less than 7% of the pre-training cost, achieving a 2$times$ speedup with an FID score of 2.86.
  arXiv  Detail & Related papers  (2024-12-02T07:05:39Z)
• From PEFT to DEFT: Parameter Efficient Finetuning for Reducing Activation Density in Transformers [52.199303258423306]
  We propose a novel density loss that encourages higher activation sparsity in pre-trained models. Our proposed method, textbfDEFT, can consistently reduce activation density by up to textbf44.94% on RoBERTa$_mathrmLarge$ and by textbf53.19% (encoder density) and textbf90.60% (decoder density) on Flan-T5$_mathrmXXL$.
  arXiv  Detail & Related papers  (2024-02-02T21:25:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.