Post-Training Pruning for Diffusion Transformers

• URL: http://arxiv.org/abs/2607.00927v2
• Date: Fri, 03 Jul 2026 01:05:20 GMT
• Title: Post-Training Pruning for Diffusion Transformers
• Abstract summary: Diffusion Transformers (DiTs) have demonstrated impressive performance in image generation but suffer from substantial computational overhead and resource consumption.<n>Traditional pruning methods amplify the relative contribution of weights in the saliency metric.<n>We propose DiT-Pruning, which improves pruning performance by introducing customized saliency criteria and pruning granularity.
• Score: 23.732968732627825
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diffusion Transformers (DiTs) have demonstrated impressive performance in image generation but suffer from substantial computational overhead and resource consumption. Post-training pruning offers a promising solution; however, due to DiTs' unique architectural design and parameter distribution, traditional pruning methods are inapplicable, leading to significant performance degradation. Specifically, prior methods developed for LLMs, which derive metrics through a series of approximations, amplify the relative contribution of weights in the saliency metric. In addition, weights in DiTs exhibit significantly larger magnitudes than those in LLMs. Moreover, existing pruning granularity overlooks variations in model structures. In this paper, we propose DiT-Pruning, which improves pruning performance by introducing customized saliency criteria and pruning granularity. We design a novel metric that balances the contributions of weights and activations from an energy-based perspective, enabling more effective identification of important elements. Furthermore, we observe distinct clustering patterns in the two-dimensional weight space. Accordingly, we adopt a clustering-aware pruning granularity, enabling effective sparse allocation. Extensive evaluations on various DiTs show that our method consistently preserves image quality, especially under high sparsity. For FLUX.1-dev at 512x512 resolution on MJHQ, DiT-Pruning achieves only a 0.001 loss in CLIP score at 50% sparsity, dramatically outperforming recent pruning methods.

Related papers

• RT-Lynx: Putting the GEMM Sparsity In a Right Way for Diffusion Models [5.786869511584382]
  We show that DiT activations are intrinsically sparse and significantly more robust to N:M semi-structured sparsification than weights.<n>We propose RT-Lynx, which applies N:M sparsification to activations and incorporates error-compensation techniques to mitigate loss of accuracy.
  arXiv  Detail & Related papers  (2026-05-26T07:09:49Z)
• Sparsity Induction for Accurate Post-Training Pruning of Large Language Models [23.002927923453118]
  Post-training sparsity (PTS) reduces model cost by removing weights from dense networks.<n>But native dense matrices lack high sparsity, making existing approaches that directly remove weights disrupt model states.<n>We propose Sparsity Induction, which promotes models toward higher sparsity at both distribution and feature levels before pruning.
  arXiv  Detail & Related papers  (2026-02-25T07:25:01Z)
• Gradually Compacting Large Language Models for Reasoning Like a Boiling Frog [72.4168434368873]
  Large Language Models (LLMs) have demonstrated impressive reasoning capabilities, but their substantial size often demands significant computational resources.<n>We propose a gradual compacting method that divides the compression process into multiple fine-grained iterations.<n>This iterative approach-reminiscent of the "boiling frog" effect-enables the model to be progressively compressed without abrupt performance loss.
  arXiv  Detail & Related papers  (2026-02-04T06:56:52Z)
• Towards Extreme Pruning of LLMs with Plug-and-Play Mixed Sparsity [32.668409666483626]
  Existing pruning methods mainly focus on designing metrics to measure the importance of network components to guide pruning.<n>We propose an efficient method based on the trace of Fisher Information Matrix (FIM) to quantitatively measure and verify the different sensitivities across layers.<n>Based on this, we propose Mixed Sparsity Pruning (MSP) which uses a pruning-oriented evolutionary algorithm (EA) to determine the optimal sparsity levels for different layers.
  arXiv  Detail & Related papers  (2025-03-14T08:05:49Z)
• RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Models [53.571195477043496]
  We propose an algorithm named Rotated Straight-Through-Estimator (RoSTE)<n>RoSTE combines quantization-aware supervised fine-tuning (QA-SFT) with an adaptive rotation strategy to reduce activation outliers.<n>Our findings reveal that the prediction error is directly proportional to the quantization error of the converged weights, which can be effectively managed through an optimized rotation configuration.
  arXiv  Detail & Related papers  (2025-02-13T06:44:33Z)
• ALoRE: Efficient Visual Adaptation via Aggregating Low Rank Experts [71.91042186338163]
  ALoRE is a novel PETL method that reuses the hypercomplex parameterized space constructed by Kronecker product to Aggregate Low Rank Experts.<n>Thanks to the artful design, ALoRE maintains negligible extra parameters and can be effortlessly merged into the frozen backbone.
  arXiv  Detail & Related papers  (2024-12-11T12:31:30Z)
• Search for Efficient Large Language Models [52.98684997131108]
  Large Language Models (LLMs) have long held sway in the realms of artificial intelligence research. Weight pruning, quantization, and distillation have been embraced to compress LLMs, targeting memory reduction and inference acceleration. Most model compression techniques concentrate on weight optimization, overlooking the exploration of optimal architectures.
  arXiv  Detail & Related papers  (2024-09-25T21:32:12Z)
• A deeper look at depth pruning of LLMs [49.30061112976263]
  Large Language Models (LLMs) are resource-intensive to train but more costly to deploy in production. Recent work has attempted to prune blocks of LLMs based on cheap proxies for estimating block importance. We show that adaptive metrics exhibit a trade-off in performance between tasks.
  arXiv  Detail & Related papers  (2024-07-23T08:40:27Z)
• Isomorphic Pruning for Vision Models [56.286064975443026]
  Structured pruning reduces the computational overhead of deep neural networks by removing redundant sub-structures. We present Isomorphic Pruning, a simple approach that demonstrates effectiveness across a range of network architectures.
  arXiv  Detail & Related papers  (2024-07-05T16:14:53Z)
• SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models [53.638791265113625]
  Sparsity-Preserved efficient fine-tuning method for large language models. Code will be made available at https://github.com/Lucky-Lance/SPP.
  arXiv  Detail & Related papers  (2024-05-25T04:55:27Z)
• Enhanced Sparsification via Stimulative Training [36.0559905521154]
  Existing methods commonly set sparsity-inducing penalty terms to suppress the importance of dropped weights. We propose a structured pruning framework, named expressivity, based on an enhanced sparsification paradigm. To reduce the huge capacity gap of distillation, we propose a mutating expansion technique.
  arXiv  Detail & Related papers  (2024-03-11T04:05:17Z)
• Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models [30.246821533532017]
  Large Language Models (LLMs) with billions of parameters are prime targets for network pruning, removing some model weights without hurting performance. We present a novel sparsity-centric pruning method for pretrained LLMs, termed Gradient-based Language Model Pruner (GBLM-Pruner)
  arXiv  Detail & Related papers  (2023-11-08T18:59:54Z)
• LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning [56.88751562302793]
  Low-rank adaption (LoRA) has emerged to fine-tune large language models (LLMs) LoRAPrune is a new framework that delivers an accurate structured pruned model in a highly memory-efficient manner. LoRAPrune achieves a reduction in perplexity by 4.81 on WikiText2 and 3.46 on PTB, while also decreasing memory usage by 52.6%.
  arXiv  Detail & Related papers  (2023-05-28T15:15:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.