ALTER: All-in-One Layer Pruning and Temporal Expert Routing for Efficient Diffusion Generation

• URL: http://arxiv.org/abs/2505.21817v1
• Date: Tue, 27 May 2025 22:59:44 GMT
• Title: ALTER: All-in-One Layer Pruning and Temporal Expert Routing for Efficient Diffusion Generation
• Abstract summary: We introduce ALTER: All-in-One Layer Pruning and Temporal Expert Routing.<n>A unified framework that transforms diffusion models into a mixture of efficient temporal experts.<n>A single-stage optimization that unifies layer pruning, expert routing, and model fine-tuning by employing a trainable hypernetwork.
• Score: 40.68265817413368
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diffusion models have demonstrated exceptional capabilities in generating high-fidelity images. However, their iterative denoising process results in significant computational overhead during inference, limiting their practical deployment in resource-constrained environments. Existing acceleration methods often adopt uniform strategies that fail to capture the temporal variations during diffusion generation, while the commonly adopted sequential pruning-then-fine-tuning strategy suffers from sub-optimality due to the misalignment between pruning decisions made on pretrained weights and the model's final parameters. To address these limitations, we introduce ALTER: All-in-One Layer Pruning and Temporal Expert Routing, a unified framework that transforms diffusion models into a mixture of efficient temporal experts. ALTER achieves a single-stage optimization that unifies layer pruning, expert routing, and model fine-tuning by employing a trainable hypernetwork, which dynamically generates layer pruning decisions and manages timestep routing to specialized, pruned expert sub-networks throughout the ongoing fine-tuning of the UNet. This unified co-optimization strategy enables significant efficiency gains while preserving high generative quality. Specifically, ALTER achieves same-level visual fidelity to the original 50-step Stable Diffusion v2.1 model while utilizing only 25.9% of its total MACs with just 20 inference steps and delivering a 3.64x speedup through 35% sparsity.

Related papers

• Rethinking Pixel Mean Flows via Interval Denoiser [95.11499828543344]
  We propose the Interval Denoiser, a framework for latent-free generation.<n>Our model achieves an FID of 4.55 in one step (1-NFE) and 3.98 in two steps (2-NFE) without perceptual losses.
  arXiv  Detail & Related papers  (2026-08-05T13:22:43Z)
• WAM-Diff2: Hierarchical AR-to-Diffusion Distillation for Highly Efficient Autonomous Driving VLA [27.102605117827398]
  Vision-Language-Action (VLA) models have emerged as a prominent paradigm for end-to-end autonomous driving.<n>They are constrained by high computational latency and exposure bias arising from sequential autoregressive decoding.<n>We introduce WAM-Diff2, a multi-task discrete diffusion VLA framework powered by a three-stage hierarchical strategy.
  arXiv  Detail & Related papers  (2026-08-02T06:45:16Z)
• PrismFlow: Residual Dynamics for Flow Matching in Time-Series Generation [24.703364668945543]
  Flow Matching offers an efficient alternative to diffusion models, but practical implementations typically rely on a single finite-capacity global vector-field estimator.<n>A monolithic estimator trained with the standard $ell$ velocity-matching objective may therefore learn an overly smoothed approximation of the local transport field.<n>This estimator-level smoothing can branch-specific dynamics, leading to spectral distortion and poor mode coverage.<n>We propose PrismFlow, a new FM method with Koopman-inspired dynamical experts.
  arXiv  Detail & Related papers  (2026-05-22T07:10:20Z)
• From Flow to One Step: Real-Time Multi-Modal Trajectory Policies via Implicit Maximum Likelihood Estimation-based Distribution Distillation [18.70033095161235]
  We propose a framework that distills a Conditional Flow Matching expert into a fast single-step student via Implicit Likelihood Estimation (IMLE)<n>A bi-directional Chamfer distance provides a set-level objective that promotes both mode coverage and fidelity.<n>A unified perception encoder further integrates multi-view RGB, depth, point clouds, and proprioception into a geometry-aware representation.
  arXiv  Detail & Related papers  (2026-03-10T09:30:05Z)
• Diff-ES: Stage-wise Structural Diffusion Pruning via Evolutionary Search [40.67449277026597]
  We introduce textbfDiff-ES, a stage-wise structural textbfDiff-usion pruning framework via textbfEvolutionary textbfSearch.<n>Our framework naturally integrates with existing structured pruning methods for diffusion models including depth and width pruning.<n>Experiments on DiT and SDXL demonstrate that Diff-ES consistently achieves wall-clock speedups while incurring minimal degradation in generation quality.
  arXiv  Detail & Related papers  (2026-03-05T12:18:40Z)
• RAPID^3: Tri-Level Reinforced Acceleration Policies for Diffusion Transformer [86.57077884971478]
  Diffusion Transformers (DiTs) excel at visual generation yet remain hampered by slow sampling.<n>We introduce RAPID3: Tri-Level Reinforced Acceleration Policies for Diffusion Transformers.<n>It delivers image-wise acceleration with zero updates to the base generator.<n>It achieves nearly 3x faster sampling with competitive generation quality.
  arXiv  Detail & Related papers  (2025-09-26T13:20:52Z)
• Duplex-GS: Proxy-Guided Weighted Blending for Real-Time Order-Independent Gaussian Splatting [37.17972426764452]
  We propose a dual-hierarchy framework that integrates proxy Gaussian representations with order-independent rendering techniques.<n>By seamlessly combining our framework with Order-Independent Transparency (OIT), we develop a physically inspired weighted sum rendering technique that simultaneously eliminates "popping" and "transparency" artifacts.<n>Our results validate the advantages of the OIT rendering paradigm in Gaussian Splatting, achieving high-quality rendering with an impressive 1.5 to 4 speedup over existing OIT based Gaussian Splatting approaches.
  arXiv  Detail & Related papers  (2025-08-05T07:44:30Z)
• Sortblock: Similarity-Aware Feature Reuse for Diffusion Model [9.749736545966694]
  Diffusion Transformers (DiTs) have demonstrated remarkable generative capabilities.<n>DiTs' sequential denoising process results in high inference latency.<n>We propose Sortblock, a training-free inference acceleration framework.
  arXiv  Detail & Related papers  (2025-08-01T08:10:54Z)
• Improving Progressive Generation with Decomposable Flow Matching [50.63174319509629]
  Decomposable Flow Matching (DFM) is a simple and effective framework for the progressive generation of visual media.<n>On Imagenet-1k 512px, DFM achieves 35.2% improvements in FDD scores over the base architecture and 26.4% over the best-performing baseline.
  arXiv  Detail & Related papers  (2025-06-24T17:58:02Z)
• Enhanced DACER Algorithm with High Diffusion Efficiency [30.13489573844592]
  We propose DACERv2, which exploits a Q-gradient field objective to guide the denoising process at each diffusion step.<n>We show that DACERv2 achieves higher performance in most complex control environments with only five diffusion steps and shows greater multimodality.
  arXiv  Detail & Related papers  (2025-05-29T13:21:58Z)
• DiffMoE: Dynamic Token Selection for Scalable Diffusion Transformers [86.5541501589166]
  DiffMoE is a batch-level global token pool that enables experts to access global token distributions during training.<n>It achieves state-of-the-art performance among diffusion models on ImageNet benchmark.<n>The effectiveness of our approach extends beyond class-conditional generation to more challenging tasks such as text-to-image generation.
  arXiv  Detail & Related papers  (2025-03-18T17:57:07Z)
• One-Step Diffusion Model for Image Motion-Deblurring [85.76149042561507]
  We propose a one-step diffusion model for deblurring (OSDD), a novel framework that reduces the denoising process to a single step.<n>To tackle fidelity loss in diffusion models, we introduce an enhanced variational autoencoder (eVAE), which improves structural restoration.<n>Our method achieves strong performance on both full and no-reference metrics.
  arXiv  Detail & Related papers  (2025-03-09T09:39:57Z)
• Diffusion Models as Optimizers for Efficient Planning in Offline RL [47.0835433289033]
  Diffusion models have shown strong competitiveness in offline reinforcement learning tasks. We propose a faster autoregressive model to handle the generation of feasible trajectories. This allows us to achieve more efficient planning without sacrificing capability.
  arXiv  Detail & Related papers  (2024-07-23T03:00:01Z)
• Spatial Annealing for Efficient Few-shot Neural Rendering [73.49548565633123]
  We introduce an accurate and efficient few-shot neural rendering method named textbfSpatial textbfAnnealing regularized textbfNeRF (textbfSANeRF)<n>By adding merely one line of code, SANeRF delivers superior rendering quality and much faster reconstruction speed compared to current few-shot neural rendering methods.
  arXiv  Detail & Related papers  (2024-06-12T02:48:52Z)
• A-SDM: Accelerating Stable Diffusion through Model Assembly and Feature Inheritance Strategies [51.7643024367548]
  Stable Diffusion Model is a prevalent and effective model for text-to-image (T2I) and image-to-image (I2I) generation. This study focuses on reducing redundant computation in SDM and optimizing the model through both tuning and tuning-free methods.
  arXiv  Detail & Related papers  (2024-05-31T21:47:05Z)
• LD-Pruner: Efficient Pruning of Latent Diffusion Models using Task-Agnostic Insights [2.8461446020965435]
  We introduce LD-Pruner, a novel performance-preserving structured pruning method for compressing Latent Diffusion Models. We demonstrate the effectiveness of our approach on three different tasks: text-to-image (T2I) generation, Unconditional Image Generation (UIG) and Unconditional Audio Generation (UAG)
  arXiv  Detail & Related papers  (2024-04-18T06:35:37Z)
• TMPQ-DM: Joint Timestep Reduction and Quantization Precision Selection for Efficient Diffusion Models [40.5153344875351]
  We introduce TMPQ-DM, which jointly optimize timestep reduction and quantization to achieve a superior performance-efficiency trade-off. For timestep reduction, we devise a non-uniform grouping scheme tailored to the non-uniform nature of the denoising process. In terms of quantization, we adopt a fine-grained layer-wise approach to allocate varying bit-widths to different layers based on their respective contributions to the final generative performance.
  arXiv  Detail & Related papers  (2024-04-15T07:51:40Z)
• Conditional Denoising Diffusion for Sequential Recommendation [62.127862728308045]
  Two prominent generative models, Generative Adversarial Networks (GANs) and Variational AutoEncoders (VAEs) GANs suffer from unstable optimization, while VAEs are prone to posterior collapse and over-smoothed generations. We present a conditional denoising diffusion model, which includes a sequence encoder, a cross-attentive denoising decoder, and a step-wise diffuser.
  arXiv  Detail & Related papers  (2023-04-22T15:32:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.