HierarchicalPrune: Position-Aware Compression for Large-Scale Diffusion Models

• URL: http://arxiv.org/abs/2508.04663v1
• Date: Wed, 06 Aug 2025 17:30:44 GMT
• Title: HierarchicalPrune: Position-Aware Compression for Large-Scale Diffusion Models
• Abstract summary: We present HierarchicalPrune, a novel compression framework grounded in a key observation.<n>DM blocks exhibit distinct functional hierarchies, where early blocks establish semantic structures while later blocks handle texture refinements.<n>Our framework brings billion-scale diffusion models into a range more suitable for on-device inference, while preserving the quality of the output images.
• Score: 15.143674484507251
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: State-of-the-art text-to-image diffusion models (DMs) achieve remarkable quality, yet their massive parameter scale (8-11B) poses significant challenges for inferences on resource-constrained devices. In this paper, we present HierarchicalPrune, a novel compression framework grounded in a key observation: DM blocks exhibit distinct functional hierarchies, where early blocks establish semantic structures while later blocks handle texture refinements. HierarchicalPrune synergistically combines three techniques: (1) Hierarchical Position Pruning, which identifies and removes less essential later blocks based on position hierarchy; (2) Positional Weight Preservation, which systematically protects early model portions that are essential for semantic structural integrity; and (3) Sensitivity-Guided Distillation, which adjusts knowledge-transfer intensity based on our discovery of block-wise sensitivity variations. As a result, our framework brings billion-scale diffusion models into a range more suitable for on-device inference, while preserving the quality of the output images. Specifically, when combined with INT4 weight quantisation, HierarchicalPrune achieves 77.5-80.4% memory footprint reduction (e.g., from 15.8 GB to 3.2 GB) and 27.9-38.0% latency reduction, measured on server and consumer grade GPUs, with the minimum drop of 2.6% in GenEval score and 7% in HPSv2 score compared to the original model. Last but not least, our comprehensive user study with 85 participants demonstrates that HierarchicalPrune maintains perceptual quality comparable to the original model while significantly outperforming prior works.

Related papers

• Cascaded Multi-Granularity Pruning for On-Device LLM Inference in Industrial IoT [4.775543545907101]
  This article presents a cascaded multi-granularity pruning framework that removes layers, attention heads, and feed-forward channels in coarse-to-fine order.<n>An information-theoretic analysis motivates this ordering, and the Structural Independence Assumption (SIA) is formalized as a checkable condition.<n> Deployed on an industrial slewing bearing fault diagnosis platform with NVIDIA DGX Spark, compressed models reduce inference latency by up to 67.2% and peak memory by 62.5%.
  arXiv  Detail & Related papers  (2026-06-25T10:44:48Z)
• TriSP: Tri-Signal Structured Pruning for Large Language Models [0.0]
  Large language models (LLMs) achieve strong performance across diverse tasks but their deployment is constrained by the memory and compute cost of their parameters.<n> Structured pruning addresses this by removing entire structures such as attention heads and Multi-Layer Perceptron (MLP) neurons to produce smaller dense models that run efficiently on standard hardware.<n>We propose TriSP (Tri-Signal Structured Pruning), an importance metric that combines weight magnitude scaled by activation norm with first-order gradient sensitivity via a geometric mean.
  arXiv  Detail & Related papers  (2026-06-09T17:02:14Z)
• Hierarchical Consistency Learning for Test-time Adaptation in Camouflage Perception [50.278200968044665]
  Camouflaged object detection (COD) aims to localize targets that exhibit minimal perceptual differences from backgrounds through physical attributes.<n>Existing methods, constrained by the static train-then-freeze paradigm, suffer from domain rigidity and dependency annotation.<n>We propose the hierarchical consistency learning framework, which integrates test-time adaptation for dynamic representation recalibration.
  arXiv  Detail & Related papers  (2026-05-25T09:57:46Z)
• Bridging Semantic and Kinematic Conditions with Diffusion-based Discrete Motion Tokenizer [55.9892973179428]
  MoTok is a discrete motion tokenizer that decouples semantic abstraction from fine-grained reconstruction.<n>On HumanML3D, our method significantly improves controllability and fidelity over MaskControl while using only one-sixth of the tokens.
  arXiv  Detail & Related papers  (2026-03-19T17:59:51Z)
• Breaking Semantic Hegemony: Decoupling Principal and Residual Subspaces for Generalized OOD Detection [10.596067236901968]
  We investigate the Simplicity Paradox in state-of-the-art (SOTA) models.<n>Models exhibit keen sensitivity in distinguishing semantically subtle OOD samples but suffer from severe Geometric Blindness.<n>We propose D-KNN, a training-free, plug-and-play geometric decoupling framework.
  arXiv  Detail & Related papers  (2026-02-05T06:32:33Z)
• HQP: Sensitivity-Aware Hybrid Quantization and Pruning for Ultra-Low-Latency Edge AI Inference [0.0]
  Hybrid Quantization and Pruning (HQP) framework designed to achieve synergistic model acceleration.<n>HQP framework achieves a peak performance gain of 3.12 times inference speedup and a 55 percent model size reduction.
  arXiv  Detail & Related papers  (2026-02-02T18:17:45Z)
• Which Layer Causes Distribution Deviation? Entropy-Guided Adaptive Pruning for Diffusion and Flow Models [77.55829017952728]
  EntPruner is an entropy-guided automatic progressive pruning framework for diffusion and flow models.<n>Experiments on DiT and SiT models demonstrate the effectiveness of EntPruner, achieving up to 2.22$times$ inference speedup.
  arXiv  Detail & Related papers  (2025-11-26T07:20:48Z)
• Deeply-Conditioned Image Compression via Self-Generated Priors [75.29511865838812]
  We introduce a framework predicated on functional decomposition, which we term Deeply-Conditioned Image Compression via self-generated priors (DCIC-sgp)<n>Our framework achieves significant BD-rate reductions of 14.4%, 15.7%, and 15.1% against the VVC test model VTM-12.1 on the Kodak, CLIC, and Tecnick datasets.
  arXiv  Detail & Related papers  (2025-10-28T14:04:19Z)
• Knowledge-Informed Neural Network for Complex-Valued SAR Image Recognition [51.03674130115878]
  We introduce the Knowledge-Informed Neural Network (KINN), a lightweight framework built upon a novel "compression-aggregation-compression" architecture.<n>KINN establishes a state-of-the-art in parameter-efficient recognition, offering exceptional generalization in data-scarce and out-of-distribution scenarios.
  arXiv  Detail & Related papers  (2025-10-23T07:12:26Z)
• Quantization Robustness to Input Degradations for Object Detection [2.0901018134712297]
  Post-training quantization (PTQ) is crucial for deploying efficient object detection models, like YOLO, on resource-constrained devices.<n>The impact of reduced precision on model robustness to real-world input degradations such as noise, blur, and compression artifacts is a significant concern.<n>This paper presents an empirical study evaluating the robustness of YOLO models across multiple precision formats.
  arXiv  Detail & Related papers  (2025-08-27T06:20:38Z)
• Exploring Layer-wise Information Effectiveness for Post-Training Quantization in Small Language Models [4.238165821317982]
  LieQ is a metric-driven framework that addresses the challenge of maintaining accuracy in sub-7B models under extreme low-bit compression.<n>Our method introduces three complementary layer-wise diagnostics-Perplexity Drop, Representational Compactness, and Top-k Energy Gain.<n>On Qwen3-4B, LieQ recovers 95.9% of FP16 baseline performance at 2.05-bit quantization, outperforming GPTQ by 19.7% and AWQ by 18.1% on average across seven zero-shot reasoning tasks.
  arXiv  Detail & Related papers  (2025-08-05T11:17:04Z)
• Process-aware and high-fidelity microstructure generation using stable diffusion [0.8060624778923473]
  We present a novel process-aware generative modeling approach based on Stable Diffusion 3.5 Large (SD3.5-Large)<n>Our method introduces numeric-aware embeddings that encode continuous variables directly into the model's conditioning.<n>We validate realism using a semantic segmentation model based on a fine-tuned U-Net with a VGG16 encoder on 24 labeled micrographs.
  arXiv  Detail & Related papers  (2025-07-01T06:16:53Z)
• Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free [81.65559031466452]
  We conduct experiments to investigate gating-augmented softmax attention variants.<n>We find that a simple modification-applying a head-specific sigmoid gate after the Scaled Dot-Product Attention (SDPA)-consistently improves performance.
  arXiv  Detail & Related papers  (2025-05-10T17:15:49Z)
• Controlled Latent Diffusion Models for 3D Porous Media Reconstruction [11.29275004613083]
  Three-dimensional digital reconstruction of porous media presents a fundamental challenge in geoscience.<n>We introduce a computational framework that addresses this challenge through latent diffusion models operating within the EDM framework.
  arXiv  Detail & Related papers  (2025-03-31T13:36:55Z)
• Self-Data Distillation for Recovering Quality in Pruned Large Language Models [1.5665059604715017]
  One-shot pruning results in significant quality degradation, particularly in tasks requiring multi-step reasoning.<n>To recover lost quality, supervised fine-tuning (SFT) is commonly applied, but it can lead to catastrophic forgetting.<n>In this work, we utilize self-data distilled fine-tuning to address these challenges.
  arXiv  Detail & Related papers  (2024-10-13T19:53:40Z)
• Comb, Prune, Distill: Towards Unified Pruning for Vision Model Compression [24.119415458653616]
  We propose a novel unified pruning framework Comb, Prune, Distill (CPD) to address both model-agnostic and task-agnostic concerns simultaneously. Our framework employs a combing step to resolve hierarchical layer-wise dependency issues, enabling architecture independence. In image classification we achieve a speedup of up to x4.3 with a accuracy loss of 1.8% and in semantic segmentation up to x1.89 with a 5.1% loss in mIoU.
  arXiv  Detail & Related papers  (2024-08-06T09:02:31Z)
• Model Inversion Attacks Through Target-Specific Conditional Diffusion Models [54.69008212790426]
  Model inversion attacks (MIAs) aim to reconstruct private images from a target classifier's training set, thereby raising privacy concerns in AI applications. Previous GAN-based MIAs tend to suffer from inferior generative fidelity due to GAN's inherent flaws and biased optimization within latent space. We propose Diffusion-based Model Inversion (Diff-MI) attacks to alleviate these issues.
  arXiv  Detail & Related papers  (2024-07-16T06:38:49Z)
• Isomorphic Pruning for Vision Models [56.286064975443026]
  Structured pruning reduces the computational overhead of deep neural networks by removing redundant sub-structures. We present Isomorphic Pruning, a simple approach that demonstrates effectiveness across a range of network architectures.
  arXiv  Detail & Related papers  (2024-07-05T16:14:53Z)
• A Generic Shared Attention Mechanism for Various Backbone Neural Networks [53.36677373145012]
  Self-attention modules (SAMs) produce strongly correlated attention maps across different layers. Dense-and-Implicit Attention (DIA) shares SAMs across layers and employs a long short-term memory module. Our simple yet effective DIA can consistently enhance various network backbones.
  arXiv  Detail & Related papers  (2022-10-27T13:24:08Z)
• SmoothNets: Optimizing CNN architecture design for differentially private deep learning [69.10072367807095]
  DPSGD requires clipping and noising of per-sample gradients. This introduces a reduction in model utility compared to non-private training. We distilled a new model architecture termed SmoothNet, which is characterised by increased robustness to the challenges of DP-SGD training.
  arXiv  Detail & Related papers  (2022-05-09T07:51:54Z)
• Efficient Micro-Structured Weight Unification and Pruning for Neural Network Compression [56.83861738731913]
  Deep Neural Network (DNN) models are essential for practical applications, especially for resource limited devices. Previous unstructured or structured weight pruning methods can hardly truly accelerate inference. We propose a generalized weight unification framework at a hardware compatible micro-structured level to achieve high amount of compression and acceleration.
  arXiv  Detail & Related papers  (2021-06-15T17:22:59Z)
• Training with Quantization Noise for Extreme Model Compression [57.51832088938618]
  We tackle the problem of producing compact models, maximizing their accuracy for a given model size. A standard solution is to train networks with Quantization Aware Training, where the weights are quantized during training and the gradients approximated with the Straight-Through Estimator. In this paper, we extend this approach to work beyond int8 fixed-point quantization with extreme compression methods.
  arXiv  Detail & Related papers  (2020-04-15T20:10:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.