Prune-then-Quantize or Quantize-then-Prune? Understanding the Impact of Compression Order in Joint Model Compression

• URL: http://arxiv.org/abs/2603.18426v1
• Date: Thu, 19 Mar 2026 02:40:23 GMT
• Title: Prune-then-Quantize or Quantize-then-Prune? Understanding the Impact of Compression Order in Joint Model Compression
• Abstract summary: Joint model compression is a powerful strategy to achieve higher efficiency by combining multiple methods such as pruning and quantization.<n>A central but underexplored factor in joint model compression is the compression order, or the sequence of different methods within the compression pipeline.
• Score: 16.625053009148306
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: What happens when multiple compression methods are combined-does the order in which they are applied matter? Joint model compression has emerged as a powerful strategy to achieve higher efficiency by combining multiple methods such as pruning and quantization. A central but underexplored factor in joint model compression is the compression order, or the sequence of different methods within the compression pipeline. Most prior studies have either sidestepped the issue by assuming orthogonality between techniques, while a few have examined them only in highly constrained cases. Consequently, the broader role of compression order in shaping model performance remains poorly understood. In this paper, we address the overlooked problem of compression order and provide both theoretical and empirical analysis. We formulate the problem of optimizing the compression order and introduce the Progressive Intensity Hypothesis, which states that weaker perturbations should precede stronger ones. We provide theoretical guarantees showing that the relative benefit of one order increases with the underlying performance gap. Extensive experiments on both language and vision models validate the hypothesis, and further show its generality to broader setups such as multi-stage compression and mixed-precision quantization.

Related papers

• Compressing What Matters: Neuron Importance Meets Data-Aware Low Rank Approximation for Language Model Compression [39.47298454012977]
  We propose an enhanced and computationally efficient algorithm for dynamic compression rate allocation.<n> Experimental results support the efficacy of the proposed approach which performs on par or substantially better than the previous state-of-the-art.
  arXiv  Detail & Related papers  (2026-06-30T09:16:00Z)
• Break Through the Compression Bottleneck: From Theory to Practice [30.01535056565781]
  Low-rank decomposition and quantization have been proven to significantly reduce the computational and memory requirements of large language models.<n>This paper provides the first mathematical proof that low-rank decomposition and quantization are non-orthogonal.<n>We propose a novel approach Diagonal Adhesive Method (DAM) which can effectively combine the two methods and mitigate the performance loss.
  arXiv  Detail & Related papers  (2026-05-11T10:50:26Z)
• Arbitrary Ratio Feature Compression via Next Token Prediction [52.10426317889982]
  Arbitrary Ratio Feature Compression (ARFC) framework supports any compression ratio with a single model.<n>ARC is an auto-regressive model that performs compression via next-gressive prediction.<n>MoS module refines the compressed tokens by utilizing multiple compression results.<n>ERGC is integrated into the training process to preserve semantic and structural relationships during compression.
  arXiv  Detail & Related papers  (2026-02-12T02:38:57Z)
• TuneComp: Joint Fine-tuning and Compression for Large Foundation Models [50.33925662486034]
  sequential fine-tuning and compression sacrifices performance, while creating a larger than necessary model as an intermediate step.<n>We propose to jointly fine-tune and compress the model by gradually distilling it to a pruned low-rank structure.<n> Experiments demonstrate that joint fine-tuning and compression significantly outperforms other sequential compression methods.
  arXiv  Detail & Related papers  (2025-05-27T23:49:35Z)
• Problem-dependent convergence bounds for randomized linear gradient compression [4.656302602746228]
  We study the impact of compression on convergence associated with the objective function.<n>We find that the impact of compression on convergence can be in terms of a quantified matrix associated with the objective function.
  arXiv  Detail & Related papers  (2024-11-19T22:26:42Z)
• Order of Compression: A Systematic and Optimal Sequence to Combinationally Compress CNN [5.25545980258284]
  We propose a systematic and optimal sequence to apply multiple compression techniques in the most effective order. Our proposed Order of Compression significantly reduces computational costs by up to 859 times on ResNet34, with negligible accuracy loss. We believe our simple yet effective exploration of the order of compression will shed light on the practice of model compression.
  arXiv  Detail & Related papers  (2024-03-26T07:26:00Z)
• Activations and Gradients Compression for Model-Parallel Training [85.99744701008802]
  We study how simultaneous compression of activations and gradients in model-parallel distributed training setup affects convergence. We find that gradients require milder compression rates than activations. Experiments also show that models trained with TopK perform well only when compression is also applied during inference.
  arXiv  Detail & Related papers  (2024-01-15T15:54:54Z)
• Unified Multivariate Gaussian Mixture for Efficient Neural Image Compression [151.3826781154146]
  latent variables with priors and hyperpriors is an essential problem in variational image compression. We find inter-correlations and intra-correlations exist when observing latent variables in a vectorized perspective. Our model has better rate-distortion performance and an impressive $3.18times$ compression speed up.
  arXiv  Detail & Related papers  (2022-03-21T11:44:17Z)
• Compression of Generative Pre-trained Language Models via Quantization [62.80110048377957]
  We find that previous quantization methods fail on generative tasks due to the textithomogeneous word embeddings We propose a token-level contrastive distillation to learn distinguishable word embeddings, and a module-wise dynamic scaling to make quantizers adaptive to different modules.
  arXiv  Detail & Related papers  (2022-03-21T02:11:35Z)
• Linear Convergent Decentralized Optimization with Compression [50.44269451541387]
  Existing decentralized algorithms with compression mainly focus on compressing DGD-type algorithms. Motivated by primal-dual algorithms, this paper proposes first underlineLinunderlineEAr convergent. underlineDecentralized with compression, LEAD.
  arXiv  Detail & Related papers  (2020-07-01T04:35:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.