GradMAP: Faster Layer Pruning with Gradient Metric and Projection Compensation

• URL: http://arxiv.org/abs/2602.14649v1
• Date: Mon, 16 Feb 2026 11:14:02 GMT
• Title: GradMAP: Faster Layer Pruning with Gradient Metric and Projection Compensation
• Authors: Hao Liu, Guangyan Li, Wensheng Zhang, Yongqiang Tang,
• Abstract summary: GradMAP is a faster layer pruning method with textbfGradient textbfMetric textbfAnd textbfProjection compensation.<n>In this study, we propose GradMAP, a faster layer pruning method with textbfGradient textbfMetric textbfAnd textbfProjection compensation.
• Score: 23.236542656505417
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) exhibit strong reasoning abilities, but their high computational costs limit their practical deployment. Recent studies reveal significant redundancy in LLMs layers, making layer pruning an active research topic. Layer pruning research primarily focuses on two aspects: measuring layer importance and recovering performance after pruning. Unfortunately, the present works fail to simultaneously maintain pruning performance and efficiency. In this study, we propose GradMAP, a faster layer pruning method with \textbf{Grad}ient \textbf{M}etric \textbf{A}nd \textbf{P}rojection compensation, which consists of two stages. In the first stage, we introduce a novel metric based on gradient magnitudes, enabling a global assessment of layer importance. Note that, it requires only a single backward propagation step per pruning decision, substantially enhancing pruning efficiency. In the second stage, we first analyze the layers with the largest mean shift resulting from pruning, and then incorporate a simple yet effective projection compensation matrix to correct this drift in one step. In this way, the degradation of model performance caused by layer pruning is effectively alleviated. Extensive experiments show that GradMAP outperforms previous layer pruning methods in both pruning speed (achieving an average $4\times$ speedup) and performance.

Related papers

• GradPruner: Gradient-Guided Layer Pruning Enabling Efficient Fine-Tuning and Inference for LLMs [10.61152477422108]
  GradPruner can prune layers of Large Language Models guided by gradients in the early stages of fine-tuning.<n>Results demonstrate that GradPruner has achieved a parameter reduction of 40% with only a 0.99% decrease in accuracy.
  arXiv  Detail & Related papers  (2026-01-27T11:41:26Z)
• High-Layer Attention Pruning with Rescaling [14.141903038286362]
  Pruning is a highly effective approach for compressing large language models (LLMs)<n>We propose a novel pruning algorithm that strategically prunes attention heads in the model's higher layers.<n>We conduct comprehensive experiments on a wide range of LLMs, including LLaMA3.1-8B, Mistral-7B-v0.3, Qwen2-7B, and Gemma2-9B.
  arXiv  Detail & Related papers  (2025-07-02T17:15:05Z)
• ACE: Exploring Activation Cosine Similarity and Variance for Accurate and Calibration-Efficient LLM Pruning [15.933542902352604]
  We propose an efficient and effective pruning method that simultaneously achieves high pruning performance and fast pruning speed.<n> Experimental results show that our method achieves up to an 18% reduction in perplexity and up to 63% decrease in pruning time on prevalent LLMs.
  arXiv  Detail & Related papers  (2025-05-28T05:25:16Z)
• A Sliding Layer Merging Method for Efficient Depth-Wise Pruning in LLMs [13.000188564679998]
  This paper reveals the Patch-like'' feature relationship between layers in large language models by analyzing the correlation of the outputs of different layers in the reproducing kernel Hilbert space.<n>We propose a sliding layer merging method that dynamically selects and fuses consecutive layers from top to bottom according to a pre-defined similarity threshold.<n>Our method outperforms existing pruning techniques in both zero-shot inference performance and retraining recovery quality after pruning.
  arXiv  Detail & Related papers  (2025-02-26T14:15:24Z)
• Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective [55.90119819642064]
  We address the challenge of determining the layer-wise sparsity rates of large language models (LLMs) through a theoretical perspective.<n>This refers to the cumulative effect of reconstruction errors throughout the sparsification process.<n>We derive a simple yet effective approach to layer-wise sparsity allocation that mitigates this issue.
  arXiv  Detail & Related papers  (2025-02-20T17:51:10Z)
• LESA: Learnable LLM Layer Scaling-Up [57.0510934286449]
  Training Large Language Models (LLMs) from scratch requires immense computational resources, making it prohibitively expensive.<n>Model scaling-up offers a promising solution by leveraging the parameters of smaller models to create larger ones.<n>We propose textbfLESA, a novel learnable method for depth scaling-up.
  arXiv  Detail & Related papers  (2025-02-19T14:58:48Z)
• A deeper look at depth pruning of LLMs [49.30061112976263]
  Large Language Models (LLMs) are resource-intensive to train but more costly to deploy in production. Recent work has attempted to prune blocks of LLMs based on cheap proxies for estimating block importance. We show that adaptive metrics exhibit a trade-off in performance between tasks.
  arXiv  Detail & Related papers  (2024-07-23T08:40:27Z)
• Bypass Back-propagation: Optimization-based Structural Pruning for Large Language Models via Policy Gradient [57.9629676017527]
  We propose an optimization-based structural pruning that learns the pruning masks in a probabilistic space directly by optimizing the loss of the pruned model.<n>We achieve this by learning an underlying Bernoulli distribution to sample binary pruning masks.<n>Experiments conducted on LLaMA, LLaMA-2, LLaMA-3, Vicuna, and Mistral models demonstrate the promising performance of our method in efficiency and effectiveness.
  arXiv  Detail & Related papers  (2024-06-15T09:31:03Z)
• Effective Layer Pruning Through Similarity Metric Perspective [0.0]
  Deep neural networks have been the predominant paradigm in machine learning for solving cognitive tasks. Pruning structures from these models is a straightforward approach to reducing network complexity. Layer pruning often hurts the network predictive ability (i.e., accuracy) at high compression rates. This work introduces an effective layer-pruning strategy that meets all underlying properties pursued by pruning methods.
  arXiv  Detail & Related papers  (2024-05-27T11:54:51Z)
• Streamlining Redundant Layers to Compress Large Language Models [21.27944103424621]
  This paper introduces LLM-Streamline, a pioneer work on layer pruning for large language models (LLMs)<n>It is based on the observation that different layers have varying impacts on hidden states, enabling the identification of less important layers to be pruned.<n>Experiments show that LLM-Streamline outperforms both previous and concurrent state-of-the-art pruning methods in terms of both performance and training efficiency.
  arXiv  Detail & Related papers  (2024-03-28T04:12:13Z)
• Class Gradient Projection For Continual Learning [99.105266615448]
  Catastrophic forgetting is one of the most critical challenges in Continual Learning (CL) We propose Class Gradient Projection (CGP), which calculates the gradient subspace from individual classes rather than tasks.
  arXiv  Detail & Related papers  (2023-11-25T02:45:56Z)
• SkipNode: On Alleviating Performance Degradation for Deep Graph Convolutional Networks [84.30721808557871]
  We conduct theoretical and experimental analysis to explore the fundamental causes of performance degradation in deep GCNs. We propose a simple yet effective plug-and-play module, Skipnode, to overcome the performance degradation of deep GCNs.
  arXiv  Detail & Related papers  (2021-12-22T02:18:31Z)
• Sparse Training via Boosting Pruning Plasticity with Neuroregeneration [79.78184026678659]
  We study the effect of pruning throughout training from the perspective of pruning plasticity. We design a novel gradual magnitude pruning (GMP) method, named gradual pruning with zero-cost neuroregeneration (GraNet) and its dynamic sparse training (DST) variant (GraNet-ST) Perhaps most impressively, the latter for the first time boosts the sparse-to-sparse training performance over various dense-to-sparse methods by a large margin with ResNet-50 on ImageNet.
  arXiv  Detail & Related papers  (2021-06-19T02:09:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.