Beyond Layer Importance in Layer-wise Sparsity: An Inter-Layer Perturbation-Absorption Perspective

• URL: http://arxiv.org/abs/2606.15161v1
• Date: Sat, 13 Jun 2026 07:16:16 GMT
• Title: Beyond Layer Importance in Layer-wise Sparsity: An Inter-Layer Perturbation-Absorption Perspective
• Abstract summary: Layer-wise redundancy in large language models has established non-uniform sparsity allocation across layers as the standard pruning approach for efficient compression.<n>We characterize this property through controlled perturbation experiments.<n>We propose absorption-aware correction, an augmentation that improves outlier and AlphaPruning by reducing perplexity by 7.13% and boosting zero-shot accuracy by 1.02% across multiple model families at 70% sparsity.
• Score: 22.968807193719044
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The considerable layer-wise redundancy in large language models (LLMs) has established non-uniform sparsity allocation across layers as the standard pruning approach for efficient compression. Existing layer-wise allocation methods that estimate allocation strategy from local signals such as activation outliers or weight spectra mainly derive from local layer importance, whereas the final post-pruning performance is also influenced by the network's subsequent compensatory capacity. In this paper, we directly characterize this property through controlled perturbation experiments. We make the following empirical findings. First, layers exhibit highly heterogeneous responses to pruning-scale perturbations. In most cases, early layers amplify perturbations, while middle and late layers actively absorb them, with relative L2 drift decreasing monotonically across depth and direction realigning toward the unperturbed hidden-state trajectory. Second, absorption is a large-perturbation phenomenon. Under small perturbations the network exhibits amplification across all layers, and the transition to absorption occurs smoothly as perturbation magnitude grows to pruning scale. This enriches the linearized accumulation theory underlying related works. Building on these findings, we define an absorption coefficient per layer and propose absorption-aware correction, an orthogonal augmentation that improves OWL and AlphaPruning by reducing perplexity by 7.13% and boosting zero-shot accuracy by 1.02% across multiple model families at 70% sparsity.

Related papers

• CascadeFormer: Depth-Tapered Transformers Motivated by Gradient Fan-in Asymmetry [34.36337082439934]
  Deep Transformers are composed of uniformly stacked residual blocks, yet their deepest layers add little value.<n>We present two efficiency methods that exploit this asymmetry.<n> CascadeFormer tapers width with depth to match the uneven information flow across layers, achieving comparable perplexity to a uniform baseline at the same training budget while reducing latency by 8.6% and increasing throughput by 9.4%.
  arXiv  Detail & Related papers  (2026-06-25T02:25:00Z)
• Learning to Perturb Hidden Representations for Generalizable Deep Learning [3.2608238154676843]
  We show that Dropout, Manifold Mixup, adversarial feature perturbation, and related methods all impose specific forms of activation perturbation but with class-agnostic or random strategies.<n>We propose Learning to Perturb Activations (LPA), which adaptively perturbs activations at a selected hidden layer with class-level perturbations learned via PGD.
  arXiv  Detail & Related papers  (2026-05-28T07:41:39Z)
• A Single Layer to Explain Them All:Understanding Massive Activations in Large Language Models [56.03096341469333]
  We investigate the origins of massive activations in large language models (LLMs)<n>We identify a specific layer named the textbfMassive Emergence Layer (ME Layer), that is consistently observed across model families.<n>We show that, within the ME Layer both the RMSNorm and the FFN parameters jointly contribute to the emergence of massive activations.<n>Motivated by this limitation, we propose a simple and effective method to reduce the rigidity of the massive activation token.
  arXiv  Detail & Related papers  (2026-05-08T21:37:27Z)
• When Does Sparsity Mitigate the Curse of Depth in LLMs [53.137717161619484]
  We show that sparsity acts as a regulator of variance propagation and thereby improves depth utilization.<n>Our results reveal sparsity as a key yet previously overlooked mechanism for effective depth scaling in large language models.
  arXiv  Detail & Related papers  (2026-03-16T15:04:16Z)
• GradMAP: Faster Layer Pruning with Gradient Metric and Projection Compensation [23.236542656505417]
  GradMAP is a faster layer pruning method with textbfGradient textbfMetric textbfAnd textbfProjection compensation.<n>In this study, we propose GradMAP, a faster layer pruning method with textbfGradient textbfMetric textbfAnd textbfProjection compensation.
  arXiv  Detail & Related papers  (2026-02-16T11:14:02Z)
• Spectral Gradient Descent Mitigates Anisotropy-Driven Misalignment: A Case Study in Phase Retrieval [13.218607858857295]
  Spectral gradient methods modify gradient updates by preserving directional information while discarding scale.<n>We investigate the mechanisms underlying these gains through a dynamical analysis of a nonlinear phase retrieval model.
  arXiv  Detail & Related papers  (2026-01-30T07:12:58Z)
• Towards the Training of Deeper Predictive Coding Neural Networks [44.14001498773255]
  Predictive coding networks are neural models that perform inference through an iterative energy minimization process.<n>While effective in shallow architectures, they suffer significant performance degradation beyond five to seven layers.<n>We show that this degradation is caused by exponentially imbalanced errors between layers during weight updates, and by predictions from the previous layers not being effective in guiding updates in deeper layers.
  arXiv  Detail & Related papers  (2025-06-30T12:44:47Z)
• Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective [55.90119819642064]
  We address the challenge of determining the layer-wise sparsity rates of large language models (LLMs) through a theoretical perspective.<n>This refers to the cumulative effect of reconstruction errors throughout the sparsification process.<n>We derive a simple yet effective approach to layer-wise sparsity allocation that mitigates this issue.
  arXiv  Detail & Related papers  (2025-02-20T17:51:10Z)
• Initialization Matters: On the Benign Overfitting of Two-Layer ReLU CNN with Fully Trainable Layers [20.25049261035324]
  We extend the analysis to two-layer ReLU convolutional neural networks (CNNs) with fully trainable layers. Our results show that the scaling of the output layer is crucial to the training dynamics. In both settings, we provide nearly matching upper and lower bounds on the test errors.
  arXiv  Detail & Related papers  (2024-10-24T20:15:45Z)
• The Unreasonable Ineffectiveness of the Deeper Layers [5.984361440126354]
  We find that removing a certain layer does not affect model performance in common question-answering benchmarks.<n>Surprisingly, with this method we find minimal degradation of performance until after a large fraction of the layers are removed.<n>From a scientific perspective, the robustness of these LLMs to the deletion of layers implies either that current pretraining methods are not properly leveraging the parameters in the deeper layers of the network or that the shallow layers play a critical role in storing knowledge.
  arXiv  Detail & Related papers  (2024-03-26T17:20:04Z)
• Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion [83.90492831583997]
  We show that a batch-normalized network can keep the optimal signal propagation properties, but avoid exploding gradients in depth. We use a Multi-Layer Perceptron (MLP) with linear activations and batch-normalization that provably has bounded depth. We also design an activation shaping scheme that empirically achieves the same properties for certain non-linear activations.
  arXiv  Detail & Related papers  (2023-10-03T12:35:02Z)
• Layer-adaptive sparsity for the Magnitude-based Pruning [88.37510230946478]
  We propose a novel importance score for global pruning, coined layer-adaptive magnitude-based pruning (LAMP) score. LAMP consistently outperforms popular existing schemes for layerwise sparsity selection.
  arXiv  Detail & Related papers  (2020-10-15T09:14:02Z)
• Deriving Differential Target Propagation from Iterating Approximate Inverses [91.3755431537592]
  We show that a particular form of target propagation, relying on learned inverses of each layer, which is differential, gives rise to an update rule which corresponds to an approximate Gauss-Newton gradient-based optimization. We consider several iterative calculations based on local auto-encoders at each layer in order to achieve more precise inversions for more accurate target propagation.
  arXiv  Detail & Related papers  (2020-07-29T22:34:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.