GPrune-LLM: Generalization-Aware Structured Pruning for Large Language Models

• URL: http://arxiv.org/abs/2603.13418v1
• Date: Thu, 12 Mar 2026 19:20:37 GMT
• Title: GPrune-LLM: Generalization-Aware Structured Pruning for Large Language Models
• Abstract summary: We propose GPrune-LLM, a generalization-aware structured pruning framework.<n>We first partition neurons into behavior-consistent modules to localize ranking competition.<n>For modules where activation-based scoring is unreliable, we switch to an activation-independent metric.
• Score: 17.33640761554548
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Structured pruning is widely used to compress large language models (LLMs), yet its effectiveness depends heavily on neuron importance estimation. Most existing methods estimate neuron importance from activation statistics on a single calibration dataset, which introduces calibration bias and degrades downstream cross-task generalization. We observe that neurons exhibit heterogeneous distribution sensitivity, with distribution-robust neurons maintaining consistent rankings across datasets and distribution-sensitive neurons showing high cross-dataset ranking variance. Based on this, we identify two structural limitations in existing methods. First, ranking all neurons within a shared space causes distribution-sensitive neurons that strongly activate on calibration inputs to dominate, crowding out distribution-robust neurons critical for out-of-distribution tasks. Second, applying activation-based importance metrics uniformly can be unreliable. Distribution-sensitive neurons that infrequently activate on calibration data receive insufficient activation signal for accurate local ranking. To address these limitations, we propose GPrune-LLM, a generalization-aware structured pruning framework that explicitly accounts for neuron differences in cross-distribution behavior. We first partition neurons into behavior-consistent modules to localize ranking competition, then evaluate activation-based metric reliability per module according to distribution sensitivity and score magnitude. For modules where activation-based scoring is unreliable, we switch to an activation-independent metric. Finally, we adaptively learn module-wise sparsity. Extensive experiments across multiple downstream tasks demonstrate GPrune-LLM's consistent improvements in post-compression generalization, particularly at high sparsity, and reduced dependence on importance metric choice.

Related papers

• Neuron-Guided Fine-Tuning: Unlocking Efficient Alignment Mechanisms for Large Language Models [73.88918843965155]
  Existing Supervised Fine-Tuning paradigms are often plagued by parameter redundancy, inconsistent data quality, and catastrophic forgetting.<n>We propose Neuron-Guided Fine-Tuning (NGFT), a holistic framework that leverages neuron activation patterns as a universal proxy to unify the fine-tuning lifecycle.
  arXiv  Detail & Related papers  (2026-09-05T06:13:24Z)
• Sparse Inter-Layer Dependencies of Transformer FFN Neurons [32.00544060412007]
  We study whether the activation of an FFN neuron can be explained by a sparse set of preceding neuron activations and attention outputs.<n>We introduce a training-free attribution method that estimates the relative influence of upstream neurons and attention outputs on a target neuron's activation.
  arXiv  Detail & Related papers  (2026-07-13T15:04:50Z)
• Neuron-Anchored Rule Extraction for Large Language Models via Contrastive Hierarchical Ablation [5.880505093493663]
  A key goal of explainable AI (XAI) is to express the decision logic of large language models (LLMs) in symbolic form.<n>We introduce MechaRule, a pipeline that grounds rule extraction in LLM circuits by efficiently localizing sparse neurons called agonists.
  arXiv  Detail & Related papers  (2026-05-04T18:27:37Z)
• Detecting and Pruning Prominent but Detrimental Neurons in Large Language Models [68.57424628540907]
  Large language models (LLMs) often develop learned mechanisms specialized to specific datasets.<n>We introduce a fine-tuning approach designed to enhance generalization by identifying and pruning neurons associated with dataset-specific mechanisms.<n>Our method employs Integrated Gradients to quantify each neuron's influence on high-confidence predictions, pinpointing those that disproportionately contribute to dataset-specific performance.
  arXiv  Detail & Related papers  (2025-07-12T08:10:10Z)
• Revisiting Large Language Model Pruning using Neuron Semantic Attribution [63.62836612864512]
  We conduct evaluations on 24 datasets and 4 tasks using popular pruning methods.<n>We surprisingly find a significant performance drop of existing pruning methods in sentiment classification tasks.<n>We propose Neuron Semantic Attribution, which learns to associate each neuron with specific semantics.
  arXiv  Detail & Related papers  (2025-03-03T13:52:17Z)
• Heterogeneous quantization regularizes spiking neural network activity [0.0]
  We present a data-blind neuromorphic signal conditioning strategy whereby analog data are normalized and quantized into spike phase representations. We extend this mechanism by adding a data-aware calibration step whereby the range and density of the quantization weights adapt to accumulated input statistics.
  arXiv  Detail & Related papers  (2024-09-27T02:25:44Z)
• Confidence Regulation Neurons in Language Models [91.90337752432075]
  This study investigates the mechanisms by which large language models represent and regulate uncertainty in next-token predictions. Entropy neurons are characterized by an unusually high weight norm and influence the final layer normalization (LayerNorm) scale to effectively scale down the logits. token frequency neurons, which we describe here for the first time, boost or suppress each token's logit proportionally to its log frequency, thereby shifting the output distribution towards or away from the unigram distribution.
  arXiv  Detail & Related papers  (2024-06-24T01:31:03Z)
• Decorrelating neurons using persistence [29.25969187808722]
  We present two regularisation terms computed from the weights of a minimum spanning tree of a clique. We demonstrate that naive minimisation of all correlations between neurons obtains lower accuracies than our regularisation terms. We include a proof of differentiability of our regularisers, thus developing the first effective topological persistence-based regularisation terms.
  arXiv  Detail & Related papers  (2023-08-09T11:09:14Z)
• Neuroevolutionary algorithms driven by neuron coverage metrics for semi-supervised classification [60.60571130467197]
  In some machine learning applications the availability of labeled instances for supervised classification is limited while unlabeled instances are abundant. We introduce neuroevolutionary approaches that exploit unlabeled instances by using neuron coverage metrics computed on the neural network architecture encoded by each candidate solution.
  arXiv  Detail & Related papers  (2023-03-05T23:38:44Z)
• Neuron with Steady Response Leads to Better Generalization [31.64957274989468]
  We propose a new regularization method called Neuron Steadiness Regularization to reduce neuron intra-class response variance. We conduct experiments on Multilayer Perceptron, Convolutional Neural Network, and Graph Neural Network with popular benchmark datasets of diverse domains.
  arXiv  Detail & Related papers  (2021-11-30T14:08:29Z)
• And/or trade-off in artificial neurons: impact on adversarial robustness [91.3755431537592]
  Presence of sufficient number of OR-like neurons in a network can lead to classification brittleness and increased vulnerability to adversarial attacks. We define AND-like neurons and propose measures to increase their proportion in the network. Experimental results on the MNIST dataset suggest that our approach holds promise as a direction for further exploration.
  arXiv  Detail & Related papers  (2021-02-15T08:19:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.