ActTail: Global Activation Sparsity in Large Language Models

• URL: http://arxiv.org/abs/2603.12272v1
• Date: Wed, 18 Feb 2026 14:46:03 GMT
• Title: ActTail: Global Activation Sparsity in Large Language Models
• Abstract summary: We propose ActTail, a TopK magnitude-based activation sparsity method with global activation sparsity allocation grounded in Heavy-Tailed Self-Regularization (HT-SR) theory.<n>We show that our method improves both perplexity and downstream task performance at high sparsity compared to uniform allocation.
• Score: 14.120733678728191
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Activation sparsity is a promising approach for accelerating large language model (LLM) inference by reducing computation and memory movement. However, existing activation sparsity methods typically apply uniform sparsity across projections, ignoring the heterogeneous statistical properties of Transformer weights and thereby amplifying performance degradation. In this paper, we propose ActTail, a TopK magnitude-based activation sparsity method with global activation sparsity allocation grounded in Heavy-Tailed Self-Regularization (HT-SR) theory. Specifically, we capture this heterogeneity via the heavy-tail exponent computed from each projection's empirical spectral density (ESD), which is used as a quantitative indicator to assign projection-specific sparsity budgets. Importantly, we provide a theoretical analysis that establishes an explicit relationship between the activation sparsity ratio and the heavy-tail exponent under the HT-SR regime, offering principled guidance for sparsity allocation beyond heuristic design. Experiments on LLaMA and Mistral models show that our method improves both perplexity and downstream task performance at high sparsity compared to uniform allocation. At 80% sparsity, perplexity is reduced by 21.8% on LLaMA-2-7B, 40.1% on LLaMA-2-13B, and 9.4% on Mistral-7B.

Related papers

• RT-Lynx: Putting the GEMM Sparsity In a Right Way for Diffusion Models [5.786869511584382]
  We show that DiT activations are intrinsically sparse and significantly more robust to N:M semi-structured sparsification than weights.<n>We propose RT-Lynx, which applies N:M sparsification to activations and incorporates error-compensation techniques to mitigate loss of accuracy.
  arXiv  Detail & Related papers  (2026-05-26T07:09:49Z)
• Towards the Connection between Activation Sparsity and Flat Minima [46.83458699343789]
  Activation sparsity emerges in blocks of standardly trained Transformers.<n>We find that activation sparsity equals a ratio between "augmented flatness" and the product of the input and activation norm.<n>We propose derivative sparsity, which reduces to activation sparsity under ReLU.
  arXiv  Detail & Related papers  (2026-05-25T09:10:40Z)
• When Does Sparsity Mitigate the Curse of Depth in LLMs [53.137717161619484]
  We show that sparsity acts as a regulator of variance propagation and thereby improves depth utilization.<n>Our results reveal sparsity as a key yet previously overlooked mechanism for effective depth scaling in large language models.
  arXiv  Detail & Related papers  (2026-03-16T15:04:16Z)
• WiSparse: Boosting LLM Inference Efficiency with Weight-Aware Mixed Activation Sparsity [32.61817486761883]
  Training-free activation sparsity is a promising approach for efficient Large Language Models inference.<n>We propose Weight-aware Mixed-Granularity Training-free Activation Sparsity (WiSparse)<n>We show that WiSparse preserves 97% of Llama3.1's dense performance, surpassing the strongest baseline by 2.23 percentage points.
  arXiv  Detail & Related papers  (2026-02-16T04:18:36Z)
• Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective [55.90119819642064]
  We address the challenge of determining the layer-wise sparsity rates of large language models (LLMs) through a theoretical perspective.<n>This refers to the cumulative effect of reconstruction errors throughout the sparsification process.<n>We derive a simple yet effective approach to layer-wise sparsity allocation that mitigates this issue.
  arXiv  Detail & Related papers  (2025-02-20T17:51:10Z)
• Sparsing Law: Towards Large Language Models with Greater Activation Sparsity [64.15238674475619]
  Activation sparsity denotes the existence of substantial weakly-contributed elements within activation outputs that can be eliminated.<n>We propose PPL-$p%$ sparsity, a precise and performance-aware activation sparsity metric.<n>We show that ReLU is more efficient as the activation function than SiLU and can leverage more training data to improve activation sparsity.
  arXiv  Detail & Related papers  (2024-11-04T17:59:04Z)
• ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Models [74.59731375779934]
  Activation sparsity refers to the existence of weakly-contributed elements among activation outputs.<n>This paper introduces a simple and effective sparsification method named "ProSparse" to push LLMs for higher activation sparsity.
  arXiv  Detail & Related papers  (2024-02-21T03:58:49Z)
• Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity [88.62935593360162]
  Large Language Models (LLMs) are renowned for their remarkable performance across diverse domains.<n>We introduce a novel LLM pruning methodology that incorporates a tailored set of non-uniform layerwise sparsity ratios, termed as Outlier Weighed Layerwise sparsity (OWL)<n>OWL exhibits a remarkable performance gain, surpassing the state-of-the-art Wanda and SparseGPT by 61.22 and 6.80 perplexity at a high sparsity level of 70%, respectively.
  arXiv  Detail & Related papers  (2023-10-08T14:22:58Z)
• A Theoretical Explanation of Activation Sparsity through Flat Minima and Adversarial Robustness [29.87592869483743]
  A recent empirical observation of activation sparsity in blocks offers an opportunity to drastically reduce computation costs for free. We propose the notion of sparsity as one source of activation sparsity and a theoretical explanation based on it.
  arXiv  Detail & Related papers  (2023-09-06T13:48:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.