Reservoir of Importance: Learning Semi-Structured Sparsity with Differentiable Subset Sampling

• URL: http://arxiv.org/abs/2608.23048v1
• Date: Mon, 24 Aug 2026 09:54:24 GMT
• Title: Reservoir of Importance: Learning Semi-Structured Sparsity with Differentiable Subset Sampling
• Abstract summary: Semi-structured $N$:$M$ sparsity has emerged as a practical direction for accelerating large language models (LLMs)<n>We propose Reservoir of Importance (RoI), a lightweight semi-structured pruning framework that learns sparsity masks through differentiable subset sampling.<n>RoI achieves competitive performance with strong memory efficiency, stability, and scalability to more aggressive $N$:$M$ sparsity patterns.
• Score: 6.232269207752904
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Semi-structured $N$:$M$ sparsity has emerged as a practical direction for accelerating large language models (LLMs). However, existing learnable-mask approaches incur substantial parameter and memory overhead, limiting their scalability to large models and aggressive sparsity regimes. In this work, we revisit semi-structured pruning from a perspective that reconciles efficiency with scalability. We propose Reservoir of Importance (RoI), a lightweight semi-structured pruning framework that learns sparsity masks through differentiable subset sampling. Unlike prior methods that model full categorical distributions over all feasible $N$:$M$ patterns, RoI introduces a compact-logit parameterization for sparsity mask learning and performs sampling without replacement to select masks, thereby reducing trainable parameters from combinatorial complexity to $\mathcal{O}({M})$. As a result, RoI requires 1.5-8.75$\times$ fewer learnable parameters and significantly lower memory cost, while remaining fully aligned with hardware-friendly sparsity patterns. Extensive evaluations across multiple scales of the Qwen2.5 LLM family (0.5-7B parameters) demonstrate that RoI achieves competitive performance with strong memory efficiency, stability, and scalability to more aggressive $N$:$M$ sparsity patterns, offering a practical path toward efficient LLM deployment.

Related papers

• SILAGE: Memory-Efficient, Full-Gradient-Free Nonconvex Optimization for Nested Finite Sums [51.49970814177172]
  Empirical risk on datasets naturally exhibits similarities in $N=nm$ total samples.<n>We provide an analysis that avoids pessimistic convergence analysis.<n>Our results improve existing state-of-the-art regimes.
  arXiv  Detail & Related papers  (2026-06-14T14:11:07Z)
• LEAP: Learnable End-to-End Adaptive Pruning of Large Language Models [19.274512633962086]
  State-of-the-art methods for unstructured pruning are layer-wise surrogates derived from the Optimal Brain Surgeon principle.<n>We introduce LEAP, which replaces this intractable parameterization with a per-weight Bernoulli-via-Gumbel sigmoid relaxation.<n>Across five LLM families from 0.5B to 8B parameters at 50% and 60% sparsity, LEAP improves six-task average zero-shot accuracy by +2.59 points on average over ADMM.
  arXiv  Detail & Related papers  (2026-05-17T07:01:13Z)
• Deterministic Differentiable Structured Pruning for Large Language Models [37.33389749907146]
  Structured pruning reduces LLM inference cost by removing low-importance architectural components.<n>We propose Deterministic Differentiable Pruning (DDP), a mask-only optimization method.<n>Compared with prior approaches, DDP offers greater expressiveness, reduced train--test mismatch, and faster convergence.
  arXiv  Detail & Related papers  (2026-03-09T07:59:17Z)
• Aligning Distributionally Robust Optimization with Practical Deep Learning Needs [70.87757502315293]
  While traditional Learning (DL) methods treat all samples equally, a significant gap exists between DRO and current DL practices.<n>This paper aims to bridge the gap by introducing an adaptive algorithm for a modified DRO objective that can handle weight assignment groups.
  arXiv  Detail & Related papers  (2025-08-22T18:17:44Z)
• MaskPro: Linear-Space Probabilistic Learning for Strict (N:M)-Sparsity on Large Language Models [53.36415620647177]
  Semi-structured sparsity offers a promising solution by strategically retaining $N$ elements out of every $M$ weights.<n>Existing (N:M)-compatible approaches typically fall into two categories: rule-based layerwise greedy search, which suffers from considerable errors, and gradient-driven learning, which incurs prohibitive training costs.<n>We propose a novel linear-space probabilistic framework named MaskPro, which aims to learn a prior categorical distribution for every $M$ consecutive weights and subsequently leverages this distribution to generate the (N:M)-sparsity throughout an $N$-way sampling
  arXiv  Detail & Related papers  (2025-06-15T15:02:59Z)
• LESA: Learnable LLM Layer Scaling-Up [57.0510934286449]
  Training Large Language Models (LLMs) from scratch requires immense computational resources, making it prohibitively expensive.<n>Model scaling-up offers a promising solution by leveraging the parameters of smaller models to create larger ones.<n>We propose textbfLESA, a novel learnable method for depth scaling-up.
  arXiv  Detail & Related papers  (2025-02-19T14:58:48Z)
• ALoRE: Efficient Visual Adaptation via Aggregating Low Rank Experts [71.91042186338163]
  ALoRE is a novel PETL method that reuses the hypercomplex parameterized space constructed by Kronecker product to Aggregate Low Rank Experts.<n>Thanks to the artful design, ALoRE maintains negligible extra parameters and can be effortlessly merged into the frozen backbone.
  arXiv  Detail & Related papers  (2024-12-11T12:31:30Z)
• MoS: Unleashing Parameter Efficiency of Low-Rank Adaptation with Mixture of Shards [35.163843138935455]
  The rapid scaling of large language models requires more lightweight finetuning methods to reduce the explosive GPU memory overhead.<n>Our research highlights the indispensable role of differentiation in reversing the detrimental effects of pure sharing.<n>We propose Mixture of Shards (MoS), incorporating both inter-layer and intra-layer sharing schemes, and integrating four nearly cost-free differentiation strategies.
  arXiv  Detail & Related papers  (2024-10-01T07:47:03Z)
• Bypass Back-propagation: Optimization-based Structural Pruning for Large Language Models via Policy Gradient [57.9629676017527]
  We propose an optimization-based structural pruning that learns the pruning masks in a probabilistic space directly by optimizing the loss of the pruned model.<n>We achieve this by learning an underlying Bernoulli distribution to sample binary pruning masks.<n>Experiments conducted on LLaMA, LLaMA-2, LLaMA-3, Vicuna, and Mistral models demonstrate the promising performance of our method in efficiency and effectiveness.
  arXiv  Detail & Related papers  (2024-06-15T09:31:03Z)
• RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptation [30.797422827190278]
  We present a new PEFT method called Robust Adaptation (RoSA) inspired by robust principal component analysis. RoSA trains $textitlow-rank$ and $textithighly-sparse$ components on top of a set of fixed pretrained weights. We show that RoSA outperforms LoRA, pure sparse fine-tuning, and alternative hybrid methods at the same parameter budget.
  arXiv  Detail & Related papers  (2024-01-09T17:09:01Z)
• Systematic Investigation of Sparse Perturbed Sharpness-Aware Minimization Optimizer [158.2634766682187]
  Deep neural networks often suffer from poor generalization due to complex and non- unstructured loss landscapes. SharpnessAware Minimization (SAM) is a popular solution that smooths the loss by minimizing the change of landscape when adding a perturbation. In this paper, we propose Sparse SAM (SSAM), an efficient and effective training scheme that achieves perturbation by a binary mask.
  arXiv  Detail & Related papers  (2023-06-30T09:33:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.