LEAP: Learnable End-to-End Adaptive Pruning of Large Language Models

• URL: http://arxiv.org/abs/2605.17289v1
• Date: Sun, 17 May 2026 07:01:13 GMT
• Title: LEAP: Learnable End-to-End Adaptive Pruning of Large Language Models
• Abstract summary: State-of-the-art methods for unstructured pruning are layer-wise surrogates derived from the Optimal Brain Surgeon principle.<n>We introduce LEAP, which replaces this intractable parameterization with a per-weight Bernoulli-via-Gumbel sigmoid relaxation.<n>Across five LLM families from 0.5B to 8B parameters at 50% and 60% sparsity, LEAP improves six-task average zero-shot accuracy by +2.59 points on average over ADMM.
• Score: 19.274512633962086
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Unstructured sparsity is now natively accelerated by recent GPU kernels and dataflow hardware, shifting the bottleneck from inference execution to the pruning algorithm. State-of-the-art methods for unstructured LLM pruning are layer-wise surrogates derived from the Optimal Brain Surgeon principle, and they sacrifice end-to-end accuracy, especially under aggressive sparsity. End-to-end alternatives such as MaskLLM and PATCH show that learnable masks can close this gap, but their categorical-over-patterns parameterization scales with the number of valid masks per row and does not port to the unstructured setting. We introduce LEAP, which replaces this intractable parameterization with a per-weight Bernoulli-via-Gumbel- sigmoid relaxation that makes end-to-end unstructured mask learning tractable. Across five LLM families from 0.5B to 8B parameters at 50% and 60% sparsity, LEAP improves six-task average zero-shot accuracy by +2.59 points on average over ADMM, the best layer-wise baseline in our sweep.

Related papers

• Reservoir of Importance: Learning Semi-Structured Sparsity with Differentiable Subset Sampling [6.232269207752904]
  Semi-structured $N$:$M$ sparsity has emerged as a practical direction for accelerating large language models (LLMs)<n>We propose Reservoir of Importance (RoI), a lightweight semi-structured pruning framework that learns sparsity masks through differentiable subset sampling.<n>RoI achieves competitive performance with strong memory efficiency, stability, and scalability to more aggressive $N$:$M$ sparsity patterns.
  arXiv  Detail & Related papers  (2026-08-24T09:54:24Z)
• Lookahead Path Likelihood Optimization for Diffusion LLMs [31.01208893976334]
  We introduce path log-likelihood (Path LL), a trajectory-conditioned objective that strongly correlates with downstream accuracy and enables principled selection of unmasking paths.<n>To optimize Path LL at inference time, we propose POKE, an efficient value estimator that predicts the expected future Path LL of a partial decoding trajectory.<n>We then integrate this lookahead signal into POKE-SMC, a Sequential Monte Carlo-based search framework for dynamically identifying optimal unmasking paths.
  arXiv  Detail & Related papers  (2026-02-03T13:12:41Z)
• SparseSwaps: Tractable LLM Pruning Mask Refinement at Scale [22.25809500403244]
  We propose a tractable and simple 1-swap algorithm that warms starts from any pruning mask and runs efficiently at LLM scale.<n>We demonstrate that our approach reduces per-layer pruning error by up to 60% over Wanda (Sun et al., 2023) and consistently improves perplexity and zero-shot accuracy across state-of-the-art GPT architectures.
  arXiv  Detail & Related papers  (2025-12-11T18:47:48Z)
• Lookahead Unmasking Elicits Accurate Decoding in Diffusion Language Models [51.12873073612084]
  Masked Diffusion Models (MDMs) as language models generate by iteratively unmasking tokens, yet their performance depends on the inference time order of unmasking.<n>We propose Lookahead Unmasking (LookUM), which addresses these concerns by reformulating sampling as path selection over all possible unmasking orders.<n>LookUM requires only two to three paths to achieve peak performance, demonstrating remarkably efficient path selection.
  arXiv  Detail & Related papers  (2025-11-04T02:37:37Z)
• Adaptive Pruning for Large Language Models with Structural Importance Awareness [66.2690963378878]
  Large language models (LLMs) have significantly improved language understanding and generation capabilities.<n>LLMs are difficult to deploy on resource-constrained edge devices due to their high computational and storage resource demands.<n>We propose structurally-aware adaptive pruning (SAAP) to significantly reduce the computational and memory costs while maintaining model performance.
  arXiv  Detail & Related papers  (2024-12-19T18:08:04Z)
• MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models [87.64417894918506]
  This work introduces MaskLLM, a learnable pruning method that establishes Semi-structured (or N:M'') Sparsity in Large Language Models.<n>MaskLLM explicitly models N:M patterns as a learnable distribution through Gumbel Softmax sampling.
  arXiv  Detail & Related papers  (2024-09-26T02:37:41Z)
• Bypass Back-propagation: Optimization-based Structural Pruning for Large Language Models via Policy Gradient [57.9629676017527]
  We propose an optimization-based structural pruning that learns the pruning masks in a probabilistic space directly by optimizing the loss of the pruned model.<n>We achieve this by learning an underlying Bernoulli distribution to sample binary pruning masks.<n>Experiments conducted on LLaMA, LLaMA-2, LLaMA-3, Vicuna, and Mistral models demonstrate the promising performance of our method in efficiency and effectiveness.
  arXiv  Detail & Related papers  (2024-06-15T09:31:03Z)
• Filter Pruning for Efficient CNNs via Knowledge-driven Differential Filter Sampler [103.97487121678276]
  Filter pruning simultaneously accelerates the computation and reduces the memory overhead of CNNs. We propose a novel Knowledge-driven Differential Filter Sampler(KDFS) with Masked Filter Modeling(MFM) framework for filter pruning.
  arXiv  Detail & Related papers  (2023-07-01T02:28:41Z)
• Non-Parametric Adaptive Network Pruning [125.4414216272874]
  We introduce non-parametric modeling to simplify the algorithm design. Inspired by the face recognition community, we use a message passing algorithm to obtain an adaptive number of exemplars. EPruner breaks the dependency on the training data in determining the "important" filters.
  arXiv  Detail & Related papers  (2021-01-20T06:18:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.