LILA: Calibration-Free Structured Pruning of Large Language Models via Latent Spectral Geometry

• URL: http://arxiv.org/abs/2609.11163v1
• Date: Thu, 10 Sep 2026 07:09:09 GMT
• Title: LILA: Calibration-Free Structured Pruning of Large Language Models via Latent Spectral Geometry
• Abstract summary: LILA scores neuron importance via the Kolmogorov--Smirnov (KS) distance between empirical singular value distributions of the full and neuron-ablated feed-forward network (FFN) weight matrix.<n>Without any fine-tuning, LILA surpasses PruneNet by 1.57pp in zero-shot accuracy on LLaMA-2-7B at 25% sparsity, and outperforms WikiText-2-d SliceGPT by up to 6.0pp across all sparsity levels.
• Score: 1.2109519547057517
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Structured pruning of large language models (LLMs) offers hardware-efficient compression, yet existing methods require calibration data, gradient computation, or large auxiliary policy networks at pruning time. LILA (\emph{Latent-Informed Layer Analysis}) scores neuron importance via the Kolmogorov--Smirnov (KS) distance between empirical singular value distributions of the full and neuron-ablated feed-forward network (FFN) weight matrix, providing a closed-form spectral rule requiring no training, calibration data, or auxiliary network. Without any fine-tuning, LILA surpasses PruneNet (45M-parameter RL policy) by 1.57~pp in zero-shot accuracy on LLaMA-2-7B at 25\% sparsity, and outperforms WikiText-2-calibrated SliceGPT by up to 6.0~pp across all sparsity levels, while preserving the original architecture. After one epoch of LoRA recovery fine-tuning, LILA achieves highly competitive performance, matching the heavily calibrated SliceGPT baseline to within a 0.48~pp margin across LLaMA-2-7B and Phi-2, despite using zero calibration data. A Neural Tangent Kernel analysis confirms a 22$\times$ reduction in functional distortion versus random pruning, providing theoretical grounding for the spectral importance criterion. Finally, extending LILA to dynamically allocate sparsity budgets via KS-scores yields state-of-the-art generative preservation at moderate compression, while uncovering fundamental single-layer architectural bottlenecks at higher compression regimes.

Related papers

• ISO: An RLVR-Native Optimization Stack [50.40395312897848]
  We operationalize spectral inheritance as Isospectral Optimization (ISO)<n>Offline, ISO-Merger combines the frame changes of shared-base specialists into a single fixed-spectrum model.<n>Online, ISO-Merger applies a chosen base, including AdamW and Muon, to the frame variables while keeping the base spectra fixed.
  arXiv  Detail & Related papers  (2026-07-21T17:51:36Z)
• TriSP: Tri-Signal Structured Pruning for Large Language Models [0.0]
  Large language models (LLMs) achieve strong performance across diverse tasks but their deployment is constrained by the memory and compute cost of their parameters.<n> Structured pruning addresses this by removing entire structures such as attention heads and Multi-Layer Perceptron (MLP) neurons to produce smaller dense models that run efficiently on standard hardware.<n>We propose TriSP (Tri-Signal Structured Pruning), an importance metric that combines weight magnitude scaled by activation norm with first-order gradient sensitivity via a geometric mean.
  arXiv  Detail & Related papers  (2026-06-09T17:02:14Z)
• Recover-LoRA for Aggressive Quantization: Reclaiming Accuracy in 2-Bit Language Models via Low-Rank Adaptation with Knowledge Distillation on Synthetic Data [3.056033670271077]
  Recover-LoRA is a data-free accuracy recovery method originally developed for general model weight corruption.<n>We show Recover-LoRA can recover accuracy lost from 2-bit quantization of the gate and up layers.<n>Our results present Recover-LoRA as a practical post-quantization accuracy recovery tool for aggressive weight compression in deployment settings.
  arXiv  Detail & Related papers  (2026-06-02T21:37:56Z)
• Convex Optimization for Alignment and Preference Learning on a Single GPU [52.997197698288936]
  Fine-tuning large language models to align with human preferences has driven the success of systems such as Gemini and ChatGPT.<n>Direct Preference Optimization (DPO) offers a simpler alternative but has limitations such as inconsistent ranking accuracy and high dependence on GPU resources.<n>We propose the Convex Optimization for Alignment and Preference Learning Algorithm (COALA), a novel lightweight strategy with strong theoretical guarantees.
  arXiv  Detail & Related papers  (2026-05-22T05:25:00Z)
• Machine Learning Models to Identify Promising Nested Antiresonance Nodeless Fiber Designs [0.0]
  We propose a machine learning framework designed to identify high-performance nested antiresonance nodeless fibers (NANFs)<n>We show that small data sets are sufficient for stable, high-accuracy performance prediction, enabling the exploration of design spaces as large as $14e6$ cases.
  arXiv  Detail & Related papers  (2026-03-02T14:13:59Z)
• Spectral Gating Networks [65.9496901693099]
  We introduce Spectral Gating Networks (SGN) to introduce frequency-rich expressivity in feed-forward networks.<n>SGN augments a standard activation pathway with a compact spectral pathway and learnable gates that allow the model to start from a stable base behavior.<n>It consistently improves accuracy-efficiency trade-offs under comparable computational budgets.
  arXiv  Detail & Related papers  (2026-02-07T20:00:49Z)
• HQP: Sensitivity-Aware Hybrid Quantization and Pruning for Ultra-Low-Latency Edge AI Inference [0.0]
  Hybrid Quantization and Pruning (HQP) framework designed to achieve synergistic model acceleration.<n>HQP framework achieves a peak performance gain of 3.12 times inference speedup and a 55 percent model size reduction.
  arXiv  Detail & Related papers  (2026-02-02T18:17:45Z)
• Boundary-Guided Policy Optimization for Memory-efficient RL of Diffusion Large Language Models [53.339700196282905]
  A key challenge in applying reinforcement learning to large language models (dLLMs) is the intractability of their likelihood functions.<n>We propose a memory-efficient RL algorithm that maximizes a specially constructed lower bound of the ELBO-based objective.<n> Experiments show that BGPO significantly outperforms previous RL algorithms for dLLMs in math problem solving, code generation, and planning tasks.
  arXiv  Detail & Related papers  (2025-10-13T17:47:50Z)
• NIRVANA: Structured pruning reimagined for large language models compression [50.651730342011014]
  We introduce NIRVANA, a novel pruning method designed to balance immediate zero-shot preservation accuracy with robust fine-tuning.<n>To further address the unique challenges posed by structured pruning, NIRVANA incorporates an adaptive sparsity allocation mechanism across layers and modules.<n>Experiments conducted on Llama3, Qwen, T5 models demonstrate that NIRVANA outperforms existing structured pruning methods under equivalent sparsity constraints.
  arXiv  Detail & Related papers  (2025-09-17T17:59:00Z)
• ESSA: Evolutionary Strategies for Scalable Alignment [8.418036456622158]
  We present ESSA, a gradient-free framework that aligns Large Language Models (LLMs) using only forward inference and black-box optimization.<n>ESSA improves the test accuracy of Qwen2.5-Math-7B by 12.6% on GSM8K and 14.8% on PRM800K, and raises the accuracy of LLaMA3.1-8B on IFEval by 22.5%.<n>In large-scale settings ESSA shows stronger scaling than gradient-based methods.
  arXiv  Detail & Related papers  (2025-07-06T16:23:07Z)
• BiLLM: Pushing the Limit of Post-Training Quantization for LLMs [53.31402059062365]
  BiLLM is a groundbreaking 1-bit post-training quantization scheme tailored for pretrained large language models. It achieves for the first time high-accuracy inference (e.g. 8.41 perplexity on LLaMA2-70B) with only 1.08-bit weights across various LLMs families.
  arXiv  Detail & Related papers  (2024-02-06T09:26:34Z)
• Compact representations of convolutional neural networks via weight pruning and quantization [63.417651529192014]
  We propose a novel storage format for convolutional neural networks (CNNs) based on source coding and leveraging both weight pruning and quantization. We achieve a reduction of space occupancy up to 0.6% on fully connected layers and 5.44% on the whole network, while performing at least as competitive as the baseline.
  arXiv  Detail & Related papers  (2021-08-28T20:39:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.