Train Overcomplete, Deploy Compact: Scaling Recovery Capacity for Structured LLM Pruning

• URL: http://arxiv.org/abs/2609.06974v1
• Date: Mon, 07 Sep 2026 03:10:32 GMT
• Title: Train Overcomplete, Deploy Compact: Scaling Recovery Capacity for Structured LLM Pruning
• Abstract summary: OverRep is a framework for structured pruning of large language models.<n>It improves retained reasoning performance over strong recovery baselines by up to 5.5 and 8.4 points at 25% and 50% pruning, respectively.<n>Our code is available at https://github.com/mmai-laboratory/OverRep.
• Score: 6.0812616788294
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models achieve strong performance across diverse tasks, but deployment remains costly because of memory, latency, and energy demands. Structured pruning reduces these costs by removing architectural components, yet its recovery stage is often limited by a mismatch between the recovery module's representational capacity and the complexity of the removed knowledge. We call this bottleneck the capacity-knowledge asymmetry and propose OverRep, an Overcomplete Reparameterization framework for structured LLM pruning. Following the principle of "train overcomplete, deploy compact", OverRep temporarily overparameterizes the recovery module during training to absorb complex knowledge distilled from the original model. After recovery, the overcomplete re-parameterization is algebraically merged into a mathematically equivalent compact module, preserving the pruned model's inference-time architecture and computational cost. OverRep further introduces an annealed activation that enables nonlinear training dynamics while converging to a linear regime for exact algebraic merging. Across three backbone families, OverRep improves retained reasoning performance over strong recovery baselines by up to 5.5 and 8.4 points at 25% and 50% pruning, respectively, while keeping memory usage and TFLOPs comparable to existing recovery methods. Our code is available at https://github.com/mmai-laboratory/OverRep.

Related papers

• MemoryCPT: An End-to-End Agent Memory Framework for Cost-Performance Trade-off [18.54644708989368]
  Long-horizon LLM agents require memory systems that recover useful evidence from large interaction histories without passing excessive context.<n>We propose MemoryCPT, an end-to-end trainable agent memory pipeline that spans offline memory construction and online query-conditioned context generation.<n> Experiments on LoCoMo and LongMemEval show that MemoryCPT improves the cost-performance trade-off over the evaluated baselines.
  arXiv  Detail & Related papers  (2026-08-05T13:37:08Z)
• Replacement Learning: Training Neural Networks with Fewer Parameters [15.09968642484538]
  Replacement Learning (RepL) is a training-time paradigm that reduces full-depth redundancy by replacing selected blocks.<n>RepL preserves local contextual continuity while avoiding unnecessary full-layer.<n>We show that RepL reduces trainable parameters, GPU memory usage, and training time while matching or surpassing standard end-to-end training.
  arXiv  Detail & Related papers  (2026-05-19T08:34:31Z)
• Structural Pruning of Large Vision Language Models: A Comprehensive Study on Pruning Dynamics, Recovery, and Data Efficiency [65.07757274822207]
  Large Vision Language Models (LVLMs) demonstrate impressive capabilities, their substantial computational and memory requirements pose deployment challenges on resource-constrained edge devices.<n>We study a complementary route: compressing existing LVLMs by applying structured pruning to the language model backbone, followed by lightweight recovery training.
  arXiv  Detail & Related papers  (2026-04-27T12:10:44Z)
• ARES: Scalable and Practical Gradient Inversion Attack in Federated Learning through Activation Recovery [73.8181449261685]
  Federated Learning (FL) enables collaborative model training by sharing model updates instead of raw data, aiming to protect user privacy.<n>Recent studies reveal that these shared updates can inadvertently leak sensitive training data through gradient inversion attacks (GIAs)
  arXiv  Detail & Related papers  (2026-03-18T11:40:44Z)
• A Free Lunch in LLM Compression: Revisiting Retraining after Pruning [23.87950717135044]
  We study the key design choices when reconstructing or retraining the remaining weights after pruning.<n>In particular, we observe a free lunch scenario: reconstructing attention and components separately within each transformer block is nearly the most resource-efficient yet achieves the best perplexity.<n>Our findings challenge the narrative that retraining should be avoided at all costs and provide important insights into post-pruning performance recovery.
  arXiv  Detail & Related papers  (2025-10-16T08:43:09Z)
• Any Image Restoration via Efficient Spatial-Frequency Degradation Adaptation [158.37640586809187]
  Restoring any degraded image efficiently via just one model has become increasingly significant.<n>Our approach, termed AnyIR, takes a unified path that leverages inherent similarity across various degradations.<n>To fuse the degradation awareness and the contextualized attention, a spatial-frequency parallel fusion strategy is proposed.
  arXiv  Detail & Related papers  (2025-04-19T09:54:46Z)
• Re-boosting Self-Collaboration Parallel Prompt GAN for Unsupervised Image Restoration [63.37145159948982]
  Unsupervised restoration approaches based on generative adversarial networks (GANs) offer a promising solution without requiring paired datasets.<n>Yet, these GAN-based approaches struggle to surpass the performance of conventional unsupervised GAN-based frameworks.<n>We propose a self-collaboration (SC) strategy for existing restoration models.
  arXiv  Detail & Related papers  (2024-08-17T16:26:59Z)
• Reconstruct the Pruned Model without Any Retraining [23.235907813011174]
  We introduce the Linear Interpolation-based Adaptive Reconstruction (LIAR) framework, which is both efficient and effective. LIAR does not require back-propagation or retraining and is compatible with various pruning criteria and modules. Our evaluations on benchmarks such as GLUE, SQuAD, WikiText, and common sense reasoning show that LIAR enables a BERT model to maintain 98% accuracy even after removing 50% of its parameters.
  arXiv  Detail & Related papers  (2024-07-18T09:30:44Z)
• SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning [63.93193829913252]
  We propose an innovative METL strategy called SHERL for resource-limited scenarios. In the early route, intermediate outputs are consolidated via an anti-redundancy operation. In the late route, utilizing minimal late pre-trained layers could alleviate the peak demand on memory overhead.
  arXiv  Detail & Related papers  (2024-07-10T10:22:35Z)
• PERP: Rethinking the Prune-Retrain Paradigm in the Era of LLMs [22.557682089926004]
  We show that updating a small subset of parameters can suffice to recover or even enhance performance after pruning.<n>We introduce two novel LoRA variants that, unlike standard LoRA, allow merging adapters back without compromising sparsity.
  arXiv  Detail & Related papers  (2023-12-23T11:45:22Z)
• Efficient Micro-Structured Weight Unification and Pruning for Neural Network Compression [56.83861738731913]
  Deep Neural Network (DNN) models are essential for practical applications, especially for resource limited devices. Previous unstructured or structured weight pruning methods can hardly truly accelerate inference. We propose a generalized weight unification framework at a hardware compatible micro-structured level to achieve high amount of compression and acceleration.
  arXiv  Detail & Related papers  (2021-06-15T17:22:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.