Locality-Aware Redundancy Pruning for LLM Depth Compression

• URL: http://arxiv.org/abs/2605.27786v1
• Date: Wed, 27 May 2026 00:09:57 GMT
• Title: Locality-Aware Redundancy Pruning for LLM Depth Compression
• Abstract summary: We propose a training-free one-shot depth pruning framework guided by representation locality.<n>We show that inter-layer redundancy can be either localized or globally distributed depending on the architecture.<n>Experiments across diverse LLM families show improvements in both perplexity and downstream task accuracy.
• Score: 6.052194197353533
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models are known to contain representational redundancy across network depth, making depth pruning an effective approach for improving inference efficiency. Existing one-shot pruning methods rely on local layer importance or fixed redundancy assumptions across architectures. We propose Locality-Aware Redundancy Pruning (LoRP), a training-free one-shot depth pruning framework guided by representation locality. We show that inter-layer redundancy can be either localized or globally distributed depending on the LLM architecture. To characterize this phenomenon, we introduce Representation Locality Score (RLS), derived from global inter-layer hidden-state similarity. Using a small calibration set, LoRP computes pairwise layer similarity, clusters layers by representational similarity, and allocates pruning according to residual intra-cluster redundancy. Experiments across diverse LLM families show improvements in both perplexity and downstream task accuracy.

Related papers

• SAND: Spatially Adaptive Network Depth for Fast Sampling of Neural Implicit Surfaces [91.71563221166417]
  Implicit neural representations are powerful for geometric modeling, but their practical use is often limited by the high computational cost.<n>We propose an efficient neural implicit geometry representation framework with spatially adaptive network depth.
  arXiv  Detail & Related papers  (2026-04-15T14:14:17Z)
• Progressive Localisation in Localist LLMs [0.0]
  This paper demonstrates that progressive localization represents the optimal architecture for creating interpretable large language models (LLMs)<n>We investigate whether interpretability constraints can be aligned with natural semantic structure while being applied strategically across network depth.<n>We show that progressive semantic localization, combining semantic block with steep adaptive locality schedules, achieves near-baseline language modeling performance while providing interpretable attention patterns.
  arXiv  Detail & Related papers  (2025-11-23T09:49:13Z)
• Mixture of Ranks with Degradation-Aware Routing for One-Step Real-World Image Super-Resolution [76.66229730098759]
  In real-world image super-resolution (Real-ISR), existing approaches mainly rely on fine-tuning pre-trained diffusion models.<n>We propose a Mixture-of-Ranks (MoR) architecture for single-step image super-resolution.<n>We introduce a fine-grained expert partitioning strategy that treats each rank in LoRA as an independent expert.
  arXiv  Detail & Related papers  (2025-11-20T04:11:44Z)
• Feedback Alignment Meets Low-Rank Manifolds: A Structured Recipe for Local Learning [7.034739490820967]
  Training deep neural networks (DNNs) with backpropagation (BP) achieves state-of-the-art accuracy but requires global error propagation and full parameterization.<n>Direct Feedback Alignment (DFA) enables local, parallelizable updates with lower memory requirements.<n>We propose a structured local learning framework that operates directly on low-rank manifold.
  arXiv  Detail & Related papers  (2025-10-29T15:03:46Z)
• Regularizing Subspace Redundancy of Low-Rank Adaptation [54.473090597164834]
  We propose ReSoRA, a method that explicitly models redundancy between mapping subspaces and adaptively Regularizes Subspace redundancy of Low-Rank Adaptation.<n>Our proposed method consistently facilitates existing state-of-the-art PETL methods across various backbones and datasets in vision-language retrieval and standard visual classification benchmarks.<n>As a training supervision, ReSoRA can be seamlessly integrated into existing approaches in a plug-and-play manner, with no additional inference costs.
  arXiv  Detail & Related papers  (2025-07-28T11:52:56Z)
• Attention-Guided Multi-Scale Local Reconstruction for Point Clouds via Masked Autoencoder Self-Supervised Learning [9.390627399366833]
  We introduce PointAMaLR, a novel self-supervised learning framework for point cloud processing.<n>PointAMaLR implements hierarchical reconstruction across multiple local regions.<n>Experiments on benchmark datasets demonstrate PointAMaLR's superior accuracy and quality in both classification and reconstruction tasks.
  arXiv  Detail & Related papers  (2025-07-05T16:17:49Z)
• Stochastic Layer-wise Learning: Scalable and Efficient Alternative to Backpropagation [1.0285749562751982]
  Backpropagation underpins modern deep learning, yet its reliance on global synchronization limits scalability and incurs high memory costs.<n>In contrast, fully local learning rules are more efficient but often struggle to maintain the cross-layer coordination needed for coherent global learning.<n>We introduce Layer-wise Learning (SLL), a layer-wise training algorithm that decomposes the global objective into coordinated layer-local updates.
  arXiv  Detail & Related papers  (2025-05-08T12:32:29Z)
• Compressing Image-to-Image Translation GANs Using Local Density Structures on Their Learned Manifold [69.33930972652594]
  Generative Adversarial Networks (GANs) have shown remarkable success in modeling complex data distributions for image-to-image translation. Existing GAN compression methods mainly rely on knowledge distillation or convolutional classifiers' pruning techniques. We propose a new approach by explicitly encouraging the pruned model to preserve the density structure of the original parameter-heavy model on its learned manifold. Our experiments on image translation GAN models, Pix2Pix and CycleGAN, with various benchmark datasets and architectures demonstrate our method's effectiveness.
  arXiv  Detail & Related papers  (2023-12-22T15:43:12Z)
• Dual-constrained Deep Semi-Supervised Coupled Factorization Network with Enriched Prior [80.5637175255349]
  We propose a new enriched prior based Dual-constrained Deep Semi-Supervised Coupled Factorization Network, called DS2CF-Net. To ex-tract hidden deep features, DS2CF-Net is modeled as a deep-structure and geometrical structure-constrained neural network. Our network can obtain state-of-the-art performance for representation learning and clustering.
  arXiv  Detail & Related papers  (2020-09-08T13:10:21Z)
• Dense Residual Network: Enhancing Global Dense Feature Flow for Character Recognition [75.4027660840568]
  This paper explores how to enhance the local and global dense feature flow by exploiting hierarchical features fully from all the convolution layers. Technically, we propose an efficient and effective CNN framework, i.e., Fast Dense Residual Network (FDRN) for text recognition.
  arXiv  Detail & Related papers  (2020-01-23T06:55:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.