Forward-Free LLM Depth Pruning via Weight Redundancy

• URL: http://arxiv.org/abs/2609.09883v2
• Date: Mon, 14 Sep 2026 21:03:38 GMT
• Title: Forward-Free LLM Depth Pruning via Weight Redundancy
• Abstract summary: Depth pruning reduces large language model (LLM) inference cost by removing complete Transformer blocks.<n>We propose Weight-Redundancy Pruning (WRP), a forward-free depth-pruning method that estimates inter-layer redundancy from checkpoint weights to select blocks.<n>WRP consistently outperforms existing forward-free magnitude pruning and approaches the performance of activation-based methods.
• Score: 0.08594140167290099
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Depth pruning reduces large language model (LLM) inference cost by removing complete Transformer blocks. Activation-based methods collect hidden states through forward passes on calibration data, while existing forward-free methods score each Transformer block separately without measuring similarity between blocks. We propose Weight-Redundancy Pruning (WRP), a forward-free depth-pruning method that estimates inter-layer redundancy from checkpoint weights to select blocks without calibration data or model forward passes. WRP compares attention output and MLP down-projection weights across layers and combines their pairwise similarities with relative projection-scale information. The resulting all-pairs similarity matrix guides layer grouping and block selection. Across multiple pruning settings, model families, and downstream tasks, WRP consistently outperforms existing forward-free magnitude pruning and approaches the performance of activation-based methods.

Related papers

• SHIFT-LLM: Distribution Shift Correction in Depth-Pruned LLMs [18.494055578192015]
  We introduce SHIFT-LLM, a training-free post-pruning correction framework that inserts a Linear Residual Adapter at each pruning site.<n>Each LRA preserves the identity pathway of the original residual block and adds a lightweight affine residual correction.<n>Experiments on five model families, six layer-selection criteria, and seven zero-shot benchmarks show that SHIFT-LLM consistently recovers accuracy lost to depth pruning.
  arXiv  Detail & Related papers  (2026-08-25T19:01:58Z)
• Dual Attention Residuals [34.183227979638595]
  We propose Dual Attention Residuals, which brings multi-stream interaction into historical retrieval.<n>For each target stream, DAR computes depth weights from normalized states in the opposite stream and applies them to values from the target stream's own history.<n>DAR consistently improves validation loss over standard residual Transformers and Attention Residuals.
  arXiv  Detail & Related papers  (2026-07-21T05:42:55Z)
• Exploring the Design Space of Reward Backpropagation for Flow Matching [47.80328464705813]
  FlowBP is a unified surrogate-trajectory framework that treats the backward trajectory itself as the design object.<n>We instantiate three variants: FlowBP-Sparse, FlowBP-Bridge, and FlowBP-Lagrange.<n>All three bound memory by the active-set size and limit gradient chaining to at most one Jacobian factor.
  arXiv  Detail & Related papers  (2026-06-09T16:36:54Z)
• Attention Residuals [38.59138244826294]
  Residual connections with PreNorm are standard in modern LLMs, yet they accumulate all layer outputs with fixed unit weights.<n>We propose Attention Residuals (AttnRes), which replaces this fixed accumulation with softmax attention over preceding layer outputs.
  arXiv  Detail & Related papers  (2026-03-16T09:32:21Z)
• Diffusion Controller: Framework, Algorithms and Parameterization [54.82539154511621]
  We present a unified control-theoretic view that casts reverse diffusion sampling as state-only control within (generalized) linearly-solvable Markov Decision Processes.<n>Under this framework, control acts by reweighting the pretrained reverse-time transition kernels, balancing terminal objectives against an $f$-divergence cost.<n>Experiments on Stable Diffusion v1.4 show consistent gains in preference-alignment win rates and improved quality-efficiency trade-offs.
  arXiv  Detail & Related papers  (2026-03-07T01:49:59Z)
• Distilling to Hybrid Attention Models via KL-Guided Layer Selection [66.06591032073744]
  This paper describes a simple and efficient recipe for layer selection that uses layer importance scores derived from a small amount of training on generic text data.<n>We find that this approach is more effective than existing approaches for layer selection, including approaches that uniformly interleave linear attentions based on a fixed ratio.
  arXiv  Detail & Related papers  (2025-12-23T18:12:22Z)
• MLPMoE: Zero-Shot Architectural Metamorphosis of Dense LLM MLPs into Static Mixture-of-Experts [0.0]
  Large Language Models (LLMs) are predominantly deployed as dense transformers, where every parameter in every feed-forward block is activated for every token.<n>Recent upcycling methods such as MoEfication, CMoE, ToMoE, and MoORE reveal that much of the useful computation lives in sparse, semi-modular substructures inside dense feed-forward networks.<n>This paper introducesMoE (MLP-Experts), a training-free transformation that restructures the dense in transformer blocks into a static, high-cardinality mixture of experts.
  arXiv  Detail & Related papers  (2025-11-26T06:14:26Z)
• MultiPruner: Balanced Structure Removal in Foundation Models [1.8434042562191815]
  Recently, state-of-the-art approaches for pruning large pre-trained models (LPMs) have demonstrated that the training-free removal of non-critical residual blocks in Transformers is viable for reducing model size.<n>We extend BlockPruner and propose MultiPruner, a pruning approach that surpasses recent training-free pruning methods by adopting a multidimensional, iterative, fine-grained pruning strategy.
  arXiv  Detail & Related papers  (2025-01-17T04:24:31Z)
• Semi-Supervised Coupled Thin-Plate Spline Model for Rotation Correction and Beyond [84.56978780892783]
  We propose CoupledTPS, which iteratively couples multiple TPS with limited control points into a more flexible and powerful transformation. In light of the laborious annotation cost, we develop a semi-supervised learning scheme to improve warping quality by exploiting unlabeled data. Experiments demonstrate the superiority and universality of CoupledTPS over the existing state-of-the-art solutions for rotation correction.
  arXiv  Detail & Related papers  (2024-01-24T13:03:28Z)
• CBQ: Cross-Block Quantization for Large Language Models [66.82132832702895]
  Post-training quantization (PTQ) has played a key role in compressing large language models (LLMs) with ultra-low costs.<n>We propose CBQ, a cross-block reconstruction-based PTQ method for LLMs.<n> CBQ employs a cross-block dependency using a reconstruction scheme, establishing long-range dependencies across multiple blocks to minimize error accumulation.
  arXiv  Detail & Related papers  (2023-12-13T07:56:27Z)
• Attentional-Biased Stochastic Gradient Descent [74.49926199036481]
  We present a provable method (named ABSGD) for addressing the data imbalance or label noise problem in deep learning. Our method is a simple modification to momentum SGD where we assign an individual importance weight to each sample in the mini-batch. ABSGD is flexible enough to combine with other robust losses without any additional cost.
  arXiv  Detail & Related papers  (2020-12-13T03:41:52Z)
• Progressively Guided Alternate Refinement Network for RGB-D Salient Object Detection [63.18846475183332]
  We aim to develop an efficient and compact deep network for RGB-D salient object detection. We propose a progressively guided alternate refinement network to refine it. Our model outperforms existing state-of-the-art approaches by a large margin.
  arXiv  Detail & Related papers  (2020-08-17T02:55:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.