SHIFT-LLM: Distribution Shift Correction in Depth-Pruned LLMs

• URL: http://arxiv.org/abs/2608.25068v1
• Date: Tue, 25 Aug 2026 19:01:58 GMT
• Title: SHIFT-LLM: Distribution Shift Correction in Depth-Pruned LLMs
• Abstract summary: We introduce SHIFT-LLM, a training-free post-pruning correction framework that inserts a Linear Residual Adapter at each pruning site.<n>Each LRA preserves the identity pathway of the original residual block and adds a lightweight affine residual correction.<n>Experiments on five model families, six layer-selection criteria, and seven zero-shot benchmarks show that SHIFT-LLM consistently recovers accuracy lost to depth pruning.
• Score: 18.494055578192015
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Depth pruning removes entire Transformer blocks to reduce the inference cost of large language models, but disrupts the hidden-state distributions expected by downstream layers, leading to significant accuracy loss. We introduce SHIFT-LLM, a training-free post-pruning correction framework that inserts a Linear Residual Adapter (LRA) at each pruning site. Each LRA preserves the identity pathway of the original residual block and adds a lightweight affine residual correction. This correction is calibrated via closed-form least-squares regression on a small held-out set, without gradient computation, to approximate the missing residual update produced by the pruned block. Together with the preserved identity pathway, the resulting LRA output approximates the hidden state produced by the original block, thereby mitigating the distributional mismatch introduced by layer removal while avoiding the expensive attention and feed-forward computations of the removed blocks. The resulting LRAs support low-rank factorization and exact merging across consecutive pruned layers for additional compression, and combine naturally with parameter-efficient fine-tuning for further recovery beyond fine-tuning the pruned model alone. Experiments on five model families, six layer-selection criteria, and seven zero-shot benchmarks show that SHIFT-LLM consistently recovers accuracy lost to depth pruning across most configurations, achieving gains up to +15.7 points on Llama-3.1-8B-Instruct while requiring only a few hundred calibration samples and no gradient computation.

Related papers

• Forward-Free LLM Depth Pruning via Weight Redundancy [0.08594140167290099]
  Depth pruning reduces large language model (LLM) inference cost by removing complete Transformer blocks.<n>We propose Weight-Redundancy Pruning (WRP), a forward-free depth-pruning method that estimates inter-layer redundancy from checkpoint weights to select blocks.<n>WRP consistently outperforms existing forward-free magnitude pruning and approaches the performance of activation-based methods.
  arXiv  Detail & Related papers  (2026-09-09T08:38:23Z)
• COEC: Calibrated Orthogonal-Equivalence Compensation for Structured Pruning of Large Language Models [13.501861808796873]
  We propose COEC (Calibrated Orthogonal-Equivalence Compensation) as a training-free compensation framework for structured pruning.<n>Experiments on the Llama-3, Llama-3.1, and Qwen2.5 model families show that COEC improves perplexity on every model and zero-shot accuracy in most settings.
  arXiv  Detail & Related papers  (2026-08-21T14:20:49Z)
• QuaSAR: Quantization Compensation via Stable Activation-Aware Rank Truncation [0.0]
  Recent training-free post-training quantization methods restore model accuracy through closed-form residual compensation.<n>We show that under the low-bit W4A4 setting, this gating mechanism fails to distinguish poorly predictable quantization error from numerical solver failure.<n>We propose a parameter-free truncated pseudoinverse solver which removes collapsed directions prior to inversion.
  arXiv  Detail & Related papers  (2026-08-14T10:00:36Z)
• SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation [64.33799467286265]
  Low-Rank Adaptation (LoRA) enables efficient adaptation of large pre-trained models to downstream tasks.<n>We show that when a full fine-tuning gradient is backpropagated to the low-rank matrices, it undergoes anisotropic scaling driven by their singular values.<n>We propose a new low-rank parameterization, SDS-LoRA, which structurally decouples singular values from the backward pass.
  arXiv  Detail & Related papers  (2026-06-15T09:27:50Z)
• TriSP: Tri-Signal Structured Pruning for Large Language Models [0.0]
  Large language models (LLMs) achieve strong performance across diverse tasks but their deployment is constrained by the memory and compute cost of their parameters.<n> Structured pruning addresses this by removing entire structures such as attention heads and Multi-Layer Perceptron (MLP) neurons to produce smaller dense models that run efficiently on standard hardware.<n>We propose TriSP (Tri-Signal Structured Pruning), an importance metric that combines weight magnitude scaled by activation norm with first-order gradient sensitivity via a geometric mean.
  arXiv  Detail & Related papers  (2026-06-09T17:02:14Z)
• Closed-Form Spectral Regularization for Multi-Task Model Merging [96.82449201305234]
  Model merging combines several independently fine-tuned experts into a single multi-task model without any training data.<n>State-of-the-art merging methods formulate merging as a layer-wise interference problem.<n>We propose SWUDI, a closed-form method that combines a soft exponential filter, which matches the gradient-flow trajectory of iterative descent.
  arXiv  Detail & Related papers  (2026-06-05T14:00:47Z)
• HARP: Hadamard-Preconditioned Adaptive Rotation Processor for Extreme LLM Quantization [48.87937677659571]
  We introduce HARP (Hadamard-preconditioned Adaptive Rotation Processor), a learnable structured two-sided processor that replaces fixed Hadamard mixing.<n>Across 2-4 bit settings on models ranging from 1B to 70B parameters, HARP improves perplexity and zero-shot accuracy over fixed RHT.<n>HarP preserves deployment efficiency, reaching 128 tok/s versus 61 tok/s for FP16.
  arXiv  Detail & Related papers  (2026-05-28T12:24:15Z)
• Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signals [83.0127582612634]
  Near-boundary Rescue (NSR) is a minimal, plug-and-play modification that retains out-of-bound tokens to recover lost signals.<n>NSR substantially improves training stability and delivers consistent gains over strong baselines such as DAPO and GSPO.
  arXiv  Detail & Related papers  (2026-05-21T16:45:31Z)
• Adaptive Signal Resuscitation: Channel-wise Post-Pruning Repair for Sparse Vision Networks [8.5133415329194]
  One-shot magnitude pruning can cause severe accuracy collapse in the high-sparsity regime.<n>We propose Adaptive Signal Resuscitation (ASR) as a training-free channel-wise repair method.<n>ASR estimates a variance-matching correction for each output channel and stabilizes it with a data-driven shrinkage rule.
  arXiv  Detail & Related papers  (2026-05-20T17:19:01Z)
• Think Before You Prune: Self-Reflective Structured Pruning for Reasoning Language Models [31.422773877490613]
  Reasoning LLMs (RLMs) deliver strong multi-step reasoning through chain-of-thought generation.<n>RLMs' large model sizes and lengthy decode-time outputs make them costly to deploy and unsuitable for resource-constrained settings.<n>We introduce RESP, a structured pruning framework that aligns pruning decisions with the model's reasoning dynamics.
  arXiv  Detail & Related papers  (2025-12-01T20:27:05Z)
• Compress to Impress: Efficient LLM Adaptation Using a Single Gradient Step on 100 Samples [57.67658635348395]
  LASER's exhaustive, per-matrix search makes it impractical for rapid deployment.<n>We show that combining these findings yields a fast and robust adaptation algorithm for downstream tasks.
  arXiv  Detail & Related papers  (2025-10-23T17:58:01Z)
• Unsupervised Conformal Inference: Bootstrapping and Alignment to Control LLM Uncertainty [49.19257648205146]
  We propose an unsupervised conformal inference framework for generation.<n>Our gates achieve close-to-nominal coverage and provide tighter, more stable thresholds than split UCP.<n>The result is a label-free, API-compatible gate for test-time filtering.
  arXiv  Detail & Related papers  (2025-09-26T23:40:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.