Low-Rank Prehab: Preparing Neural Networks for SVD Compression

• URL: http://arxiv.org/abs/2512.01980v1
• Date: Mon, 01 Dec 2025 18:37:53 GMT
• Title: Low-Rank Prehab: Preparing Neural Networks for SVD Compression
• Abstract summary: Inspired by prehabilitation in surgery, we introduce a pre-compression fine-tuning stage, Low-Rank Prehab.<n>We show that Prehab substantially reduces the immediate accuracy drop after compression and consistently improves post-finetuning performance.<n>Our method outperforms state-of-the-art SVD-based techniques such as SVD-LLM.
• Score: 12.618746274492421
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Low-rank approximation methods such as singular value decomposition (SVD) and its variants (e.g., Fisher-weighted SVD, Activation SVD) have recently emerged as effective tools for neural network compression. In this setting, decomposition acts as a "surgical" intervention, followed by fine-tuning that serves as "rehab" to recover accuracy. Inspired by prehabilitation in surgery, we introduce a pre-compression fine-tuning stage, Low-Rank Prehab, that explicitly encourages low-rank structure in weight matrices while preserving task performance. By conditioning the model before SVD, Prehab steers weights toward spectrally compact regions of the parameter space, enabling smoother low-rank approximation and improved recovery. Experiments on large language models (LLMs) and other Transformer-based architectures, including Vision Transformers (ViTs), show that Prehab substantially reduces the immediate accuracy drop after compression and consistently improves post-finetuning performance. Across a wide range of compression ratios, our method outperforms state-of-the-art SVD-based techniques such as SVD-LLM, highlighting the importance of preparing models for compression rather than only improving the compression and recovery stages. Source code is available at https://github.com/niqretnuh/PREHAB-SVD

Related papers

• Mind the Approximation: Fisher-Weighted SVD Compression for ViTs [37.05832234320274]
  FACTS is a structured Fisher Approximation tailored to Vision Transformers (ViTs)<n>It enforces token-local aggregation while preserving within-token activation-gradient dependence.<n>It consistently improves accuracy-efficiency trade-offs without requiring finetuning.
  arXiv  Detail & Related papers  (2026-09-07T07:50:50Z)
• Importance-Aware Low-Rank Distillation of Diffusion Transformers [15.737230365885907]
  Diffusion Transformers (DiTs) have emerged as a dominant architecture for high-quality text-to-image generation.<n>We introduce SVDtrunc, a two-step block-level compression scheme.<n>We find that SVDtrunc produces smooth degradation even under substantial global compression.
  arXiv  Detail & Related papers  (2026-09-04T02:25:11Z)
• SVD-Surgeon: Optimal Singular-Value Surgery for Large Language Model Compression [47.182582580115195]
  We introduce SVD-Surgeon, a training-free method that brings the Optimal Brain Surgeon (OBS) framework to the singular-value basis.<n>As it operates directly on the singular-value factorization, SVD-Surgeon can be layered on top of existing SVD compressors.
  arXiv  Detail & Related papers  (2026-06-22T16:33:16Z)
• Low-Rank Compression of Pretrained Models via Randomized Subspace Iteration [2.1016374925364616]
  Low-rank decomposition based on the singular value decomposition (SVD) provides a principled approach for model reduction.<n> Randomized alternatives such as randomized SVD (RSVD) improve efficiency, yet they suffer from poor approximation quality when the singular value spectrum decays slowly.<n>We propose randomized subspace (RSI) as a more effective alternative.
  arXiv  Detail & Related papers  (2026-04-03T02:47:03Z)
• Vanishing Contributions: A Unified Approach to Smoothly Transition Neural Models into Compressed Form [2.6986106861637165]
  Vanishing Contributions (VCON) is a general approach for smoothly transitioning neural models into compressed form.<n>We evaluate VCON across computer vision and natural language processing benchmarks, in combination with multiple compression strategies.
  arXiv  Detail & Related papers  (2025-10-09T15:17:13Z)
• OmniSAT: Compact Action Token, Faster Auto Regression [70.70037017501357]
  We introduce an Omni Swift Action Tokenizer, which learns a compact, transferable action representation.<n>The resulting discrete tokenization shortens the training sequence by 6.8$times$, and lowers the target entropy.
  arXiv  Detail & Related papers  (2025-10-08T03:55:24Z)
• ResSVD: Residual Compensated SVD for Large Language Model Compression [12.539815070352116]
  Large language models (LLMs) have demonstrated impressive capabilities in a wide range of downstream natural language processing tasks.<n>We propose ResSVD, a new post-training SVD-based LLM compression method.<n>We leverage the residual matrix generated during the truncation process to reduce truncation loss.
  arXiv  Detail & Related papers  (2025-05-26T15:14:54Z)
• Optimizing Singular Spectrum for Large Language Model Compression [95.7621116637755]
  We introduce SoCo, a novel compression framework that learns to rescale the decomposed components of SVD in a data-driven manner.<n>Thanks to the learnable singular spectrum, SoCo adaptively prunes components according to the sparsified importance scores.<n> Experimental evaluations across multiple LLMs and benchmarks demonstrate that SoCo surpasses the state-of-the-art methods in model compression.
  arXiv  Detail & Related papers  (2025-02-20T23:18:39Z)
• AdaSVD: Adaptive Singular Value Decomposition for Large Language Models [75.1196637934987]
  Singular Value Decomposition (SVD) has emerged as a promising compression technique for large language models (LLMs)<n>Existing SVD-based methods often struggle to effectively mitigate the errors introduced by SVD truncation.<n>We propose AdaSVD, an adaptive SVD-based LLM compression approach.
  arXiv  Detail & Related papers  (2025-02-03T14:34:37Z)
• Convolutional Neural Network Compression Based on Low-Rank Decomposition [3.3295360710329738]
  This paper proposes a model compression method that integrates Variational Bayesian Matrix Factorization. VBMF is employed to estimate the rank of the weight tensor at each layer. Experimental results show that for both high and low compression ratios, our compression model exhibits advanced performance.
  arXiv  Detail & Related papers  (2024-08-29T06:40:34Z)
• Numerical Optimizations for Weighted Low-rank Estimation on Language Model [73.12941276331316]
  Singular value decomposition (SVD) is one of the most popular compression methods that approximates a target matrix with smaller matrices. Standard SVD treats the parameters within the matrix with equal importance, which is a simple but unrealistic assumption. We show that our method can perform better than current SOTA methods in neural-based language models.
  arXiv  Detail & Related papers  (2022-11-02T00:58:02Z)
• Language model compression with weighted low-rank factorization [73.61874728240568]
  We introduce Fisher information to weigh the importance of parameters affecting the model prediction. We find that our resulting task accuracy is much closer to the original model's performance. Our method can directly compress a task-specific model while achieving better performance than other compact model strategies.
  arXiv  Detail & Related papers  (2022-06-30T21:57:07Z)
• Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification [53.50708351813565]
  We propose SVD training, the first method to explicitly achieve low-rank DNNs during training without applying SVD on every step. We empirically show that SVD training can significantly reduce the rank of DNN layers and achieve higher reduction on computation load under the same accuracy.
  arXiv  Detail & Related papers  (2020-04-20T02:40:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.