When pre-training hurts LoRA fine-tuning: a dynamical analysis via single-index models

• URL: http://arxiv.org/abs/2602.02855v1
• Date: Mon, 02 Feb 2026 22:02:52 GMT
• Title: When pre-training hurts LoRA fine-tuning: a dynamical analysis via single-index models
• Abstract summary: Excessive pre-training can slow down fine-tuning optimization.<n>We study this phenomenon for low-rank adaptation (LoRA) fine-tuning on single-index models trained under one-pass SGD.
• Score: 12.383124986345443
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pre-training on a source task is usually expected to facilitate fine-tuning on similar downstream problems. In this work, we mathematically show that this naive intuition is not always true: excessive pre-training can computationally slow down fine-tuning optimization. We study this phenomenon for low-rank adaptation (LoRA) fine-tuning on single-index models trained under one-pass SGD. Leveraging a summary statistics description of the fine-tuning dynamics, we precisely characterize how the convergence rate depends on the initial fine-tuning alignment and the degree of non-linearity of the target task. The key take away is that even when the pre-training and down- stream tasks are well aligned, strong pre-training can induce a prolonged search phase and hinder convergence. Our theory thus provides a unified picture of how pre-training strength and task difficulty jointly shape the dynamics and limitations of LoRA fine-tuning in a nontrivial tractable model.

Related papers

• Compute-Optimal Pretrain--Fine-tune in Ridge Gradient Descent [18.338094508559468]
  Under a fixed training budget, compute spent improving the upstream objective reduces the compute available for downstream adaptation.<n>We cast this allocation as a compute-split problem under a two-stage pretrain--fine-tune procedure with fixed total optimisation budget.<n>Our results show that the allocation depends on how pretraining directions affect fine-tuning predictions and how fine-tuning shifts are seen through downstream data geometry.
  arXiv  Detail & Related papers  (2026-09-14T19:23:59Z)
• High-Dimensional Theory of LoRA Fine-Tuning in a Solvable Attention Model [0.0]
  We introduce a solvable framework in which a single-head attention layer is first pre-trained on a data-abundant task and subsequently adapted via a rank-one LoRA update on limited data.<n>Our analysis shows that the impact of pre-training on LoRA is summarized by an effective noise term, from which we derive prescriptions for the optimal pre-training procedure.
  arXiv  Detail & Related papers  (2026-06-04T09:05:59Z)
• Convergent Stochastic Training of Attention and Understanding LoRA [40.81245220414135]
  Transformers have revolutionized machine learning and deploying attention layers in the model is increasingly standard across a myriad of applications.<n>For large models, it is common to implement Low Rank Adaptation (LoRA), whereby a factorized parameterization of them is trained, to achieve a surprisingly beneficial accuracy-size tradeoff.<n>We prove that for any mild regularization, the empirical regression loss on a attention layer and LoRA on a shallow neural net induce Poincaré inequality for the corresponding Gibbs' measure.
  arXiv  Detail & Related papers  (2026-05-08T16:22:08Z)
• (How) Learning Rates Regulate Catastrophic Overtraining [31.988459260329353]
  Supervised fine-tuning (SFT) is a common first stage of LLM post-training.<n>We investigate catastrophic forgetting in finetuning through the lens of implicit regularization of the learning rate.<n>We link forgetting to overtraining: learning rate decay increases the sharpness of the pretrained model, which in turn exacerbates catastrophic forgetting during SFT.
  arXiv  Detail & Related papers  (2026-04-15T08:53:42Z)
• Curvature-Guided LoRA: Steering in the pretrained NTK subspace [60.35296431630704]
  We introduce the prediction alignment problem, aiming to match the predictor obtained via PEFT to that of full fine-tuning at the level of outputs.<n>We show that this objective naturally leads to a curvature-aware, second-order formulation, where optimal low-rank updates correspond to a Newton-like, curvature-whitened gradient.<n>Based on this insight, we propose Curvature-Guided LoRA (CG-LoRA), which selects and scales adaptation directions using local curvature information.
  arXiv  Detail & Related papers  (2026-03-31T14:46:39Z)
• Beyond Precision: Training-Inference Mismatch is an Optimization Problem and Simple LR Scheduling Fixes It [24.70923739848818]
  We show that gradient noise and training-inference mismatch escalate in tandem as training progresses.<n>We find that the mismatch can be effectively suppressed by shrinking the update size.<n>We propose a simple yet effective solution: a specialized Learning Rate scheduler.
  arXiv  Detail & Related papers  (2026-02-02T09:00:53Z)
• The Coverage Principle: How Pre-Training Enables Post-Training [70.25788947586297]
  We study how pre-training shapes the success of the final model.<n>We uncover a mechanism that explains the power of coverage in predicting downstream performance.
  arXiv  Detail & Related papers  (2025-10-16T17:53:50Z)
• ResAD: Normalized Residual Trajectory Modeling for End-to-End Autonomous Driving [64.42138266293202]
  ResAD is a Normalized Residual Trajectory Modeling framework.<n>It reframes the learning task to predict the residual deviation from an inertial reference.<n>On the NAVSIM benchmark, ResAD achieves a state-of-the-art PDMS of 88.6 using a vanilla diffusion policy.
  arXiv  Detail & Related papers  (2025-10-09T17:59:36Z)
• Pay Attention to Small Weights [26.613296190219103]
  NanoADAM dynamically updates only the small-magnitude weights during finetuning.<n>It preserves large-magnitude weights, which are likely to encode critical features learned during pretraining.
  arXiv  Detail & Related papers  (2025-06-26T15:22:55Z)
• Enhancing Robustness of Vision-Language Models through Orthogonality Learning and Self-Regularization [77.62516752323207]
  We introduce an orthogonal fine-tuning method for efficiently fine-tuning pretrained weights and enabling enhanced robustness and generalization. A self-regularization strategy is further exploited to maintain the stability in terms of zero-shot generalization of VLMs, dubbed OrthSR. For the first time, we revisit the CLIP and CoOp with our method to effectively improve the model on few-shot image classficiation scenario.
  arXiv  Detail & Related papers  (2024-07-11T10:35:53Z)
• How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression? [92.90857135952231]
  Transformers pretrained on diverse tasks exhibit remarkable in-context learning (ICL) capabilities. We study ICL in one of its simplest setups: pretraining a linearly parameterized single-layer linear attention model for linear regression.
  arXiv  Detail & Related papers  (2023-10-12T15:01:43Z)
• The Marginal Value of Momentum for Small Learning Rate SGD [20.606430391298815]
  Momentum is known to accelerate the convergence of gradient descent in strongly convex settings without gradient noise regimes. Experiments show that momentum indeed has limited benefits for both optimization and generalization in practical training where the optimal learning rate is not very large.
  arXiv  Detail & Related papers  (2023-07-27T21:01:26Z)
• Debiased Fine-Tuning for Vision-language Models by Prompt Regularization [56.48290708901531]
  We present a new paradigm for fine-tuning large-scale vision pre-trained models on downstream task, dubbed Prompt Regularization (ProReg)<n>ProReg uses the prediction by prompting the pretrained model to regularize the fine-tuning.<n>We show the consistently strong performance of ProReg compared with conventional fine-tuning, zero-shot prompt, prompt tuning, and other state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-01-29T11:53:55Z)
• Extrapolation for Large-batch Training in Deep Learning [72.61259487233214]
  We show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer.
  arXiv  Detail & Related papers  (2020-06-10T08:22:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.