Fine-Tuning Without Forgetting via Loss-Adaptive Learning Rates

• URL: http://arxiv.org/abs/2605.20005v1
• Date: Tue, 19 May 2026 15:36:52 GMT
• Title: Fine-Tuning Without Forgetting via Loss-Adaptive Learning Rates
• Abstract summary: Fine-tuning large language models on new data improves task performance but degrades capabilities learned during pretraining.<n>Existing methods mitigate this by modifying the fine-tuning objective to suppress high-loss tokens or sequences.<n>We introduce FINCH, a loss-adaptive learning-rate schedule that reduces forgetting by 93% on average while matching the task performance of standard fine-tuning.
• Score: 10.06646055401747
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Fine-tuning large language models on new data improves task performance but degrades capabilities learned during pretraining, a phenomenon known as catastrophic forgetting. Existing methods mitigate this by modifying the fine-tuning objective to suppress high-loss tokens or sequences, but these tokens are essential for learning new tasks, especially those with poor pretraining coverage. In such settings, hard tokens should still contribute to learning, so forgetting must be controlled without suppressing them. We identify a simple mechanism for doing so: per-step forgetting is bounded by the product of the learning rate and the square root of the current training loss. This suggests that high-loss batches are especially prone to inducing forgetting. Motivated by this observation, we introduce FINCH, a loss-adaptive learning-rate schedule that reduces the learning rate on high-loss batches and increases it as the model converges, while leaving the fine-tuning objective unchanged. Across knowledge acquisition, science, and low-resource language adaptation benchmarks, FINCH reduces forgetting by 93% on average while matching the task performance of standard fine-tuning. On Qwen3-4B knowledge acquisition, FINCH cuts TruthfulQA degradation by 5x and reverses HaluEval degradation, while better preserving confidence calibration. Overall, our results show that learning-rate schedules are an effective tool to shape model behavior during fine-tuning, beyond just target-task optimization.

Related papers

• Mitigating Early Training Collapse in CTR Models [0.0]
  Deep neural models for click-through rate prediction often exhibit a sharp decline in validation performance immediately after the first training epoch.<n>In this study, we analyze this behavior using large-scale industrial datasets and evaluate practical mitigation strategies.
  arXiv  Detail & Related papers  (2026-06-20T04:34:39Z)
• Double Preconditioning (DoPr): Optimization for Test-Time Performance, not Validation Loss [26.33868416147844]
  We introduce a new optimization paradigm called double-preconditioning (DoPr)<n>DoPr combines gradient-wise preconditioning, as in Adam and Muon, with activation-wise preconditioning (AP)<n>We show that the addition of AP yields a drop-in intervention for increasing downstream model performance across a range of test-time settings.
  arXiv  Detail & Related papers  (2026-06-04T17:22:58Z)
• Entropy-Guided Token Dropout: Training Autoregressive Language Models with Limited Domain Data [89.96277093034547]
  We introduce EntroDrop, an entropy-guided token dropout method that functions as structured data regularization.<n>We show that EntroDrop consistently outperforms standard regularization baselines and maintains robust performance throughout extended multi-epoch training.
  arXiv  Detail & Related papers  (2025-12-29T12:35:51Z)
• ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining [53.893792844055106]
  Large language model pretraining is compute-intensive, yet many tokens contribute marginally to learning, resulting in inefficiency.<n>We introduce Selective Efficient Language Modeling, a risk-aware algorithm that improves training efficiency and distributional robustness by performing online token-level batch selection.<n> Experiments on GPT-2 pretraining show that ESLM significantly reduces training FLOPs while maintaining or improving both perplexity and downstream performance compared to baselines.
  arXiv  Detail & Related papers  (2025-05-26T12:23:26Z)
• Quiet Feature Learning in Algorithmic Tasks [1.9249287163937978]
  We train Transformer-based language models on ten foundational algorithmic tasks.<n>We observe pronounced phase transitions in their loss curves that deviate from established power-law scaling trends.<n>Probing the models' internal representations reveals that quiet features are learned prior to any decrease in task loss.
  arXiv  Detail & Related papers  (2025-05-06T22:18:50Z)
• Sculpting Subspaces: Constrained Full Fine-Tuning in LLMs for Continual Learning [19.27175827358111]
  Continual learning in large language models (LLMs) is prone to catastrophic forgetting, where adapting to new tasks significantly degrades performance on previously learned ones.<n>We propose a novel continual full fine-tuning approach leveraging adaptive singular value decomposition (SVD)<n>We evaluate our approach extensively on standard continual learning benchmarks using both encoder-decoder (T5-Large) and decoder-only (LLaMA-2 7B) models.
  arXiv  Detail & Related papers  (2025-04-09T17:59:42Z)
• Fine Tuning without Catastrophic Forgetting via Selective Low Rank Adaptation [13.084333776247743]
  Fine-tuning can reduce robustness to distribution shifts, impacting out-of-distribution (OOD) performance.<n>We propose a parameter-efficient fine-tuning (PEFT) method, using an indicator function to selectively activate Low-Rank Adaptation (LoRA) blocks.<n>We demonstrate that effective fine-tuning can be achieved with as few as 5% of active blocks, substantially improving efficiency.
  arXiv  Detail & Related papers  (2025-01-26T03:22:22Z)
• Enhancing Robustness of Vision-Language Models through Orthogonality Learning and Self-Regularization [77.62516752323207]
  We introduce an orthogonal fine-tuning method for efficiently fine-tuning pretrained weights and enabling enhanced robustness and generalization. A self-regularization strategy is further exploited to maintain the stability in terms of zero-shot generalization of VLMs, dubbed OrthSR. For the first time, we revisit the CLIP and CoOp with our method to effectively improve the model on few-shot image classficiation scenario.
  arXiv  Detail & Related papers  (2024-07-11T10:35:53Z)
• Stepping on the Edge: Curvature Aware Learning Rate Tuners [24.95412499942206]
  Curvature information is the largest eigenvalue of the loss Hessian, known as the sharpness. Recent work has shown that curvature information undergoes complex dynamics during training. We analyze the closed-loop feedback effect between learning rate tuning and curvature.
  arXiv  Detail & Related papers  (2024-07-08T17:56:00Z)
• Low-rank finetuning for LLMs: A fairness perspective [54.13240282850982]
  Low-rank approximation techniques have become the de facto standard for fine-tuning Large Language Models. This paper investigates the effectiveness of these methods in capturing the shift of fine-tuning datasets from the initial pre-trained data distribution. We show that low-rank fine-tuning inadvertently preserves undesirable biases and toxic behaviors.
  arXiv  Detail & Related papers  (2024-05-28T20:43:53Z)
• Towards Robust Continual Learning with Bayesian Adaptive Moment Regularization [51.34904967046097]
  Continual learning seeks to overcome the challenge of catastrophic forgetting, where a model forgets previously learnt information. We introduce a novel prior-based method that better constrains parameter growth, reducing catastrophic forgetting. Results show that BAdam achieves state-of-the-art performance for prior-based methods on challenging single-headed class-incremental experiments.
  arXiv  Detail & Related papers  (2023-09-15T17:10:51Z)
• Extrapolation for Large-batch Training in Deep Learning [72.61259487233214]
  We show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer.
  arXiv  Detail & Related papers  (2020-06-10T08:22:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.