Improved Supervised Fine-Tuning for Large Language Models to Mitigate Catastrophic Forgetting

• URL: http://arxiv.org/abs/2506.09428v2
• Date: Sat, 28 Jun 2025 02:26:03 GMT
• Title: Improved Supervised Fine-Tuning for Large Language Models to Mitigate Catastrophic Forgetting
• Authors: Fei Ding, Baiqiao Wang,
• Abstract summary: Supervised Fine-Tuning (SFT) is a critical step for enhancing the instruction-following capabilities of Large Language Models (LLMs)<n>SFT often leads to a degradation of the model's general abilities, a phenomenon known as catastrophic forgetting.<n>We propose a novel and cost-effective SFT method that effectively mitigates catastrophic forgetting without requiring access to the original SFT data.
• Score: 1.5595148909011116
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Supervised Fine-Tuning (SFT) is a critical step for enhancing the instruction-following capabilities of Large Language Models (LLMs) and adapting them to specialized domains. However, SFT often leads to a degradation of the model's general abilities, a phenomenon known as catastrophic forgetting. This problem is exacerbated when third-party practitioners fine-tune open-source models, as the original SFT data is typically not available. To address this challenge, we propose a novel and cost-effective SFT method that effectively mitigates catastrophic forgetting without requiring access to the original SFT data. Our approach first reconstructs the likely instruction distribution of the base model. It then employs a multi-model generation and filtering pipeline to synthesize a high-quality general-purpose dataset. This synthetic dataset is mixed with new, domain-specific data for fine-tuning. Experimental results show that our method not only preserves the model's capabilities in general domains but also improves task-specific performance, outperforming baselines that use publicly available SFT datasets.

Related papers

• On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification [50.30835290642069]
  We present a simple yet theoretically motivated improvement to Supervised Fine-Tuning (SFT) for the Large Language Model (LLM)<n>We reveal that standard SFT gradients implicitly encode a problematic reward structure that may severely restrict the generalization capabilities of model.<n>We propose Dynamic Fine-Tuning (DFT), stabilizing gradient updates for each token by dynamically rescaling the objective function with the probability of this token.
  arXiv  Detail & Related papers  (2025-08-07T17:59:04Z)
• Differentiation-Based Extraction of Proprietary Data from Fine-Tuned LLMs [13.835835256858653]
  This paper studies the critical research problem of extracting data fromSupervised Fine-Tuning (SFT) datasets.<n>We develop a novel extraction method specifically designed for SFT models, called Differentiated Data Extraction (DDE)<n>Our results show that DDE consistently outperforms existing extraction baselines in all attack settings.
  arXiv  Detail & Related papers  (2025-06-20T02:43:36Z)
• Minifinetuning: Low-Data Generation Domain Adaptation through Corrective Self-Distillation [48.52666201053625]
  Minifinetuning reduces the effects of overfitting-induced degeneralization in low-data settings.<n>MFT demonstrates 2-10x more favourable specialization-to-degeneralization ratios than standard finetuning.
  arXiv  Detail & Related papers  (2025-05-30T01:54:12Z)
• Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data [73.04828796123581]
  Supervised fine-tuning (SFT) has become a crucial step for aligning pretrained large language models (LLMs)<n>We introduce Discriminative Fine-Tuning (DFT), an improved variant of SFT, which mitigates the burden of collecting human-labeled preference data.<n>Our contributions include: (i) a discriminative probabilistic framework for fine-tuning LLMs by explicitly modeling the discriminative likelihood of an answer among all possible outputs given an input; (ii) efficient algorithms to optimize this discriminative likelihood; and (iii) extensive experiments demonstrating DFT's effectiveness
  arXiv  Detail & Related papers  (2025-02-25T22:38:55Z)
• PASER: Post-Training Data Selection for Efficient Pruned Large Language Model Recovery [11.20326903218271]
  Post-training techniques such as instruction tuning are commonly employed to recover model performance.<n>However, some irrelevant instructions may also introduce negative effects to model capacity recovery.<n>We propose textbfPost-training dtextbfAta textbfSelection method for textbfEfficient pruned large language model textbfRecovery (textbfPASER)
  arXiv  Detail & Related papers  (2025-02-18T07:11:08Z)
• Balancing the Budget: Understanding Trade-offs Between Supervised and Preference-Based Finetuning [18.381178799923514]
  Post-training of Large Language Models often involves a pipeline of Supervised Finetuning (SFT) followed by Preference Finetuning (PFT)<n>We study how to optimally allocate a fixed training data budget between the two stages.
  arXiv  Detail & Related papers  (2025-02-16T21:57:35Z)
• Preference-Oriented Supervised Fine-Tuning: Favoring Target Model Over Aligned Large Language Models [12.500777267361102]
  We introduce a novel textbfpreference-textbforiented supervised textbffine-textbftuning approach, namely PoFT.<n>The intuition is to boost SFT by imposing a particular preference: textitfavoring the target model over aligned LLMs on the same SFT data.<n>PoFT achieves stable and consistent improvements over the SFT baselines across different training datasets and base models.
  arXiv  Detail & Related papers  (2024-12-17T12:49:14Z)
• Mitigating Training Imbalance in LLM Fine-Tuning via Selective Parameter Merging [11.223074654129915]
  Supervised fine-tuning (SFT) is crucial for adapting Large Language Models (LLMs) to specific tasks. We propose to mitigate this imbalance by merging SFT models fine-tuned with different data orders.
  arXiv  Detail & Related papers  (2024-10-01T08:44:31Z)
• ASFT: Aligned Supervised Fine-Tuning through Absolute Likelihood [14.512464277772194]
  Aligned Supervised Fine-Tuning (ASFT) is an effective approach that better aligns Large Language Models with pair-wise datasets. ASFT mitigates the issue where the DPO loss function decreases the probability of generating human-dispreferred data. Extensive experiments demonstrate that ASFT is an effective alignment approach, consistently outperforming existing methods.
  arXiv  Detail & Related papers  (2024-09-14T11:39:13Z)
• Functional Graphical Models: Structure Enables Offline Data-Driven Optimization [111.28605744661638]
  We show how structure can enable sample-efficient data-driven optimization. We also present a data-driven optimization algorithm that infers the FGM structure itself.
  arXiv  Detail & Related papers  (2024-01-08T22:33:14Z)
• Federated Learning with Projected Trajectory Regularization [65.6266768678291]
  Federated learning enables joint training of machine learning models from distributed clients without sharing their local data. One key challenge in federated learning is to handle non-identically distributed data across the clients. We propose a novel federated learning framework with projected trajectory regularization (FedPTR) for tackling the data issue.
  arXiv  Detail & Related papers  (2023-12-22T02:12:08Z)
• DavIR: Data Selection via Implicit Reward for Large Language Models [62.59514469369608]
  DavIR is a model-based data selection method for post-training Large Language Models.<n>We show that 6% of Alpaca dataset selected with DavIR can steer both the LLaMA and Gemma model family to produce superior performance compared to the same models trained on the full 52K dataset.
  arXiv  Detail & Related papers  (2023-10-16T07:26:24Z)
• SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models [28.764782216513037]
  Federated Learning (FL) can benefit from distributed and private data of the FL edge clients for fine-tuning. We propose a method called SLoRA, which overcomes the key limitations of LoRA in high heterogeneous data scenarios. Our experimental results demonstrate that SLoRA achieves performance comparable to full fine-tuning.
  arXiv  Detail & Related papers  (2023-08-12T10:33:57Z)
• Hyperparameter-free Continuous Learning for Domain Classification in Natural Language Understanding [60.226644697970116]
  Domain classification is the fundamental task in natural language understanding (NLU) Most existing continual learning approaches suffer from low accuracy and performance fluctuation. We propose a hyper parameter-free continual learning model for text data that can stably produce high performance under various environments.
  arXiv  Detail & Related papers  (2022-01-05T02:46:16Z)
• Extrapolation for Large-batch Training in Deep Learning [72.61259487233214]
  We show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer.
  arXiv  Detail & Related papers  (2020-06-10T08:22:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.