HFT: Half Fine-Tuning for Large Language Models

• URL: http://arxiv.org/abs/2404.18466v1
• Date: Mon, 29 Apr 2024 07:07:58 GMT
• Title: HFT: Half Fine-Tuning for Large Language Models
• Authors: Tingfeng Hui, Zhenyu Zhang, Shuohuan Wang, Weiran Xu, Yu Sun, Hua Wu,
• Abstract summary: Large language models (LLMs) with one or more fine-tuning phases have become a necessary step to unlock various capabilities. In this paper, we find that by regularly resetting partial parameters, LLMs can restore some of the original knowledge. We introduce Half Fine-Tuning (HFT) for LLMs, as a substitute for full fine-tuning (FFT), to mitigate the forgetting issues.
• Score: 42.60438623804577
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) with one or more fine-tuning phases have become a necessary step to unlock various capabilities, enabling LLMs to follow natural language instructions or align with human preferences. However, it carries the risk of catastrophic forgetting during sequential training, the parametric knowledge or the ability learned in previous stages may be overwhelmed by incoming training data. In this paper, we find that by regularly resetting partial parameters, LLMs can restore some of the original knowledge. Inspired by this, we introduce Half Fine-Tuning (HFT) for LLMs, as a substitute for full fine-tuning (FFT), to mitigate the forgetting issues, where half of the parameters are selected to learn new tasks while the other half are frozen to remain previous knowledge. We provide a feasibility analysis from the perspective of optimization and interpret the parameter selection operation as a regularization term. Without changing the model architecture, HFT could be seamlessly integrated into existing fine-tuning frameworks. Extensive experiments and analysis on supervised fine-tuning, direct preference optimization, and continual learning consistently demonstrate the effectiveness, robustness, and efficiency of HFT. Compared with FFT, HFT not only significantly alleviates the forgetting problem, but also achieves the best performance in a series of downstream benchmarks, with an approximately 30% reduction in training time.

Related papers

• SFTMix: Elevating Language Model Instruction Tuning with Mixup Recipe [30.03925858123481]
  We propose SFTMix, a novel recipe that elevates instruction-tuning performance beyond the conventional NTP paradigm. Based on training dynamics, we argue that examples with different confidence levels should play distinct roles during the instruction-tuning process. This approach enables SFTMix to significantly outperform NTP across a wide range of instruction-following and healthcare domain-specific SFT tasks.
  arXiv  Detail & Related papers  (2024-10-07T17:52:21Z)
• PAFT: A Parallel Training Paradigm for Effective LLM Fine-Tuning [17.73193523921637]
  Large language models (LLMs) have shown remarkable abilities in diverse natural language processing (NLP) tasks. LLMs generally undergo supervised fine-tuning (SFT) followed by preference alignment to be usable in downstream applications. This paper introduces PAFT, a new PArallel training paradigm for effective LLM Fine-Tuning.
  arXiv  Detail & Related papers  (2024-06-25T20:11:37Z)
• Gradient-Mask Tuning Elevates the Upper Limits of LLM Performance [51.36243421001282]
  Gradient-Mask Tuning (GMT) is a method that selectively updates parameters during training based on their gradient information. Our empirical results across various tasks demonstrate that GMT not only outperforms traditional fine-tuning methods but also elevates the upper limits of LLM performance.
  arXiv  Detail & Related papers  (2024-06-21T17:42:52Z)
• Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Process [26.196705232699884]
  We introduce Intuitive Fine-Tuning (IFT) to integrate SFT and Preference Optimization into a single process. IFT performs comparably or even superiorly to sequential recipes of SFT and some typical Preference Optimization methods. An explainable Frozen Lake game further validates the effectiveness of IFT for getting competitive policy.
  arXiv  Detail & Related papers  (2024-05-20T08:23:28Z)
• An Emulator for Fine-Tuning Large Language Models using Small Language Models [91.02498576056057]
  We introduce emulated fine-tuning (EFT), a principled and practical method for sampling from a distribution that approximates the result of pre-training and fine-tuning at different scales. We show that EFT enables test-time adjustment of competing behavioral traits like helpfulness and harmlessness without additional training. Finally, a special case of emulated fine-tuning, which we call LM up-scaling, avoids resource-intensive fine-tuning of large pre-trained models by ensembling them with small fine-tuned models.
  arXiv  Detail & Related papers  (2023-10-19T17:57:16Z)
• FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning [70.38817963253034]
  This paper first discusses these challenges of federated fine-tuning LLMs, and introduces our package FS-LLM as a main contribution. We provide comprehensive federated parameter-efficient fine-tuning algorithm implementations and versatile programming interfaces for future extension in FL scenarios. We conduct extensive experiments to validate the effectiveness of FS-LLM and benchmark advanced LLMs with state-of-the-art parameter-efficient fine-tuning algorithms in FL settings.
  arXiv  Detail & Related papers  (2023-09-01T09:40:36Z)
• Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning [126.84770886628833]
  Existing finetuning methods either tune all parameters of the pretrained model (full finetuning) or only tune the last linear layer (linear probing) We propose a new parameter-efficient finetuning method termed as SSF, representing that researchers only need to Scale and Shift the deep Features extracted by a pre-trained model to catch up with the performance full finetuning.
  arXiv  Detail & Related papers  (2022-10-17T08:14:49Z)
• Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning [81.3514358542452]
  Few-shot in-context learning (ICL) incurs substantial computational, memory, and storage costs because it involves processing all of the training examples every time a prediction is made. parameter-efficient fine-tuning offers an alternative paradigm where a small set of parameters are trained to enable a model to perform the new task. In this paper, we rigorously compare few-shot ICL and parameter-efficient fine-tuning and demonstrate that the latter offers better accuracy as well as dramatically lower computational costs.
  arXiv  Detail & Related papers  (2022-05-11T17:10:41Z)
• A Fast and Efficient Conditional Learning for Tunable Trade-Off between Accuracy and Robustness [11.35810118757863]
  Existing models that achieve state-of-the-art (SOTA) performance on both clean and adversarially-perturbed images rely on convolution operations conditioned with feature-wise linear modulation (FiLM) layers. We present a fast learnable once-for-all adversarial training (FLOAT) algorithm, which instead of the existing FiLM-based conditioning, presents a unique weight conditioned learning that requires no additional layer. In particular, we add scaled noise to the weight tensors that enables a trade-off between clean and adversarial performance.
  arXiv  Detail & Related papers  (2022-03-28T19:25:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.