Natural Language Fine-Tuning

• URL: http://arxiv.org/abs/2412.20382v1
• Date: Sun, 29 Dec 2024 07:02:45 GMT
• Title: Natural Language Fine-Tuning
• Authors: Jia Liu, Yue Wang, Zhiqi Lin, Min Chen, Yixue Hao, Long Hu,
• Abstract summary: We introduce Natural Language Fine-Tuning (NLFT), which utilizes natural language for fine-tuning for the first time. Since linguistic information is effectively utilized in NLFT, our proposed method significantly reduces training costs. It markedly enhances training efficiency, comprehensively outperforming reinforcement fine-tuning algorithms in accuracy, time-saving, and resource conservation.
• Score: 13.143016409660484
• License:
• Abstract: Large language model fine-tuning techniques typically depend on extensive labeled data, external guidance, and feedback, such as human alignment, scalar rewards, and demonstration. However, in practical application, the scarcity of specific knowledge poses unprecedented challenges to existing fine-tuning techniques. In this paper, focusing on fine-tuning tasks in specific domains with limited data, we introduce Natural Language Fine-Tuning (NLFT), which utilizes natural language for fine-tuning for the first time. By leveraging the strong language comprehension capability of the target LM, NLFT attaches the guidance of natural language to the token-level outputs. Then, saliency tokens are identified with calculated probabilities. Since linguistic information is effectively utilized in NLFT, our proposed method significantly reduces training costs. It markedly enhances training efficiency, comprehensively outperforming reinforcement fine-tuning algorithms in accuracy, time-saving, and resource conservation. Additionally, on the macro level, NLFT can be viewed as a token-level fine-grained optimization of SFT, thereby efficiently replacing the SFT process without the need for warm-up (as opposed to ReFT requiring multiple rounds of warm-up with SFT). Compared to SFT, NLFT does not increase the algorithmic complexity, maintaining O(n). Extensive experiments on the GSM8K dataset demonstrate that NLFT, with only 50 data instances, achieves an accuracy increase that exceeds SFT by 219%. Compared to ReFT, the time complexity and space complexity of NLFT are reduced by 78.27% and 92.24%, respectively. The superior technique of NLFT is paving the way for the deployment of various innovative LLM fine-tuning applications when resources are limited at network edges. Our code has been released at https://github.com/Julia-LiuJ/NLFT.

Related papers

• Optimizing Language Models for Grammatical Acceptability: A Comparative Study of Fine-Tuning Techniques [0.0]
  This study explores the fine-tuning (FT) of the Open Pre-trained Transformer (OPT-125M) for grammatical tasks using the CoLA dataset.
  arXiv  Detail & Related papers  (2025-01-14T05:41:09Z)
• Preference-Oriented Supervised Fine-Tuning: Favoring Target Model Over Aligned Large Language Models [12.500777267361102]
  We introduce a novel textbfpreference-textbforiented supervised textbffine-textbftuning approach, namely PoFT. The intuition is to boost SFT by imposing a particular preference: textitfavoring the target model over aligned LLMs on the same SFT data. PoFT achieves stable and consistent improvements over the SFT baselines across different training datasets and base models.
  arXiv  Detail & Related papers  (2024-12-17T12:49:14Z)
• UFT: Unifying Fine-Tuning of SFT and RLHF/DPO/UNA through a Generalized Implicit Reward Function [18.54945183526789]
  We introduce Unified Fine-Tuning (UFT), which integrates SFT and alignment into a single training stage. Our experimental results demonstrate that UFT outperforms SFT on instruction-tuning data alone. When combining instruction-tuning data with alignment data, UFT effectively prevents catastrophic forgetting.
  arXiv  Detail & Related papers  (2024-10-28T18:34:25Z)
• Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Process [26.196705232699884]
  We introduce Intuitive Fine-Tuning (IFT) to integrate SFT and Preference Optimization into a single process. IFT performs comparably or even superiorly to sequential recipes of SFT and some typical Preference Optimization methods. An explainable Frozen Lake game further validates the effectiveness of IFT for getting competitive policy.
  arXiv  Detail & Related papers  (2024-05-20T08:23:28Z)
• Prefix Text as a Yarn: Eliciting Non-English Alignment in Foundation Language Model [50.339632513018934]
  supervised fine-tuning (SFT) has been a straightforward approach for tailoring the output of foundation large language model (LLM) to specific preferences. We critically examine this hypothesis within the scope of cross-lingual generation tasks. We introduce a novel training-free alignment method named PreTTY, which employs minimal task-related prior tokens.
  arXiv  Detail & Related papers  (2024-04-25T17:19:36Z)
• Learning to Compress Prompt in Natural Language Formats [54.06967020905763]
  Large language models (LLMs) are great at processing multiple natural language processing tasks. LLMs are constrained by inferior performance with long context, slow inference speed, and the high cost of computing the results. This work aims to compress lengthy prompts in the form of natural language with LLM transferability.
  arXiv  Detail & Related papers  (2024-02-28T20:41:21Z)
• LaFFi: Leveraging Hybrid Natural Language Feedback for Fine-tuning Language Models [14.087415157225715]
  Fine-tuning Large Language Models (LLMs) adapts a trained model to specific downstream tasks. Supervised Fine-Tuning (SFT) is a common approach, where an LLM is trained to produce desired answers. This paper introduces an alternative to SFT called Natural Language Feedback for Finetuning LLMs (LaFFi)
  arXiv  Detail & Related papers  (2023-12-31T21:18:16Z)
• Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes [53.4856038354195]
  Pre-trained large language models (LLMs) need fine-tuning to improve their responsiveness to natural language instructions. FedKSeed employs zeroth-order optimization with a finite set of random seeds. It significantly reduces transmission requirements between the server and clients to just a few random seeds.
  arXiv  Detail & Related papers  (2023-12-11T13:03:21Z)
• Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimization [71.87335804334616]
  Federated learning (FL) is a promising paradigm to enable collaborative model training with decentralized data. The training process of Large Language Models (LLMs) generally incurs the update of significant parameters. This paper proposes an efficient partial prompt tuning approach to improve performance and efficiency simultaneously.
  arXiv  Detail & Related papers  (2023-10-23T16:37:59Z)
• AsySQN: Faster Vertical Federated Learning Algorithms with Better Computation Resource Utilization [159.75564904944707]
  We propose an asynchronous quasi-Newton (AsySQN) framework for vertical federated learning (VFL) The proposed algorithms make descent steps scaled by approximate without calculating the inverse Hessian matrix explicitly. We show that the adopted asynchronous computation can make better use of the computation resource.
  arXiv  Detail & Related papers  (2021-09-26T07:56:10Z)
• Dynamic Attention-based Communication-Efficient Federated Learning [85.18941440826309]
  Federated learning (FL) offers a solution to train a global machine learning model. FL suffers performance degradation when client data distribution is non-IID. We propose a new adaptive training algorithm $textttAdaFL$ to combat this degradation.
  arXiv  Detail & Related papers  (2021-08-12T14:18:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.