Mitigating Training Imbalance in LLM Fine-Tuning via Selective Parameter Merging

• URL: http://arxiv.org/abs/2410.03743v1
• Date: Tue, 1 Oct 2024 08:44:31 GMT
• Title: Mitigating Training Imbalance in LLM Fine-Tuning via Selective Parameter Merging
• Authors: Yiming Ju, Ziyi Ni, Xingrun Xing, Zhixiong Zeng, hanyu Zhao, Siqi Fan, Zheng Zhang,
• Abstract summary: Supervised fine-tuning (SFT) is crucial for adapting Large Language Models (LLMs) to specific tasks. We propose to mitigate this imbalance by merging SFT models fine-tuned with different data orders.
• Score: 11.223074654129915
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Supervised fine-tuning (SFT) is crucial for adapting Large Language Models (LLMs) to specific tasks. In this work, we demonstrate that the order of training data can lead to significant training imbalances, potentially resulting in performance degradation. Consequently, we propose to mitigate this imbalance by merging SFT models fine-tuned with different data orders, thereby enhancing the overall effectiveness of SFT. Additionally, we introduce a novel technique, "parameter-selection merging," which outperforms traditional weighted-average methods on five datasets. Further, through analysis and ablation studies, we validate the effectiveness of our method and identify the sources of performance improvements.

Related papers

• Efficient Source-Free Time-Series Adaptation via Parameter Subspace Disentanglement [0.7558576228782637]
  We propose a framework for efficient Source-Free Domain Adaptation (SFDA) Our approach introduces an improved paradigm for source-model preparation and target-side adaptation. We demonstrate that our framework is compatible with various SFDA methods and achieves significant computational efficiency.
  arXiv  Detail & Related papers  (2024-10-03T02:12:03Z)
• Enhancing Training Data Attribution for Large Language Models with Fitting Error Consideration [74.09687562334682]
  We introduce a novel training data attribution method called Debias and Denoise Attribution (DDA) Our method significantly outperforms existing approaches, achieving an averaged AUC of 91.64%. DDA exhibits strong generality and scalability across various sources and different-scale models like LLaMA2, QWEN2, and Mistral.
  arXiv  Detail & Related papers  (2024-10-02T07:14:26Z)
• Unveiling the Flaws: Exploring Imperfections in Synthetic Data and Mitigation Strategies for Large Language Models [89.88010750772413]
  Synthetic data has been proposed as a solution to address the issue of high-quality data scarcity in the training of large language models (LLMs) Our work delves into these specific flaws associated with question-answer (Q-A) pairs, a prevalent type of synthetic data, and presents a method based on unlearning techniques to mitigate these flaws. Our work has yielded key insights into the effective use of synthetic data, aiming to promote more robust and efficient LLM training.
  arXiv  Detail & Related papers  (2024-06-18T08:38:59Z)
• On Task Performance and Model Calibration with Supervised and Self-Ensembled In-Context Learning [71.44986275228747]
  In-context learning (ICL) has become an efficient approach propelled by the recent advancements in large language models (LLMs) However, both paradigms are prone to suffer from the critical problem of overconfidence (i.e., miscalibration)
  arXiv  Detail & Related papers  (2023-12-21T11:55:10Z)
• Order Matters in the Presence of Dataset Imbalance for Multilingual Learning [53.74649778447903]
  We present a simple yet effective method of pre-training on high-resource tasks, followed by fine-tuning on a mixture of high/low-resource tasks. We show its improvements in neural machine translation (NMT) and multi-lingual language modeling.
  arXiv  Detail & Related papers  (2023-12-11T05:46:57Z)
• LoBaSS: Gauging Learnability in Supervised Fine-tuning Data [64.27898739929734]
  Supervised Fine-Tuning (SFT) serves as a crucial phase in aligning Large Language Models (LLMs) to specific task prerequisites. We introduce a new dimension in SFT data selection: learnability. We present the Loss Based SFT Data Selection (LoBaSS) method, utilizing data learnability as the principal criterion for the selection SFT data.
  arXiv  Detail & Related papers  (2023-10-16T07:26:24Z)
• Scaling Laws for Sparsely-Connected Foundation Models [70.41266138010657]
  We explore the impact of parameter sparsity on the scaling behavior of Transformers trained on massive datasets. We identify the first scaling law describing the relationship between weight sparsity, number of non-zero parameters, and amount of training data.
  arXiv  Detail & Related papers  (2023-09-15T16:29:27Z)
• SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models [28.764782216513037]
  Federated Learning (FL) can benefit from distributed and private data of the FL edge clients for fine-tuning. We propose a method called SLoRA, which overcomes the key limitations of LoRA in high heterogeneous data scenarios. Our experimental results demonstrate that SLoRA achieves performance comparable to full fine-tuning.
  arXiv  Detail & Related papers  (2023-08-12T10:33:57Z)
• Rethink the Effectiveness of Text Data Augmentation: An Empirical Analysis [4.771833920251869]
  We evaluate the effectiveness of three different FT methods in conjugation with back-translation across an array of 7 diverse NLP tasks. Our findings reveal that continued pre-training on augmented data can effectively improve the FT performance of the downstream tasks. Our finding highlights the potential of DA as a powerful tool for bolstering LMs' performance.
  arXiv  Detail & Related papers  (2023-06-13T10:14:58Z)
• Empirical Analysis of the Strengths and Weaknesses of PEFT Techniques for LLMs [1.867982979635437]
  We provide a benchmark of various PEFT techniques and evaluate model performance across different data scales. Contrary to popular belief, we empirically prove that PEFT techniques converge slower than full tuning in low data scenarios. We further optimize these PEFT techniques by selectively choosing which parts of the model to train, and find that these techniques can be applied with significantly fewer parameters.
  arXiv  Detail & Related papers  (2023-04-28T17:39:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.