Learning or Self-aligning? Rethinking Instruction Fine-tuning

• URL: http://arxiv.org/abs/2402.18243v3
• Date: Sun, 11 Aug 2024 17:15:06 GMT
• Title: Learning or Self-aligning? Rethinking Instruction Fine-tuning
• Authors: Mengjie Ren, Boxi Cao, Hongyu Lin, Cao Liu, Xianpei Han, Ke Zeng, Guanglu Wan, Xunliang Cai, Le Sun,
• Abstract summary: We design a knowledge intervention framework to decouple the potential underlying factors of IFT. Experiments reveal that attempting to learn additional world knowledge through IFT often struggles to yield positive impacts. Maintaining internal knowledge consistency before and after IFT is a critical factor for achieving successful IFT.
• Score: 45.62941531473104
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Instruction Fine-tuning~(IFT) is a critical phase in building large language models~(LLMs). Previous works mainly focus on the IFT's role in the transfer of behavioral norms and the learning of additional world knowledge. However, the understanding of the underlying mechanisms of IFT remains significantly limited. In this paper, we design a knowledge intervention framework to decouple the potential underlying factors of IFT, thereby enabling individual analysis of different factors. Surprisingly, our experiments reveal that attempting to learn additional world knowledge through IFT often struggles to yield positive impacts and can even lead to markedly negative effects. Further, we discover that maintaining internal knowledge consistency before and after IFT is a critical factor for achieving successful IFT. Our findings reveal the underlying mechanisms of IFT and provide robust support for some very recent and potential future works.

Related papers

• Accurate Forgetting for Heterogeneous Federated Continual Learning [89.08735771893608]
  We propose a new concept accurate forgetting (AF) and develop a novel generative-replay methodMethodwhich selectively utilizes previous knowledge in federated networks. We employ a probabilistic framework based on a normalizing flow model to quantify the credibility of previous knowledge.
  arXiv  Detail & Related papers  (2025-02-20T02:35:17Z)
• Navigating the Helpfulness-Truthfulness Trade-Off with Uncertainty-Aware Instruction Fine-Tuning [79.48839334040197]
  Instruction Fine-tuning (IFT) can enhance the helpfulness of Large Language Models (LLMs) IFT steers LLMs to generate responses with long-tail knowledge that is not well covered during pre-training, leading to more informative but less truthful answers when generalizing to unseen tasks. We propose $textbfUNIT$, a novel IFT paradigm to address this trade-off.
  arXiv  Detail & Related papers  (2025-02-17T16:10:30Z)
• Fisher-Guided Selective Forgetting: Mitigating The Primacy Bias in Deep Reinforcement Learning [2.532202013576547]
  Deep Reinforcement Learning (DRL) systems often tend to overfit to early experiences, a phenomenon known as the primacy bias (PB) This paper presents a comprehensive investigation of PB through the lens of the Fisher Information Matrix (FIM)
  arXiv  Detail & Related papers  (2025-02-02T13:54:47Z)
• Instruction Tuning for Large Language Models: A Survey [52.86322823501338]
  We make a systematic review of the literature, including the general methodology of supervised fine-tuning (SFT) We also review the potential pitfalls of SFT along with criticism against it, along with efforts pointing out current deficiencies of existing strategies.
  arXiv  Detail & Related papers  (2023-08-21T15:35:16Z)
• CARE: Certifiably Robust Learning with Reasoning via Variational Inference [26.210129662748862]
  We propose a certifiably robust learning with reasoning pipeline (CARE) CARE achieves significantly higher certified robustness compared with the state-of-the-art baselines. We additionally conducted different ablation studies to demonstrate the empirical robustness of CARE and the effectiveness of different knowledge integration.
  arXiv  Detail & Related papers  (2022-09-12T07:15:52Z)
• Rethinking Importance Weighting for Transfer Learning [71.81262398144946]
  Key assumption in supervised learning is that training and test data follow the same probability distribution. As real-world machine learning tasks are becoming increasingly complex, novel approaches are explored to cope with such challenges.
  arXiv  Detail & Related papers  (2021-12-19T14:35:25Z)
• Confounder Identification-free Causal Visual Feature Learning [84.28462256571822]
  We propose a novel Confounder Identification-free Causal Visual Feature Learning (CICF) method, which obviates the need for identifying confounders. CICF models the interventions among different samples based on front-door criterion, and then approximates the global-scope intervening effect upon the instance-level interventions. We uncover the relation between CICF and the popular meta-learning strategy MAML, and provide an interpretation of why MAML works from the theoretical perspective.
  arXiv  Detail & Related papers  (2021-11-26T10:57:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.