Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuning

• URL: http://arxiv.org/abs/2502.11019v1
• Date: Sun, 16 Feb 2025 07:06:17 GMT
• Title: Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuning
• Authors: Gangwei Jiang, Caigao Jiang, Zhaoyi Li, Siqiao Xue, Jun Zhou, Linqi Song, Defu Lian, Yin Wei,
• Abstract summary: Catastrophic forgetting (CF) poses a significant challenge in machine learning, where a model forgets previously learned information upon learning new tasks. Our study explores CF across various settings, discovering that model forgetting is influenced by both the specific training tasks and the models themselves. We propose a novel function vector guided training methodology, incorporating a regularization technique to stabilize the FV and forgetting.
• Score: 35.34985930216027
• License:
• Abstract: Catastrophic forgetting (CF) poses a significant challenge in machine learning, where a model forgets previously learned information upon learning new tasks. Despite the advanced capabilities of Large Language Models (LLMs), they continue to face challenges with CF during continual learning. The majority of existing research focuses on analyzing forgetting patterns through a singular training sequence, thereby overlooking the intricate effects that diverse tasks have on model behavior. Our study explores CF across various settings, discovering that model forgetting is influenced by both the specific training tasks and the models themselves. To this end, we interpret forgetting by examining the function vector (FV), a compact representation of functions in LLMs, offering a model-dependent indicator for the occurrence of CF. Through theoretical and empirical analyses, we demonstrated that CF in LLMs primarily stems from biases in function activation rather than the overwriting of task processing functions. Leveraging these insights, we propose a novel function vector guided training methodology, incorporating a regularization technique to stabilize the FV and mitigate forgetting. Empirical tests on four benchmarks confirm the effectiveness of our proposed training method, substantiating our theoretical framework concerning CF and model function dynamics. We plan to make our code publicly accessible in the near future.

Related papers

• An Analysis for Reasoning Bias of Language Models with Small Initialization [8.380004565348619]
  Large Language Models (LLMs) have revolutionized Natural Language Processing by demonstrating exceptional performance across diverse tasks. This study investigates the impact of the parameter initialization scale on the training behavior and task preferences of LLMs.
  arXiv  Detail & Related papers  (2025-02-05T15:23:26Z)
• The Inherent Limits of Pretrained LLMs: The Unexpected Convergence of Instruction Tuning and In-Context Learning Capabilities [51.594836904623534]
  We investigate whether instruction-tuned models possess fundamentally different capabilities from base models that are prompted using in-context examples. We show that the performance of instruction-tuned models is significantly correlated with the in-context performance of their base counterparts. Specifically, we extend this understanding to instruction-tuned models, suggesting that their pretraining data similarly sets a limiting boundary on the tasks they can solve.
  arXiv  Detail & Related papers  (2025-01-15T10:57:55Z)
• Theoretical Insights into Overparameterized Models in Multi-Task and Replay-Based Continual Learning [37.745896674964186]
  Multi-task learning (MTL) aims to improve the generalization performance of a model on multiple related tasks by training it simultaneously on those tasks. Continual learning (CL) involves adapting to new sequentially arriving tasks over time without forgetting the previously acquired knowledge. We develop theoretical results describing the effect of various system parameters on the model's performance in an MTL setup. Our results reveal the impact of buffer size and model capacity on the forgetting rate in a CL setup and help shed light on some of the state-of-the-art CL methods.
  arXiv  Detail & Related papers  (2024-08-29T23:22:40Z)
• Latent Causal Probing: A Formal Perspective on Probing with Causal Models of Data [3.376269351435396]
  We develop a formal perspective on probing using structural causal models (SCM) We extend a recent study of LMs in the context of a synthetic grid-world navigation task. Our techniques provide robust empirical evidence for the ability of LMs to induce the latent concepts underlying text.
  arXiv  Detail & Related papers  (2024-07-18T17:59:27Z)
• DETAIL: Task DEmonsTration Attribution for Interpretable In-context Learning [75.68193159293425]
  In-context learning (ICL) allows transformer-based language models to learn a specific task with a few "task demonstrations" without updating their parameters. We propose an influence function-based attribution technique, DETAIL, that addresses the specific characteristics of ICL. We experimentally prove the wide applicability of DETAIL by showing our attribution scores obtained on white-box models are transferable to black-box models in improving model performance.
  arXiv  Detail & Related papers  (2024-05-22T15:52:52Z)
• Learning Car-Following Behaviors Using Bayesian Matrix Normal Mixture Regression [17.828808886958736]
  Car-following (CF) behaviors are crucial for microscopic traffic simulation. Many data-driven methods, despite their robustness, operate as "black boxes" with limited interpretability. This work introduces a Bayesian Matrix Normal Mixture Regression (MNMR) model that simultaneously captures feature correlations and temporal dynamics inherent in CF behaviors.
  arXiv  Detail & Related papers  (2024-04-24T17:55:47Z)
• Learning from models beyond fine-tuning [78.20895343699658]
  Learn From Model (LFM) focuses on the research, modification, and design of foundation models (FM) based on the model interface. The study of LFM techniques can be broadly categorized into five major areas: model tuning, model distillation, model reuse, meta learning and model editing. This paper gives a comprehensive review of the current methods based on FM from the perspective of LFM.
  arXiv  Detail & Related papers  (2023-10-12T10:20:36Z)
• Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactuals [67.64770842323966]
  Causal explanations of predictions of NLP systems are essential to ensure safety and establish trust. Existing methods often fall short of explaining model predictions effectively or efficiently. We propose two approaches for counterfactual (CF) approximation.
  arXiv  Detail & Related papers  (2023-10-01T07:31:04Z)
• Iterative Forward Tuning Boosts In-Context Learning in Language Models [88.25013390669845]
  In this study, we introduce a novel two-stage framework to boost in-context learning in large language models (LLMs) Specifically, our framework delineates the ICL process into two distinct stages: Deep-Thinking and test stages. The Deep-Thinking stage incorporates a unique attention mechanism, i.e., iterative enhanced attention, which enables multiple rounds of information accumulation.
  arXiv  Detail & Related papers  (2023-05-22T13:18:17Z)
• Post Hoc Explanations of Language Models Can Improve Language Models [43.2109029463221]
  We present a novel framework, Amplifying Model Performance by Leveraging In-Context Learning with Post Hoc Explanations (AMPLIFY) We leverage post hoc explanation methods which output attribution scores (explanations) capturing the influence of each of the input features on model predictions. Our framework, AMPLIFY, leads to prediction accuracy improvements of about 10-25% over a wide range of tasks.
  arXiv  Detail & Related papers  (2023-05-19T04:46:04Z)
• Feeding What You Need by Understanding What You Learned [54.400455868448695]
  Machine Reading (MRC) reveals the ability to understand a given text passage and answer questions based on it. Existing research works in MRC rely heavily on large-size models and corpus to improve the performance evaluated by metrics such as Exact Match. We argue that a deep understanding of model capabilities and data properties can help us feed a model with appropriate training data.
  arXiv  Detail & Related papers  (2022-03-05T14:15:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.