On the Usage of Continual Learning for Out-of-Distribution Generalization in Pre-trained Language Models of Code

• URL: http://arxiv.org/abs/2305.04106v2
• Date: Tue, 22 Aug 2023 14:10:06 GMT
• Title: On the Usage of Continual Learning for Out-of-Distribution Generalization in Pre-trained Language Models of Code
• Authors: Martin Weyssow, Xin Zhou, Kisub Kim, David Lo and Houari Sahraoui
• Abstract summary: Pre-trained language models (PLMs) have become a prevalent technique in deep learning for code. We study two widely used PLM architectures on two downstream tasks, API call and API usage prediction. To address these issues, we implement five continual learning approaches, including replay-based and regularization-based methods.
• Score: 12.708117108874083
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pre-trained language models (PLMs) have become a prevalent technique in deep learning for code, utilizing a two-stage pre-training and fine-tuning procedure to acquire general knowledge about code and specialize in a variety of downstream tasks. However, the dynamic nature of software codebases poses a challenge to the effectiveness and robustness of PLMs. In particular, world-realistic scenarios potentially lead to significant differences between the distribution of the pre-training and test data, i.e., distribution shift, resulting in a degradation of the PLM's performance on downstream tasks. In this paper, we stress the need for adapting PLMs of code to software data whose distribution changes over time, a crucial problem that has been overlooked in previous works. The motivation of this work is to consider the PLM in a non-stationary environment, where fine-tuning data evolves over time according to a software evolution scenario. Specifically, we design a scenario where the model needs to learn from a stream of programs containing new, unseen APIs over time. We study two widely used PLM architectures, i.e., a GPT2 decoder and a RoBERTa encoder, on two downstream tasks, API call and API usage prediction. We demonstrate that the most commonly used fine-tuning technique from prior work is not robust enough to handle the dynamic nature of APIs, leading to the loss of previously acquired knowledge i.e., catastrophic forgetting. To address these issues, we implement five continual learning approaches, including replay-based and regularization-based methods. Our findings demonstrate that utilizing these straightforward methods effectively mitigates catastrophic forgetting in PLMs across both downstream tasks while achieving comparable or superior performance.

Related papers

• T4P: Test-Time Training of Trajectory Prediction via Masked Autoencoder and Actor-specific Token Memory [39.021321011792786]
  Trajectory prediction is a challenging problem that requires considering interactions among multiple actors. Data-driven approaches have been used to address this complex problem, but they suffer from unreliable predictions under distribution shifts during test time. We propose several online learning methods using regression loss from the ground truth of observed data. Our method surpasses the performance of existing state-of-the-art online learning methods in terms of both prediction accuracy and computational efficiency.
  arXiv  Detail & Related papers  (2024-03-15T06:47:14Z)
• Learn or Recall? Revisiting Incremental Learning with Pre-trained Language Models [21.95081572612883]
  Most assume that catastrophic forgetting is the biggest obstacle to achieving superior IL performance. We propose a frustratingly easy method called SEQ* for IL with PLMs. Results show that SEQ* has competitive or superior performance compared to state-of-the-art (SOTA) IL methods.
  arXiv  Detail & Related papers  (2023-12-13T04:14:22Z)
• PILOT: A Pre-Trained Model-Based Continual Learning Toolbox [71.63186089279218]
  This paper introduces a pre-trained model-based continual learning toolbox known as PILOT. On the one hand, PILOT implements some state-of-the-art class-incremental learning algorithms based on pre-trained models, such as L2P, DualPrompt, and CODA-Prompt. On the other hand, PILOT fits typical class-incremental learning algorithms within the context of pre-trained models to evaluate their effectiveness.
  arXiv  Detail & Related papers  (2023-09-13T17:55:11Z)
• Revisit Few-shot Intent Classification with PLMs: Direct Fine-tuning vs. Continual Pre-training [20.98770732015944]
  Few-shot intent detection involves training a deep learning model to classify utterances based on their underlying intents using only a small amount of labeled data. We show that continual pre-training may not be essential, since the overfitting problem of PLMs on this task may not be as serious as expected. To maximize the utilization of the limited available data, we propose a context augmentation method and leverage sequential self-distillation to boost performance.
  arXiv  Detail & Related papers  (2023-06-08T15:26:52Z)
• Task-Guided IRL in POMDPs that Scales [22.594913269327353]
  In inverse linear reinforcement learning (IRL), a learning agent infers a reward function encoding the underlying task using demonstrations from experts. Most IRL techniques require the computationally forward problem -- computing an optimal policy given a reward function -- in POMDPs. We develop an algorithm that reduces the information while increasing the data efficiency.
  arXiv  Detail & Related papers  (2022-12-30T21:08:57Z)
• PIVOT: Prompting for Video Continual Learning [50.80141083993668]
  We introduce PIVOT, a novel method that leverages extensive knowledge in pre-trained models from the image domain. Our experiments show that PIVOT improves state-of-the-art methods by a significant 27% on the 20-task ActivityNet setup.
  arXiv  Detail & Related papers  (2022-12-09T13:22:27Z)
• Let Offline RL Flow: Training Conservative Agents in the Latent Space of Normalizing Flows [58.762959061522736]
  offline reinforcement learning aims to train a policy on a pre-recorded and fixed dataset without any additional environment interactions. We build upon recent works on learning policies in latent action spaces and use a special form of Normalizing Flows for constructing a generative model. We evaluate our method on various locomotion and navigation tasks, demonstrating that our approach outperforms recently proposed algorithms.
  arXiv  Detail & Related papers  (2022-11-20T21:57:10Z)
• CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning [92.36705236706678]
  "CodeRL" is a new framework for program synthesis tasks through pretrained LMs and deep reinforcement learning. During inference, we introduce a new generation procedure with a critical sampling strategy. For the model backbones, we extended the encoder-decoder architecture of CodeT5 with enhanced learning objectives.
  arXiv  Detail & Related papers  (2022-07-05T02:42:15Z)
• ELLE: Efficient Lifelong Pre-training for Emerging Data [91.52652408402815]
  Current pre-trained language models (PLM) are typically trained with static data, ignoring that in real-world scenarios, streaming data of various sources may continuously grow. We propose ELLE, aiming at efficient lifelong pre-training for emerging data. ELLE consists of (1) function preserved model expansion, which flexibly expands an existing PLM's width and depth to improve the efficiency of knowledge acquisition; and (2) pre-trained domain prompts, which disentangle the versatile knowledge learned during pre-training and stimulate the proper knowledge for downstream tasks.
  arXiv  Detail & Related papers  (2022-03-12T01:53:53Z)
• Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora [31.136334214818305]
  We study a lifelong language model pretraining challenge where a PTLM is continually updated so as to adapt to emerging data. Over a domain-incremental research paper stream and a chronologically ordered tweet stream, we incrementally pretrain a PTLM with different continual learning algorithms. Our experiments show continual learning algorithms improve knowledge preservation, with logit distillation being the most effective approach.
  arXiv  Detail & Related papers  (2021-10-16T09:59:33Z)
• Learning to Continuously Optimize Wireless Resource In Episodically Dynamic Environment [55.91291559442884]
  This work develops a methodology that enables data-driven methods to continuously learn and optimize in a dynamic environment. We propose to build the notion of continual learning into the modeling process of learning wireless systems. Our design is based on a novel min-max formulation which ensures certain fairness" across different data samples.
  arXiv  Detail & Related papers  (2020-11-16T08:24:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.