Investigating the Pre-Training Dynamics of In-Context Learning: Task Recognition vs. Task Learning

• URL: http://arxiv.org/abs/2406.14022v1
• Date: Thu, 20 Jun 2024 06:37:47 GMT
• Title: Investigating the Pre-Training Dynamics of In-Context Learning: Task Recognition vs. Task Learning
• Authors: Xiaolei Wang, Xinyu Tang, Wayne Xin Zhao, Ji-Rong Wen,
• Abstract summary: In-context learning (ICL) is potentially attributed to two major abilities: task recognition (TR) and task learning (TL) We take the first step by examining the pre-training dynamics of the emergence of ICL. We propose a simple yet effective method to better integrate these two abilities for ICL at inference time.
• Score: 99.05401042153214
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The emergence of in-context learning (ICL) is potentially attributed to two major abilities: task recognition (TR) for recognizing the task from demonstrations and utilizing pre-trained priors, and task learning (TL) for learning from demonstrations. However, relationships between the two abilities and how such relationships affect the emergence of ICL is unclear. In this paper, we take the first step by examining the pre-training dynamics of the emergence of ICL. With carefully designed metrics, we find that these two abilities are, in fact, competitive during pre-training. Moreover, we observe a strong negative correlation between the competition and ICL performance. Further analysis of common pre-training factors (i.e., model size, dataset size, and data curriculum) demonstrates possible ways to manage the competition. Based on these insights, we propose a simple yet effective method to better integrate these two abilities for ICL at inference time. Through adaptive ensemble learning, the performance of ICL can be significantly boosted, enabling two small models to outperform a larger one with more than twice the parameters. The code is available at https://github.com/RUCAIBox/Competitive-ICL.

Related papers

• Can In-context Learning Really Generalize to Out-of-distribution Tasks? [36.11431280689549]
  We investigate the mechanism of in-context learning (ICL) on out-of-distribution (OOD) tasks that were not encountered during training. We reveal that Transformers may struggle to learn OOD task functions through ICL.
  arXiv  Detail & Related papers  (2024-10-13T02:10:26Z)
• ICLEval: Evaluating In-Context Learning Ability of Large Language Models [68.7494310749199]
  In-Context Learning (ICL) is a critical capability of Large Language Models (LLMs) as it empowers them to comprehend and reason across interconnected inputs. Existing evaluation frameworks primarily focus on language abilities and knowledge, often overlooking the assessment of ICL ability. We introduce the ICLEval benchmark to evaluate the ICL abilities of LLMs, which encompasses two key sub-abilities: exact copying and rule learning.
  arXiv  Detail & Related papers  (2024-06-21T08:06:10Z)
• In-context Contrastive Learning for Event Causality Identification [26.132189768472067]
  Event Causality Identification aims at determining the existence of a causal relation between two events. Recent prompt learning-based approaches have shown promising improvements on the ECI task. This paper proposes an In-Context Contrastive Learning model that utilizes contrastive learning to enhance the effectiveness of both positive and negative demonstrations.
  arXiv  Detail & Related papers  (2024-05-17T03:32:15Z)
• The Strong Pull of Prior Knowledge in Large Language Models and Its Impact on Emotion Recognition [74.04775677110179]
  In-context Learning (ICL) has emerged as a powerful paradigm for performing natural language tasks with Large Language Models (LLM) We show that LLMs have strong yet inconsistent priors in emotion recognition that ossify their predictions. Our results suggest that caution is needed when using ICL with larger LLMs for affect-centered tasks outside their pre-training domain.
  arXiv  Detail & Related papers  (2024-03-25T19:07:32Z)
• Let's Learn Step by Step: Enhancing In-Context Learning Ability with Curriculum Learning [9.660673938961416]
  Demonstration ordering is an important strategy for in-context learning (ICL) We propose a simple but effective demonstration ordering method for ICL, named the few-shot In-Context Curriculum Learning (ICCL)
  arXiv  Detail & Related papers  (2024-02-16T14:55:33Z)
• Understanding In-Context Learning via Supportive Pretraining Data [55.648777340129364]
  In-context learning (ICL) improves language models' performance on a variety of NLP tasks by simply demonstrating a handful of examples at inference time. It is not well understood why ICL ability emerges, as the model has never been specifically trained on such demonstrations. Our work takes a first step towards understanding ICL via analyzing instance-level pretraining data.
  arXiv  Detail & Related papers  (2023-06-26T22:14:04Z)
• Decoupled Adversarial Contrastive Learning for Self-supervised Adversarial Robustness [69.39073806630583]
  Adversarial training (AT) for robust representation learning and self-supervised learning (SSL) for unsupervised representation learning are two active research fields. We propose a two-stage framework termed Decoupled Adversarial Contrastive Learning (DeACL)
  arXiv  Detail & Related papers  (2022-07-22T06:30:44Z)
• Robust Pre-Training by Adversarial Contrastive Learning [120.33706897927391]
  Recent work has shown that, when integrated with adversarial training, self-supervised pre-training can lead to state-of-the-art robustness. We improve robustness-aware self-supervised pre-training by learning representations consistent under both data augmentations and adversarial perturbations.
  arXiv  Detail & Related papers  (2020-10-26T04:44:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.