Are Emergent Abilities in Large Language Models just In-Context Learning?

• URL: http://arxiv.org/abs/2309.01809v2
• Date: Mon, 15 Jul 2024 12:21:56 GMT
• Title: Are Emergent Abilities in Large Language Models just In-Context Learning?
• Authors: Sheng Lu, Irina Bigoulaeva, Rachneet Sachdeva, Harish Tayyar Madabushi, Iryna Gurevych,
• Abstract summary: We present a novel theory that explains emergent abilities, taking into account their potential confounding factors. Our findings suggest that purported emergent abilities are not truly emergent, but result from a combination of in-context learning, model memory, and linguistic knowledge.
• Score: 46.561464069450444
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Large language models, comprising billions of parameters and pre-trained on extensive web-scale corpora, have been claimed to acquire certain capabilities without having been specifically trained on them. These capabilities, referred to as "emergent abilities," have been a driving force in discussions regarding the potentials and risks of language models. A key challenge in evaluating emergent abilities is that they are confounded by model competencies that arise through alternative prompting techniques, including in-context learning, which is the ability of models to complete a task based on a few examples. We present a novel theory that explains emergent abilities, taking into account their potential confounding factors, and rigorously substantiate this theory through over 1000 experiments. Our findings suggest that purported emergent abilities are not truly emergent, but result from a combination of in-context learning, model memory, and linguistic knowledge. Our work is a foundational step in explaining language model performance, providing a template for their efficient use and clarifying the paradox of their ability to excel in some instances while faltering in others. Thus, we demonstrate that their capabilities should not be overestimated.

Related papers

• Can Language Models Learn to Skip Steps? [59.84848399905409]
  We study the ability to skip steps in reasoning. Unlike humans, who may skip steps to enhance efficiency or to reduce cognitive load, models do not possess such motivations. Our work presents the first exploration into human-like step-skipping ability.
  arXiv  Detail & Related papers  (2024-11-04T07:10:24Z)
• Large Language Models are Limited in Out-of-Context Knowledge Reasoning [65.72847298578071]
  Large Language Models (LLMs) possess extensive knowledge and strong capabilities in performing in-context reasoning. This paper focuses on a significant aspect of out-of-context reasoning: Out-of-Context Knowledge Reasoning (OCKR), which is to combine multiple knowledge to infer new knowledge.
  arXiv  Detail & Related papers  (2024-06-11T15:58:59Z)
• Auxiliary task demands mask the capabilities of smaller language models [2.938889003635811]
  We show that evaluation methods with greater task demands yield lower performance than evaluations with reduced demands. Our results illustrate that LM performance should not be interpreted as a direct indication of intelligence.
  arXiv  Detail & Related papers  (2024-04-03T02:56:52Z)
• Physics of Language Models: Part 3.2, Knowledge Manipulation [51.68385617116854]
  This paper investigates four fundamental knowledge manipulation tasks. We show that language models excel in knowledge retrieval but struggle even in the simplest classification or comparison tasks. Our findings also apply to modern pretrained language models such as GPT-4.
  arXiv  Detail & Related papers  (2023-09-25T17:50:41Z)
• Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuning [56.03057119008865]
  We show that scaling diffusion language models can effectively make them strong language learners. We build competent diffusion language models at scale by first acquiring knowledge from massive data. Experiments show that scaling diffusion language models consistently improves performance across downstream language tasks.
  arXiv  Detail & Related papers  (2023-08-23T16:01:12Z)
• SINC: Self-Supervised In-Context Learning for Vision-Language Tasks [64.44336003123102]
  We propose a framework to enable in-context learning in large language models. A meta-model can learn on self-supervised prompts consisting of tailored demonstrations. Experiments show that SINC outperforms gradient-based methods in various vision-language tasks.
  arXiv  Detail & Related papers  (2023-07-15T08:33:08Z)
• Define, Evaluate, and Improve Task-Oriented Cognitive Capabilities for Instruction Generation Models [5.975913042883176]
  Recent work studies the cognitive capabilities of language models through psychological tests designed for humans. We formulate task-oriented cognitive capabilities, which are human-like cognitive capabilities that language models leverage to perform tasks.
  arXiv  Detail & Related papers  (2022-12-21T04:43:19Z)
• ALERT: Adapting Language Models to Reasoning Tasks [43.8679673685468]
  ALERT is a benchmark and suite of analyses for assessing language models' reasoning ability. ALERT provides a test bed to asses any language model on fine-grained reasoning skills. We find that language models learn more reasoning skills during finetuning stage compared to pretraining state.
  arXiv  Detail & Related papers  (2022-12-16T05:15:41Z)
• Overcoming Barriers to Skill Injection in Language Modeling: Case Study in Arithmetic [14.618731441943847]
  We develop a novel framework that enables language models to be mathematically proficient while retaining their linguistic prowess. Specifically, we offer information-theoretic interventions to overcome the catastrophic forgetting of linguistic skills that occurs while injecting non-linguistic skills into language models.
  arXiv  Detail & Related papers  (2022-11-03T18:53:30Z)
• What Artificial Neural Networks Can Tell Us About Human Language Acquisition [47.761188531404066]
  Rapid progress in machine learning for natural language processing has the potential to transform debates about how humans learn language. To increase the relevance of learnability results from computational models, we need to train model learners without significant advantages over humans.
  arXiv  Detail & Related papers  (2022-08-17T00:12:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.