A Survey to Recent Progress Towards Understanding In-Context Learning

• URL: http://arxiv.org/abs/2402.02212v3
• Date: Fri, 24 Jan 2025 19:04:04 GMT
• Title: A Survey to Recent Progress Towards Understanding In-Context Learning
• Authors: Haitao Mao, Guangliang Liu, Yao Ma, Rongrong Wang, Kristen Johnson, Jiliang Tang,
• Abstract summary: In-Context Learning (ICL) empowers Large Language Models (LLMs) with the ability to learn from a few examples provided in the prompt. Despite encouragingly empirical success, the underlying mechanism of ICL remains unclear.
• Score: 37.933016939520684
• License:
• Abstract: In-Context Learning (ICL) empowers Large Language Models (LLMs) with the ability to learn from a few examples provided in the prompt, enabling downstream generalization without the requirement for gradient updates. Despite encouragingly empirical success, the underlying mechanism of ICL remains unclear. Existing research remains ambiguous with various viewpoints, utilizing intuition-driven and ad-hoc technical solutions to interpret ICL. In this paper, we leverage a data generation perspective to reinterpret recent efforts from a systematic angle, demonstrating the potential broader usage of these popular technical solutions. For a conceptual definition, we rigorously adopt the terms of skill recognition and skill learning. Skill recognition selects one learned data generation function previously seen during pre-training while skill learning can learn new data generation functions from in-context data. Furthermore, we provide insights into the strengths and weaknesses of both abilities, emphasizing their commonalities through the perspective of data generation. This analysis suggests potential directions for future research.

Related papers

• In-Context Learning with Topological Information for Knowledge Graph Completion [3.035601871864059]
  We develop a novel method that incorporates topological information through in-context learning to enhance knowledge graph performance. Our approach achieves strong performance in the transductive setting i.e., nodes in the test graph dataset are present in the training graph dataset. Our method demonstrates superior performance compared to baselines on the ILPC-small and ILPC-large datasets.
  arXiv  Detail & Related papers  (2024-12-11T19:29:36Z)
• Coding for Intelligence from the Perspective of Category [66.14012258680992]
  Coding targets compressing and reconstructing data, and intelligence. Recent trends demonstrate the potential homogeneity of these two fields. We propose a novel problem of Coding for Intelligence from the category theory view.
  arXiv  Detail & Related papers  (2024-07-01T07:05:44Z)
• Large Language Models are Limited in Out-of-Context Knowledge Reasoning [65.72847298578071]
  Large Language Models (LLMs) possess extensive knowledge and strong capabilities in performing in-context reasoning. This paper focuses on a significant aspect of out-of-context reasoning: Out-of-Context Knowledge Reasoning (OCKR), which is to combine multiple knowledge to infer new knowledge.
  arXiv  Detail & Related papers  (2024-06-11T15:58:59Z)
• C-ICL: Contrastive In-context Learning for Information Extraction [54.39470114243744]
  c-ICL is a novel few-shot technique that leverages both correct and incorrect sample constructions to create in-context learning demonstrations. Our experiments on various datasets indicate that c-ICL outperforms previous few-shot in-context learning methods.
  arXiv  Detail & Related papers  (2024-02-17T11:28:08Z)
• SINC: Self-Supervised In-Context Learning for Vision-Language Tasks [64.44336003123102]
  We propose a framework to enable in-context learning in large language models. A meta-model can learn on self-supervised prompts consisting of tailored demonstrations. Experiments show that SINC outperforms gradient-based methods in various vision-language tasks.
  arXiv  Detail & Related papers  (2023-07-15T08:33:08Z)
• The Contribution of Knowledge in Visiolinguistic Learning: A Survey on Tasks and Challenges [0.0]
  Current datasets used for visiolinguistic (VL) pre-training only contain a limited amount of visual and linguistic knowledge. External knowledge sources such as knowledge graphs (KGs) and Large Language Models (LLMs) are able to cover such generalization gaps.
  arXiv  Detail & Related papers  (2023-03-04T13:12:18Z)
• Semi-Supervised and Unsupervised Deep Visual Learning: A Survey [76.2650734930974]
  Semi-supervised learning and unsupervised learning offer promising paradigms to learn from an abundance of unlabeled visual data. We review the recent advanced deep learning algorithms on semi-supervised learning (SSL) and unsupervised learning (UL) for visual recognition from a unified perspective.
  arXiv  Detail & Related papers  (2022-08-24T04:26:21Z)
• Semantics of the Black-Box: Can knowledge graphs help make deep learning systems more interpretable and explainable? [4.2111286819721485]
  Recent innovations in deep learning (DL) have shown enormous potential to impact individuals and society. Black-Box nature of DL models and over-reliance on massive amounts of data poses challenges for interpretability and explainability of the system. This article demonstrates how knowledge, provided as a knowledge graph, is incorporated into DL methods using knowledge-infused learning.
  arXiv  Detail & Related papers  (2020-10-16T22:55:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.