"In-Context Learning" or: How I learned to stop worrying and love "Applied Information Retrieval"

• URL: http://arxiv.org/abs/2405.01116v1
• Date: Thu, 02 May 2024 09:25:24 GMT
• Title: "In-Context Learning" or: How I learned to stop worrying and love "Applied Information Retrieval"
• Authors: Andrew Parry, Debasis Ganguly, Manish Chandra,
• Abstract summary: In-context learning (ICL) has evolved as a new paradigm for natural language processing (NLP) ICL is conceptually similar to a non-parametric approach, such as $k$-NN. Similar examples in ICL retrieved from a training set relate to a set of documents retrieved from a collection in IR.
• Score: 9.264121218481133
• License:
• Abstract: With the increasing ability of large language models (LLMs), in-context learning (ICL) has evolved as a new paradigm for natural language processing (NLP), where instead of fine-tuning the parameters of an LLM specific to a downstream task with labeled examples, a small number of such examples is appended to a prompt instruction for controlling the decoder's generation process. ICL, thus, is conceptually similar to a non-parametric approach, such as $k$-NN, where the prediction for each instance essentially depends on the local topology, i.e., on a localised set of similar instances and their labels (called few-shot examples). This suggests that a test instance in ICL is analogous to a query in IR, and similar examples in ICL retrieved from a training set relate to a set of documents retrieved from a collection in IR. While standard unsupervised ranking models can be used to retrieve these few-shot examples from a training set, the effectiveness of the examples can potentially be improved by re-defining the notion of relevance specific to its utility for the downstream task, i.e., considering an example to be relevant if including it in the prompt instruction leads to a correct prediction. With this task-specific notion of relevance, it is possible to train a supervised ranking model (e.g., a bi-encoder or cross-encoder), which potentially learns to optimally select the few-shot examples. We believe that the recent advances in neural rankers can potentially find a use case for this task of optimally choosing examples for more effective downstream ICL predictions.

Related papers

• Context-aware Prompt Tuning: Advancing In-Context Learning with Adversarial Methods [69.36397993451742]
  This work introduces Context-aware Prompt Tuning (CPT), a method inspired by ICL, PT, and adversarial attacks. We modify specific context tokens, considering the unique structure of input and output formats. Inspired by adversarial attacks, we adjust the input based on the labels present in the context, focusing on minimizing, rather than maximizing, the loss.
  arXiv  Detail & Related papers  (2024-10-22T17:45:47Z)
• Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplars [66.823588073584]
  Large language models (LLMs) have shown impressive capabilities in real-world applications. The quality of these exemplars in the prompt greatly impacts performance. Existing methods fail to adequately account for the impact of exemplar ordering on the performance.
  arXiv  Detail & Related papers  (2024-05-25T08:23:05Z)
• Implicit In-context Learning [37.0562059811099]
  In-context Learning (ICL) empowers large language models to adapt to unseen tasks during inference by prefixing a few demonstration examples prior to test queries. We introduce Implicit In-context Learning (I2CL), an innovative paradigm that addresses the challenges associated with traditional ICL by absorbing demonstration examples within the activation space. I2CL achieves few-shot performance with zero-shot cost and exhibits robustness against the variation of demonstration examples.
  arXiv  Detail & Related papers  (2024-05-23T14:57:52Z)
• ParaICL: Towards Robust Parallel In-Context Learning [74.38022919598443]
  Large language models (LLMs) have become the norm in natural language processing. Few-shot in-context learning (ICL) relies on the choice of few-shot demonstration examples. We propose a novel method named parallel in-context learning (ParaICL)
  arXiv  Detail & Related papers  (2024-03-31T05:56:15Z)
• One size doesn't fit all: Predicting the Number of Examples for In-Context Learning [16.712595387955574]
  In-context learning (ICL) refers to the process of adding a small number of localized examples (ones that are semantically similar to the input) to a training set of labelled data. Our work alleviates the limitations of this 'one fits all' approach by dynamically predicting the number of examples for each data instance to be used in few-shot inference. Our experiments on a number of text classification benchmarks show that AICL substantially outperforms standard ICL by up to 17%.
  arXiv  Detail & Related papers  (2024-03-11T03:28:13Z)
• A Mechanism for Sample-Efficient In-Context Learning for Sparse Retrieval Tasks [29.764014766305174]
  We show how a transformer model is able to perform ICL under reasonable assumptions on the pre-training process and the downstream tasks. We establish that this entire procedure is implementable using the transformer mechanism.
  arXiv  Detail & Related papers  (2023-05-26T15:49:43Z)
• RetICL: Sequential Retrieval of In-Context Examples with Reinforcement Learning [53.52699766206808]
  We propose Retrieval for In-Context Learning (RetICL), a learnable method for modeling and optimally selecting examples sequentially for in-context learning. We evaluate RetICL on math word problem solving and scientific question answering tasks and show that it consistently outperforms or matches and learnable baselines.
  arXiv  Detail & Related papers  (2023-05-23T20:15:56Z)
• Explaining Emergent In-Context Learning as Kernel Regression [61.57151500616111]
  Large language models (LLMs) have initiated a paradigm shift in transfer learning. In this paper, we investigate the reason why a transformer-based language model can accomplish in-context learning after pre-training. We find that during ICL, the attention and hidden features in LLMs match the behaviors of a kernel regression.
  arXiv  Detail & Related papers  (2023-05-22T06:45:02Z)
• Improving Few-Shot Performance of Language Models via Nearest Neighbor Calibration [12.334422701057674]
  We propose a novel nearest-neighbor calibration framework for in-context learning. It is inspired by a phenomenon that the in-context learning paradigm produces incorrect labels when inferring training instances. Experiments on various few-shot text classification tasks demonstrate that our method significantly improves in-context learning.
  arXiv  Detail & Related papers  (2022-12-05T12:49:41Z)
• Reordering Examples Helps during Priming-based Few-Shot Learning [6.579039107070663]
  We show that PERO can learn to generalize efficiently using as few as 10 examples. We demonstrate the effectiveness of the proposed method on the tasks of sentiment classification, natural language inference and fact retrieval.
  arXiv  Detail & Related papers  (2021-06-03T11:02:36Z)
• An Empirical Comparison of Instance Attribution Methods for NLP [62.63504976810927]
  We evaluate the degree to which different potential instance attribution agree with respect to the importance of training samples. We find that simple retrieval methods yield training instances that differ from those identified via gradient-based methods.
  arXiv  Detail & Related papers  (2021-04-09T01:03:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.