Unifying Demonstration Selection and Compression for In-Context Learning

• URL: http://arxiv.org/abs/2405.17062v2
• Date: Sat, 15 Jun 2024 21:16:57 GMT
• Title: Unifying Demonstration Selection and Compression for In-Context Learning
• Authors: Jun Gao, Ziqiang Cao, Wenjie Li,
• Abstract summary: We propose an ICL framework UniICL, which Unifies demonstration selection and compression, and final response generation via a single frozen LLM. UniICL is a parameter-efficient framework that only contains 17M trainable parameters originating from the projection layer.
• Score: 14.545490629324295
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In-context learning (ICL) facilitates large language models (LLMs) exhibiting spectacular emergent capabilities in various scenarios. Unfortunately, introducing demonstrations easily makes the prompt length explode, bringing a significant burden to hardware. In addition, random demonstrations usually achieve limited improvements in ICL, necessitating demonstration selection among accessible candidates. Previous studies introduce extra modules to perform demonstration compression or selection independently. In this paper, we propose an ICL framework UniICL, which Unifies demonstration selection and compression, and final response generation via a single frozen LLM. Specifically, UniICL first projects actual demonstrations and inference text inputs into short virtual tokens, respectively. Then, virtual tokens are applied to select suitable demonstrations by measuring semantic similarity within latent space among candidate demonstrations and inference input. Finally, inference text inputs together with selected virtual demonstrations are fed into the same frozen LLM for response generation. Notably, UniICL is a parameter-efficient framework that only contains 17M trainable parameters originating from the projection layer. We conduct experiments and analysis over in- and out-domain datasets of both generative and understanding tasks, encompassing ICL scenarios with plentiful and limited demonstration candidates. Results show that UniICL effectively unifies $12 \times$ compression, demonstration selection, and response generation, efficiently scaling up the baseline from 4-shot to 64-shot ICL in IMDb with 24 GB CUDA allocation

Related papers

• Show, Don't Tell: Aligning Language Models with Demonstrated Feedback [54.10302745921713]
  Demonstration ITerated Task Optimization (DITTO) directly aligns language model outputs to a user's demonstrated behaviors. We evaluate DITTO's ability to learn fine-grained style and task alignment across domains such as news articles, emails, and blog posts.
  arXiv  Detail & Related papers  (2024-06-02T23:13:56Z)
• ParaICL: Towards Robust Parallel In-Context Learning [74.38022919598443]
  Large language models (LLMs) have become the norm in natural language processing. Few-shot in-context learning (ICL) relies on the choice of few-shot demonstration examples. We propose a novel method named parallel in-context learning (ParaICL)
  arXiv  Detail & Related papers  (2024-03-31T05:56:15Z)
• In-Context Learning with Iterative Demonstration Selection [32.62104857810135]
  Large language models (LLMs) have demonstrated strong few-shot learning ability via in-context learning (ICL) The performance of ICL has been shown to be highly sensitive to the selection of few-shot demonstrations. We propose Iterative Demonstration Selection (IDS) to leverage the merits of both dimensions.
  arXiv  Detail & Related papers  (2023-10-15T16:40:19Z)
• Dynamic Demonstrations Controller for In-Context Learning [51.3439660534631]
  In-Context Learning (ICL) is a new paradigm for natural language processing (NLP), where a large language model observes a small number of demonstrations and a test instance as its input. Previous studies have revealed that ICL is sensitive to the selection and the ordering of demonstrations. We propose a Dynamic Demonstrations Controller (D$2$Controller), which can improve the ICL performance by adjusting the number of demonstrations.
  arXiv  Detail & Related papers  (2023-09-30T14:04:22Z)
• Are Human-generated Demonstrations Necessary for In-context Learning? [22.783456038837794]
  Self-contemplation prompting strategy (SEC) is a paradigm free from human-crafted demonstrations. Extensive experiments in arithmetic reasoning, commonsense reasoning, multi-task language understanding, and code generation benchmarks, show that SEC significantly outperforms the zero-shot learning strategy.
  arXiv  Detail & Related papers  (2023-09-26T05:10:08Z)
• Ambiguity-Aware In-Context Learning with Large Language Models [27.20414960164616]
  In-context learning (ICL) i.e. showing LLMs task-specific demonstrations has led to downstream gains with no task-specific fine-tuning required. This study investigates how to select good demonstrations for ICL. We find that it is beneficial to not only choose semantically similar ICL demonstrations but also to choose those that help resolve the inherent label ambiguity surrounding the test example.
  arXiv  Detail & Related papers  (2023-09-14T17:48:34Z)
• Exploring Demonstration Ensembling for In-context Learning [75.35436025709049]
  In-context learning (ICL) operates by showing language models (LMs) examples of input-output pairs for a given task. The standard approach for ICL is to prompt the LMd demonstrations followed by the test input. In this work, we explore Demonstration Ensembling (DENSE) as an alternative to simple concatenation.
  arXiv  Detail & Related papers  (2023-08-17T04:45:19Z)
• Self-ICL: Zero-Shot In-Context Learning with Self-Generated Demonstrations [38.4166247280112]
  Self-ICL is a framework which bootstraps LMs' intrinsic capabilities to perform zero-shot ICL. Self-ICL outperforms zero-shot baselines on both average accuracy and head-to-head comparison.
  arXiv  Detail & Related papers  (2023-05-24T11:22:34Z)
• Dr.ICL: Demonstration-Retrieved In-context Learning [29.142262267850704]
  In-context learning (ICL) teaching a large language model to perform a task with few-shot demonstrations has emerged as a strong paradigm for using LLMs. Recent research suggests that retrieving semantically similar demonstrations to the input from a pool of available demonstrations results in better performance. This work expands the applicability of retrieval-based ICL approaches by demonstrating that even simple word-overlap similarity measures such as BM25 outperform randomly selected demonstrations.
  arXiv  Detail & Related papers  (2023-05-23T14:55:25Z)
• Iterative Forward Tuning Boosts In-Context Learning in Language Models [88.25013390669845]
  In this study, we introduce a novel two-stage framework to boost in-context learning in large language models (LLMs) Specifically, our framework delineates the ICL process into two distinct stages: Deep-Thinking and test stages. The Deep-Thinking stage incorporates a unique attention mechanism, i.e., iterative enhanced attention, which enables multiple rounds of information accumulation.
  arXiv  Detail & Related papers  (2023-05-22T13:18:17Z)
• ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extraction [56.790794611002106]
  Large language models (LLMs) have demonstrated remarkable results in various natural language processing (NLP) tasks with in-context learning. We propose a simple but effective in-context learning framework called ICL-D3IE. Specifically, we extract the most difficult and distinct segments from hard training documents as hard demonstrations.
  arXiv  Detail & Related papers  (2023-03-09T06:24:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.