Enhancing Input-Label Mapping in In-Context Learning with Contrastive Decoding

• URL: http://arxiv.org/abs/2502.13738v1
• Date: Wed, 19 Feb 2025 14:04:46 GMT
• Title: Enhancing Input-Label Mapping in In-Context Learning with Contrastive Decoding
• Authors: Keqin Peng, Liang Ding, Yuanxin Ouyang, Meng Fang, Yancheng Yuan, Dacheng Tao,
• Abstract summary: Large language models (LLMs) excel at a range of tasks through in-context learning (ICL) We introduce In-Context Contrastive Decoding (ICCD), a novel method that emphasizes input-label mapping. ICCD emphasizes input-label mapping by contrasting the output distributions between positive and negative in-context examples.
• Score: 71.01099784480597
• License:
• Abstract: Large language models (LLMs) excel at a range of tasks through in-context learning (ICL), where only a few task examples guide their predictions. However, prior research highlights that LLMs often overlook input-label mapping information in ICL, relying more on their pre-trained knowledge. To address this issue, we introduce In-Context Contrastive Decoding (ICCD), a novel method that emphasizes input-label mapping by contrasting the output distributions between positive and negative in-context examples. Experiments on 7 natural language understanding (NLU) tasks show that our ICCD method brings consistent and significant improvement (up to +2.1 improvement on average) upon 6 different scales of LLMs without requiring additional training. Our approach is versatile, enhancing performance with various demonstration selection methods, demonstrating its broad applicability and effectiveness. The code and scripts will be publicly released.

Related papers

• Language Models Can See Better: Visual Contrastive Decoding For LLM Multimodal Reasoning [15.877954360180468]
  Training Multimodal Large Language Models (MLLMs) is resource-intensive and constrained by various training limitations. We propose the Modular-based Visual Contrastive Decoding (MVCD) framework to move this obstacle. Our framework leverages LLMs' In-Context Learning (ICL) capability and the proposed visual contrastive-example decoding (CED) Our results show consistent improvement in model accuracy and well explain the effective components inside our decoding strategy.
  arXiv  Detail & Related papers  (2025-02-17T12:47:00Z)
• Rectifying Demonstration Shortcut in In-Context Learning [15.08431909212102]
  Large language models (LLMs) are able to solve various tasks with only a few demonstrations utilizing their in-context learning (ICL) abilities. LLMs often rely on their pre-trained semantic priors of demonstrations rather than on the input-label relationships to proceed with ICL prediction.
  arXiv  Detail & Related papers  (2024-03-14T15:30:14Z)
• C-ICL: Contrastive In-context Learning for Information Extraction [54.39470114243744]
  c-ICL is a novel few-shot technique that leverages both correct and incorrect sample constructions to create in-context learning demonstrations. Our experiments on various datasets indicate that c-ICL outperforms previous few-shot in-context learning methods.
  arXiv  Detail & Related papers  (2024-02-17T11:28:08Z)
• Prompt Highlighter: Interactive Control for Multi-Modal LLMs [50.830448437285355]
  This study targets a critical aspect of multi-modal LLMs' (LLMs&VLMs) inference: explicit controllable text generation. We introduce a novel inference method, Prompt Highlighter, which enables users to highlight specific prompt spans to interactively control the focus during generation. We find that, during inference, guiding the models with highlighted tokens through the attention weights leads to more desired outputs.
  arXiv  Detail & Related papers  (2023-12-07T13:53:29Z)
• IERL: Interpretable Ensemble Representation Learning -- Combining CrowdSourced Knowledge and Distributed Semantic Representations [11.008412414253662]
  Large Language Models (LLMs) encode meanings of words in the form of distributed semantics. Recent studies have shown that LLMs tend to generate unintended, inconsistent, or wrong texts as outputs. We propose a novel ensemble learning method, Interpretable Ensemble Representation Learning (IERL), that systematically combines LLM and crowdsourced knowledge representations.
  arXiv  Detail & Related papers  (2023-06-24T05:02:34Z)
• OverPrompt: Enhancing ChatGPT through Efficient In-Context Learning [49.38867353135258]
  We propose OverPrompt, leveraging the in-context learning capability of LLMs to handle multiple task inputs. Our experiments show that OverPrompt can achieve cost-efficient zero-shot classification without causing significant detriment to task performance.
  arXiv  Detail & Related papers  (2023-05-24T10:08:04Z)
• ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extraction [56.790794611002106]
  Large language models (LLMs) have demonstrated remarkable results in various natural language processing (NLP) tasks with in-context learning. We propose a simple but effective in-context learning framework called ICL-D3IE. Specifically, we extract the most difficult and distinct segments from hard training documents as hard demonstrations.
  arXiv  Detail & Related papers  (2023-03-09T06:24:50Z)
• Dense Contrastive Visual-Linguistic Pretraining [53.61233531733243]
  Several multimodal representation learning approaches have been proposed that jointly represent image and text. These approaches achieve superior performance by capturing high-level semantic information from large-scale multimodal pretraining. We propose unbiased Dense Contrastive Visual-Linguistic Pretraining to replace the region regression and classification with cross-modality region contrastive learning.
  arXiv  Detail & Related papers  (2021-09-24T07:20:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.