Affinity and Diversity: A Unified Metric for Demonstration Selection via Internal Representations

• URL: http://arxiv.org/abs/2502.14380v1
• Date: Thu, 20 Feb 2025 09:12:51 GMT
• Title: Affinity and Diversity: A Unified Metric for Demonstration Selection via Internal Representations
• Authors: Mariko Kato, Hakaze Cho, Yoshihiro Sakai, Naoya Inoue,
• Abstract summary: We propose a unified metric--affinity and diversity--that leverages ICL model's internal representations. Our experiments show that both affinity and diversity strongly correlate with test accuracies, indicating their effectiveness for demonstration selection.
• Score: 2.4866936275046405
• License:
• Abstract: The performance of In-Context Learning (ICL) is highly sensitive to the selected demonstrations. Existing approaches to demonstration selection optimize different objectives, yielding inconsistent results. To address this, we propose a unified metric--affinity and diversity--that leverages ICL model's internal representations. Our experiments show that both affinity and diversity strongly correlate with test accuracies, indicating their effectiveness for demonstration selection. Moreover, we show that our proposed metrics align well with various previous works to unify the inconsistency.

Related papers

• Demonstration Selection for In-Context Learning via Reinforcement Learning [16.103533806505403]
  Relevance-Diversity Enhanced Selection (RDES) is an innovative approach to optimize the selection of diverse reference demonstrations for text classification tasks. RDES employs a Q-learning framework to dynamically identify demonstrations that maximize both diversity and relevance to the classification objective. We demonstrate that RDES significantly enhances classification accuracy compared to ten established baselines.
  arXiv  Detail & Related papers  (2024-12-05T08:33:52Z)
• DemoShapley: Valuation of Demonstrations for In-Context Learning [20.26604061802236]
  Large language models (LLMs) leveraging in-context learning (ICL) have set new benchmarks in few-shot learning across various tasks without needing task-specific fine-tuning. We introduce DemoShapley which is inspired by the Data Shapley valuation theorem. Our findings reveal that DemoShapley not only enhances model performance in terms of accuracy and fairness but also generalizes queries from domains distinct from those of the in-context demonstrations.
  arXiv  Detail & Related papers  (2024-10-10T01:35:03Z)
• DETAIL: Task DEmonsTration Attribution for Interpretable In-context Learning [75.68193159293425]
  In-context learning (ICL) allows transformer-based language models to learn a specific task with a few "task demonstrations" without updating their parameters. We propose an influence function-based attribution technique, DETAIL, that addresses the specific characteristics of ICL. We experimentally prove the wide applicability of DETAIL by showing our attribution scores obtained on white-box models are transferable to black-box models in improving model performance.
  arXiv  Detail & Related papers  (2024-05-22T15:52:52Z)
• ParaICL: Towards Robust Parallel In-Context Learning [74.38022919598443]
  Large language models (LLMs) have become the norm in natural language processing. Few-shot in-context learning (ICL) relies on the choice of few-shot demonstration examples. We propose a novel method named parallel in-context learning (ParaICL)
  arXiv  Detail & Related papers  (2024-03-31T05:56:15Z)
• Revisiting Demonstration Selection Strategies in In-Context Learning [66.11652803887284]
  Large language models (LLMs) have shown an impressive ability to perform a wide range of tasks using in-context learning (ICL) In this work, we first revisit the factors contributing to this variance from both data and model aspects, and find that the choice of demonstration is both data- and model-dependent. We propose a data- and model-dependent demonstration selection method, textbfTopK + ConE, based on the assumption that textitthe performance of a demonstration positively correlates with its contribution to the model's understanding of the test samples.
  arXiv  Detail & Related papers  (2024-01-22T16:25:27Z)
• In-Context Learning with Iterative Demonstration Selection [32.62104857810135]
  Large language models (LLMs) have demonstrated strong few-shot learning ability via in-context learning (ICL) The performance of ICL has been shown to be highly sensitive to the selection of few-shot demonstrations. We propose Iterative Demonstration Selection (IDS) to leverage the merits of both dimensions.
  arXiv  Detail & Related papers  (2023-10-15T16:40:19Z)
• Leveraging Diffusion Disentangled Representations to Mitigate Shortcuts in Underspecified Visual Tasks [92.32670915472099]
  We propose an ensemble diversification framework exploiting the generation of synthetic counterfactuals using Diffusion Probabilistic Models (DPMs) We show that diffusion-guided diversification can lead models to avert attention from shortcut cues, achieving ensemble diversity performance comparable to previous methods requiring additional data collection.
  arXiv  Detail & Related papers  (2023-10-03T17:37:52Z)
• In-Context Demonstration Selection with Cross Entropy Difference [95.21947716378641]
  Large language models (LLMs) can use in-context demonstrations to improve performance on zero-shot tasks. We present a cross-entropy difference (CED) method for selecting in-context demonstrations.
  arXiv  Detail & Related papers  (2023-05-24T05:04:00Z)
• Concurrent Discrimination and Alignment for Self-Supervised Feature Learning [52.213140525321165]
  Existing self-supervised learning methods learn by means of pretext tasks which are either (1) discriminating that explicitly specify which features should be separated or (2) aligning that precisely indicate which features should be closed together. In this work, we combine the positive aspects of the discriminating and aligning methods, and design a hybrid method that addresses the above issue. Our method explicitly specifies the repulsion and attraction mechanism respectively by discriminative predictive task and concurrently maximizing mutual information between paired views. Our experiments on nine established benchmarks show that the proposed model consistently outperforms the existing state-of-the-art results of self-supervised and transfer
  arXiv  Detail & Related papers  (2021-08-19T09:07:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.