Fast Training Dataset Attribution via In-Context Learning

• URL: http://arxiv.org/abs/2408.11852v1
• Date: Wed, 14 Aug 2024 20:48:45 GMT
• Title: Fast Training Dataset Attribution via In-Context Learning
• Authors: Milad Fotouhi, Mohammad Taha Bahadori, Oluwaseyi Feyisetan, Payman Arabshahi, David Heckerman,
• Abstract summary: We investigate the use of in-context learning and prompt engineering to estimate the contributions of training data in instruction-tuned large language models (LLMs) We propose two novel approaches: (1) a similarity-based approach that measures the difference between LLM outputs with and without provided context, and (2) a mixture distribution model approach that frames the problem of identifying contribution scores as a matrix factorization task.
• Score: 9.542023122304096
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We investigate the use of in-context learning and prompt engineering to estimate the contributions of training data in the outputs of instruction-tuned large language models (LLMs). We propose two novel approaches: (1) a similarity-based approach that measures the difference between LLM outputs with and without provided context, and (2) a mixture distribution model approach that frames the problem of identifying contribution scores as a matrix factorization task. Our empirical comparison demonstrates that the mixture model approach is more robust to retrieval noise in in-context learning, providing a more reliable estimation of data contributions.

Related papers

• Optimizing Pretraining Data Mixtures with LLM-Estimated Utility [52.08428597962423]
  Large Language Models improve with increasing amounts of high-quality training data. We find token-counts outperform manual and learned mixes, indicating that simple approaches for dataset size and diversity are surprisingly effective. We propose two complementary approaches: UtiliMax, which extends token-based $200s by incorporating utility estimates from reduced-scale ablations, achieving up to a 10.6x speedup over manual baselines; and Model Estimated Data Utility (MEDU), which leverages LLMs to estimate data utility from small samples, matching ablation-based performance while reducing computational requirements by $simx.
  arXiv  Detail & Related papers  (2025-01-20T21:10:22Z)
• An Active Learning Framework for Inclusive Generation by Large Language Models [32.16984263644299]
  Large Language Models (LLMs) generate text representative of diverse sub-populations. We propose a novel clustering-based active learning framework, enhanced with knowledge distillation. We construct two new datasets in tandem with model training, showing a performance improvement of 2%-10% over baseline models.
  arXiv  Detail & Related papers  (2024-10-17T15:09:35Z)
• A Statistical Framework for Data-dependent Retrieval-Augmented Models [46.781026675083254]
  Modern ML systems increasingly augment input instances with additional relevant information to enhance final prediction. We study such models with two components: 1) a em retriever to identify the relevant information out of a large corpus via a data-dependent metric; and 2) a em predictor that consumes the input instances along with the retrieved information to make the final predictions.
  arXiv  Detail & Related papers  (2024-08-27T20:51:06Z)
• Embedding And Clustering Your Data Can Improve Contrastive Pretraining [0.0]
  We explore extending training data stratification beyond source granularity by leveraging a pretrained text embedding model and the classic k-means clustering algorithm. Experimentally, we observe a notable increase in NDCG@10 when pretraining a BERT-based text embedding model on query-passage pairs from the MSMARCO passage retrieval dataset.
  arXiv  Detail & Related papers  (2024-07-26T17:36:40Z)
• Anchor-aware Deep Metric Learning for Audio-visual Retrieval [11.675472891647255]
  Metric learning aims at capturing the underlying data structure and enhancing the performance of tasks like audio-visual cross-modal retrieval (AV-CMR) Recent works employ sampling methods to select impactful data points from the embedding space during training. However, the model training fails to fully explore the space due to the scarcity of training data points. We propose an innovative Anchor-aware Deep Metric Learning (AADML) method to address this challenge.
  arXiv  Detail & Related papers  (2024-04-21T22:44:44Z)
• Explore In-Context Segmentation via Latent Diffusion Models [132.26274147026854]
  latent diffusion model (LDM) is an effective minimalist for in-context segmentation. We build a new and fair in-context segmentation benchmark that includes both image and video datasets.
  arXiv  Detail & Related papers  (2024-03-14T17:52:31Z)
• C-ICL: Contrastive In-context Learning for Information Extraction [54.39470114243744]
  c-ICL is a novel few-shot technique that leverages both correct and incorrect sample constructions to create in-context learning demonstrations. Our experiments on various datasets indicate that c-ICL outperforms previous few-shot in-context learning methods.
  arXiv  Detail & Related papers  (2024-02-17T11:28:08Z)
• Revisiting Demonstration Selection Strategies in In-Context Learning [66.11652803887284]
  Large language models (LLMs) have shown an impressive ability to perform a wide range of tasks using in-context learning (ICL) In this work, we first revisit the factors contributing to this variance from both data and model aspects, and find that the choice of demonstration is both data- and model-dependent. We propose a data- and model-dependent demonstration selection method, textbfTopK + ConE, based on the assumption that textitthe performance of a demonstration positively correlates with its contribution to the model's understanding of the test samples.
  arXiv  Detail & Related papers  (2024-01-22T16:25:27Z)
• Improving Open Information Extraction with Large Language Models: A Study on Demonstration Uncertainty [52.72790059506241]
  Open Information Extraction (OIE) task aims at extracting structured facts from unstructured text. Despite the potential of large language models (LLMs) like ChatGPT as a general task solver, they lag behind state-of-the-art (supervised) methods in OIE tasks.
  arXiv  Detail & Related papers  (2023-09-07T01:35:24Z)
• Multimodal Relation Extraction with Cross-Modal Retrieval and Synthesis [89.04041100520881]
  This research proposes to retrieve textual and visual evidence based on the object, sentence, and whole image. We develop a novel approach to synthesize the object-level, image-level, and sentence-level information for better reasoning between the same and different modalities.
  arXiv  Detail & Related papers  (2023-05-25T15:26:13Z)
• Distributionally Robust Learning in Heterogeneous Contexts [29.60681287631439]
  We consider the problem of learning from training data obtained in different contexts, where the test data is subject to distributional shifts. We develop a distributionally robust method that focuses on excess risks and achieves a more appropriate trade-off between performance and robustness than the conventional and overly conservative minimax approach.
  arXiv  Detail & Related papers  (2021-05-18T14:00:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.