Influcoder: Distilling Decoders' Gradient Influence Rankings into an Encoder for Data Attribution

• URL: http://arxiv.org/abs/2606.13668v1
• Date: Thu, 11 Jun 2026 17:58:33 GMT
• Title: Influcoder: Distilling Decoders' Gradient Influence Rankings into an Encoder for Data Attribution
• Abstract summary: Data Attribution methods estimate how samples in a training dataset can precondition a model to generate certain outputs.<n>We propose a method, Influcoder, as a quick and cost-effective approach to influence-based Data Attribution at scale.
• Score: 3.1857077981565793
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: With the growth of LLMs' (Large Language Models) capabilities, there has been an increasing push to curate high quality datasets by filtering samples in the training data. In general, Data Attribution (DA) methods aim to estimate how individual samples in a training dataset can precondition a model to generate certain outputs. As an example, one might be interested in which samples in the data could be the source of toxic behavior after training the LLM. Many methods quantify this conditioning through the paradigm of influence functions. While methods of this family are effective in its function, they lack the necessary processing speed and storage compactness to be practically implemented on large datasets. We propose a method, Influcoder, as a quick and cost-effective approach to influence-based Data Attribution at scale.

Related papers

• An Empirical Study on Influence-Based Pretraining Data Selection for Code Large Language Models [30.70386124413396]
  This paper explores the effectiveness of data-influence-score filtering within the context of programming-related datasets.<n>We pre-train a Code-LLMs with 1 billion parameters from scratch on a dataset of 100 billion code tokens.<n>Our findings show that data-influence-score filtering based on validation-set-loss can enhance models programming performance.
  arXiv  Detail & Related papers  (2026-04-09T03:48:11Z)
• Efficient Data Selection at Scale via Influence Distillation [53.03573620682107]
  This paper introduces Influence Distillation, a mathematicallyjustified framework for data selection.<n>By distilling each sample's influence on a target distribution, our method assigns model-specific weights that are used to select training data.<n>Experiments show that Influence Distillation matches or outperforms state-of-the-art performance while achieving up to $3.5times$ faster selection.
  arXiv  Detail & Related papers  (2025-05-25T09:08:00Z)
• Capturing the Temporal Dependence of Training Data Influence [100.91355498124527]
  We formalize the concept of trajectory-specific leave-one-out influence, which quantifies the impact of removing a data point during training.<n>We propose data value embedding, a novel technique enabling efficient approximation of trajectory-specific LOO.<n>As data value embedding captures training data ordering, it offers valuable insights into model training dynamics.
  arXiv  Detail & Related papers  (2024-12-12T18:28:55Z)
• Beyond Efficiency: Molecular Data Pruning for Enhanced Generalization [30.738229850748137]
  MolPeg is a Molecular data Pruning framework for enhanced Generalization. It focuses on the source-free data pruning scenario, where data pruning is applied with pretrained models. It consistently outperforms existing DP methods across four downstream tasks.
  arXiv  Detail & Related papers  (2024-09-02T09:06:04Z)
• Entropy Law: The Story Behind Data Compression and LLM Performance [115.70395740286422]
  We find that model performance is negatively correlated to the compression ratio of training data, which usually yields a lower training loss. Based on the findings of the entropy law, we propose a quite efficient and universal data selection method. We also present an interesting application of entropy law that can detect potential performance risks at the beginning of model training.
  arXiv  Detail & Related papers  (2024-07-09T08:14:29Z)
• Measuring Sample Importance in Data Pruning for Language Models based on Information Entropy [4.079147243688765]
  We consider a data pruning method based on information entropy.<n>We propose that the samples in the training corpus be ranked in terms of their informativeness.<n> Experiments reveal that the proposed information-based pruning can improve upon various language modeling and downstream tasks.
  arXiv  Detail & Related papers  (2024-06-20T09:09:34Z)
• How to Train Data-Efficient LLMs [56.41105687693619]
  We study data-efficient approaches for pre-training language models (LLMs) We find that Ask-LLM and Density sampling are the best methods in their respective categories. In our comparison of 19 samplers, involving hundreds of evaluation tasks and pre-training runs, we find that Ask-LLM and Density are the best methods in their respective categories.
  arXiv  Detail & Related papers  (2024-02-15T02:27:57Z)
• LESS: Selecting Influential Data for Targeted Instruction Tuning [64.78894228923619]
  We propose LESS, an efficient algorithm to estimate data influences and perform Low-rank gradiEnt Similarity Search for instruction data selection. We show that training on a LESS-selected 5% of the data can often outperform training on the full dataset across diverse downstream tasks. Our method goes beyond surface form cues to identify data that the necessary reasoning skills for the intended downstream application.
  arXiv  Detail & Related papers  (2024-02-06T19:18:04Z)
• DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Models [31.65198592956842]
  We propose DataInf, an efficient influence approximation method that is practical for large-scale generative AI models. Our theoretical analysis shows that DataInf is particularly well-suited for parameter-efficient fine-tuning techniques such as LoRA. In applications to RoBERTa-large, Llama-2-13B-chat, and stable-diffusion-v1.5 models, DataInf effectively identifies the most influential fine-tuning examples better than other approximate influence scores.
  arXiv  Detail & Related papers  (2023-10-02T04:59:19Z)
• From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning [52.257422715393574]
  We introduce a self-guided methodology for Large Language Models (LLMs) to autonomously discern and select cherry samples from open-source datasets. Our key innovation, the Instruction-Following Difficulty (IFD) metric, emerges as a pivotal metric to identify discrepancies between a model's expected responses and its intrinsic generation capability.
  arXiv  Detail & Related papers  (2023-08-23T09:45:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.