Rescaled Influence Functions: Accurate Data Attribution in High Dimension

• URL: http://arxiv.org/abs/2506.06656v1
• Date: Sat, 07 Jun 2025 04:19:21 GMT
• Title: Rescaled Influence Functions: Accurate Data Attribution in High Dimension
• Authors: Ittai Rubinstein, Samuel B. Hopkins,
• Abstract summary: We present rescaled influence functions (RIF), a new tool for data attribution which can be used as a drop-in replacement for influence functions.<n>We compare IF and RIF on a range of real-world datasets, showing that RIFs offer significantly better predictions in practice.
• Score: 6.812390750464419
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: How does the training data affect a model's behavior? This is the question we seek to answer with data attribution. The leading practical approaches to data attribution are based on influence functions (IF). IFs utilize a first-order Taylor approximation to efficiently predict the effect of removing a set of samples from the training set without retraining the model, and are used in a wide variety of machine learning applications. However, especially in the high-dimensional regime (# params $\geq \Omega($# samples$)$), they are often imprecise and tend to underestimate the effect of sample removals, even for simple models such as logistic regression. We present rescaled influence functions (RIF), a new tool for data attribution which can be used as a drop-in replacement for influence functions, with little computational overhead but significant improvement in accuracy. We compare IF and RIF on a range of real-world datasets, showing that RIFs offer significantly better predictions in practice, and present a theoretical analysis explaining this improvement. Finally, we present a simple class of data poisoning attacks that would fool IF-based detections but would be detected by RIF.

Related papers

• Influence Functions for Preference Dataset Pruning [0.6138671548064356]
  In this work, we adapt the TL;DR dataset for reward model training to demonstrate how conjugate-gradient approximated influence functions can be used to filter datasets.<n>In our experiments, influence function filtering yields a small retraining accuracy uplift of 1.5% after removing 10% of training examples.<n>We also show that gradient similarity outperforms influence functions for detecting helpful training examples.
  arXiv  Detail & Related papers  (2025-07-18T19:43:36Z)
• IF-GUIDE: Influence Function-Guided Detoxification of LLMs [53.051109450536885]
  We study how training data contributes to the emergence of toxic behaviors in large-language models.<n>We propose a $proactive approach that leverages influence functions to identify harmful tokens within any training data and suppress their impact during training.<n>We present a novel adaptation that measures token-level attributions from training data to model toxicity, along with techniques for selecting toxic training documents and a learning objective.
  arXiv  Detail & Related papers  (2025-06-02T15:32:36Z)
• Statistically Testing Training Data for Unwanted Error Patterns using Rule-Oriented Regression [0.5831737970661137]
  We provide a method to test training data for flaws, to establish a trustworthy ground-truth for a subsequent training of machine learning models.<n>Our approach extends the abilities of conventional statistical testing by letting the test-condition'' be any condition to describe a pattern in the data.<n>We provide an open source implementation for demonstration and experiments.
  arXiv  Detail & Related papers  (2025-03-24T09:52:36Z)
• DUPRE: Data Utility Prediction for Efficient Data Valuation [49.60564885180563]
  Cooperative game theory-based data valuation, such as Data Shapley, requires evaluating the data utility and retraining the ML model for multiple data subsets.<n>Our framework, textttDUPRE, takes an alternative yet complementary approach that reduces the cost per subset evaluation by predicting data utilities instead of evaluating them by model retraining.<n>Specifically, given the evaluated data utilities of some data subsets, textttDUPRE fits a emphGaussian process (GP) regression model to predict the utility of every other data subset.
  arXiv  Detail & Related papers  (2025-02-22T08:53:39Z)
• Deeper Understanding of Black-box Predictions via Generalized Influence Functions [6.649753747542211]
  Influence functions (IFs) elucidate how data changes model. Increasing size and non- interpretity in large-scale models make IFs inaccurate. We introduce generalized IFs, precisely estimating target parameters' influence while nullifying nuisance gradient changes on fixed parameters.
  arXiv  Detail & Related papers  (2023-12-09T14:17:12Z)
• DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Models [31.65198592956842]
  We propose DataInf, an efficient influence approximation method that is practical for large-scale generative AI models. Our theoretical analysis shows that DataInf is particularly well-suited for parameter-efficient fine-tuning techniques such as LoRA. In applications to RoBERTa-large, Llama-2-13B-chat, and stable-diffusion-v1.5 models, DataInf effectively identifies the most influential fine-tuning examples better than other approximate influence scores.
  arXiv  Detail & Related papers  (2023-10-02T04:59:19Z)
• Machine Learning Force Fields with Data Cost Aware Training [94.78998399180519]
  Machine learning force fields (MLFF) have been proposed to accelerate molecular dynamics (MD) simulation. Even for the most data-efficient MLFFs, reaching chemical accuracy can require hundreds of frames of force and energy labels. We propose a multi-stage computational framework -- ASTEROID, which lowers the data cost of MLFFs by leveraging a combination of cheap inaccurate data and expensive accurate data.
  arXiv  Detail & Related papers  (2023-06-05T04:34:54Z)
• Boosting Differentiable Causal Discovery via Adaptive Sample Reweighting [62.23057729112182]
  Differentiable score-based causal discovery methods learn a directed acyclic graph from observational data. We propose a model-agnostic framework to boost causal discovery performance by dynamically learning the adaptive weights for the Reweighted Score function, ReScore.
  arXiv  Detail & Related papers  (2023-03-06T14:49:59Z)
• Leveraging Unlabeled Data to Predict Out-of-Distribution Performance [63.740181251997306]
  Real-world machine learning deployments are characterized by mismatches between the source (training) and target (test) distributions. In this work, we investigate methods for predicting the target domain accuracy using only labeled source data and unlabeled target data. We propose Average Thresholded Confidence (ATC), a practical method that learns a threshold on the model's confidence, predicting accuracy as the fraction of unlabeled examples.
  arXiv  Detail & Related papers  (2022-01-11T23:01:12Z)
• FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debugging [112.19994766375231]
  Influence functions approximate the 'influences' of training data-points for test predictions. We present FastIF, a set of simple modifications to influence functions that significantly improves their run-time. Our experiments demonstrate the potential of influence functions in model interpretation and correcting model errors.
  arXiv  Detail & Related papers  (2020-12-31T18:02:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.