Influence Functions for Preference Dataset Pruning

• URL: http://arxiv.org/abs/2507.14344v1
• Date: Fri, 18 Jul 2025 19:43:36 GMT
• Title: Influence Functions for Preference Dataset Pruning
• Authors: Daniel Fein, Gabriela Aranguiz-Dias,
• Abstract summary: In this work, we adapt the TL;DR dataset for reward model training to demonstrate how conjugate-gradient approximated influence functions can be used to filter datasets.<n>In our experiments, influence function filtering yields a small retraining accuracy uplift of 1.5% after removing 10% of training examples.<n>We also show that gradient similarity outperforms influence functions for detecting helpful training examples.
• Score: 0.6138671548064356
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Language models are commonly fine-tuned via reinforcement learning to alter their behavior or elicit new capabilities. Datasets used for these purposes, and particularly human preference datasets, are often noisy. The relatively small size post-training datasets, combined with parameter-efficient fine-tuning methods, enable the use of influence functions approximations to detect and prune training examples that are harmful to performance on a validation set. In this work, we adapt the TL;DR dataset for reward model training to demonstrate how conjugate-gradient approximated influence functions can be used to filter datasets. In our experiments, influence function filtering yields a small retraining accuracy uplift of 1.5% after removing 10% of training examples. We also show that gradient similarity outperforms influence functions for detecting helpful training examples. This suggests that local curvature is important for detecting harmful training examples, but less so for identifying helpful examples.

Related papers

• Rescaled Influence Functions: Accurate Data Attribution in High Dimension [6.812390750464419]
  We present rescaled influence functions (RIF), a new tool for data attribution which can be used as a drop-in replacement for influence functions.<n>We compare IF and RIF on a range of real-world datasets, showing that RIFs offer significantly better predictions in practice.
  arXiv  Detail & Related papers  (2025-06-07T04:19:21Z)
• Efficient Data Selection at Scale via Influence Distillation [53.03573620682107]
  This paper introduces Influence Distillation, a mathematicallyjustified framework for data selection.<n>By distilling each sample's influence on a target distribution, our method assigns model-specific weights that are used to select training data.<n>Experiments show that Influence Distillation matches or outperforms state-of-the-art performance while achieving up to $3.5times$ faster selection.
  arXiv  Detail & Related papers  (2025-05-25T09:08:00Z)
• Detecting Instruction Fine-tuning Attack on Language Models with Influence Function [6.760293300577228]
  Instruction fine-tuning attacks undermine model alignment and pose security risks in real-world deployment.<n>We present a simple and effective approach to detect and mitigate such attacks using influence functions.<n>We are the first to apply influence functions for detecting language model instruction fine-tuning attacks on large-scale datasets.
  arXiv  Detail & Related papers  (2025-04-12T00:50:28Z)
• Capturing the Temporal Dependence of Training Data Influence [100.91355498124527]
  We formalize the concept of trajectory-specific leave-one-out influence, which quantifies the impact of removing a data point during training.<n>We propose data value embedding, a novel technique enabling efficient approximation of trajectory-specific LOO.<n>As data value embedding captures training data ordering, it offers valuable insights into model training dynamics.
  arXiv  Detail & Related papers  (2024-12-12T18:28:55Z)
• Exploring Learning Complexity for Efficient Downstream Dataset Pruning [8.990878450631596]
  Existing dataset pruning methods require training on the entire dataset.<n>We propose a straightforward, novel, and training-free hardness score named Distorting-based Learning Complexity (DLC)<n>Our method is motivated by the observation that easy samples learned faster can also be learned with fewer parameters.
  arXiv  Detail & Related papers  (2024-02-08T02:29:33Z)
• Unlearning Traces the Influential Training Data of Language Models [31.33791825286853]
  This paper presents UnTrac: unlearning traces the influence of a training dataset on the model's performance. We propose a more scalable approach, UnTrac-Inv, which unlearns a test dataset and evaluates the unlearned model on training datasets.
  arXiv  Detail & Related papers  (2024-01-26T23:17:31Z)
• Stubborn Lexical Bias in Data and Models [50.79738900885665]
  We use a new statistical method to examine whether spurious patterns in data appear in models trained on the data. We apply an optimization approach to *reweight* the training data, reducing thousands of spurious correlations. Surprisingly, though this method can successfully reduce lexical biases in the training data, we still find strong evidence of corresponding bias in the trained models.
  arXiv  Detail & Related papers  (2023-06-03T20:12:27Z)
• Boosting Differentiable Causal Discovery via Adaptive Sample Reweighting [62.23057729112182]
  Differentiable score-based causal discovery methods learn a directed acyclic graph from observational data. We propose a model-agnostic framework to boost causal discovery performance by dynamically learning the adaptive weights for the Reweighted Score function, ReScore.
  arXiv  Detail & Related papers  (2023-03-06T14:49:59Z)
• If Influence Functions are the Answer, Then What is the Question? [7.873458431535409]
  Influence functions efficiently estimate the effect of removing a single training data point on a model's learned parameters. While influence estimates align well with leave-one-out retraining for linear models, recent works have shown this alignment is often poor in neural networks.
  arXiv  Detail & Related papers  (2022-09-12T16:17:43Z)
• FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debugging [112.19994766375231]
  Influence functions approximate the 'influences' of training data-points for test predictions. We present FastIF, a set of simple modifications to influence functions that significantly improves their run-time. Our experiments demonstrate the potential of influence functions in model interpretation and correcting model errors.
  arXiv  Detail & Related papers  (2020-12-31T18:02:34Z)
• Influence Functions in Deep Learning Are Fragile [52.31375893260445]
  influence functions approximate the effect of samples in test-time predictions. influence estimates are fairly accurate for shallow networks. Hessian regularization is important to get highquality influence estimates.
  arXiv  Detail & Related papers  (2020-06-25T18:25:59Z)
• Provably Efficient Causal Reinforcement Learning with Confounded Observational Data [135.64775986546505]
  We study how to incorporate the dataset (observational data) collected offline, which is often abundantly available in practice, to improve the sample efficiency in the online setting. We propose the deconfounded optimistic value iteration (DOVI) algorithm, which incorporates the confounded observational data in a provably efficient manner.
  arXiv  Detail & Related papers  (2020-06-22T14:49:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.