STRIDE: Training Data Attribution via Sparse Recovery from Subset Perturbations

• URL: http://arxiv.org/abs/2606.05165v1
• Date: Wed, 03 Jun 2026 17:59:36 GMT
• Title: STRIDE: Training Data Attribution via Sparse Recovery from Subset Perturbations
• Abstract summary: Training Data Attribution seeks to trace a model's predictions back to its training data.<n>Most approaches approximate this effect in the parameter space using gradients.<n>We model the functional effect of training data in the activation space.
• Score: 46.963298896594814
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Training Data Attribution (TDA) seeks to trace a model's predictions back to its training data. The gold standard for TDA relies on causal interventions, observing how a model changes when data is added or removed, but repeated retraining is computationally challenging for Large Language Models (LLMs). Consequently, most approaches approximate this effect in the parameter space using gradients. However, tracking gradients across billions of parameters is not only prohibitively expensive but relies on local approximations. In this work, we propose a shift: rather than estimating parameter changes, we model the functional effect of training data in the activation space. We introduce STRIDE (Steering-based Training Data Influence Decomposition), a framework that formulates TDA as a sparse recovery problem in the spirit of compressive sensing. STRIDE learns lightweight "steering operators" that mimic the behavioral shift caused by training on data subsets. By measuring how these operators perturb test predictions, we recover individual training example influences via sparse linear decomposition. STRIDE achieves state-of-the-art for LLM pre-training attribution while being an order of magnitude ($13\times$) faster than previous art. We further validate its practical utility through downstream applications including data selection, data contamination, and qualitative analysis.

Related papers

• Data Attribution in Large Language Models via Bidirectional Gradient Optimization [35.61634772862795]
  Large Language Models (LLMs) are increasingly deployed across diverse applications.<n>understanding which training data most influenced a model's output remains a fundamental open problem.<n>We address this challenge by expanding upon the inverse formulation: How would training data be affected if the model had seen the generated output during training?<n>Our method perturbs the base model using bidirectional gradient optimization (gradient ascent and descent) on a generated text sample and measures the resulting change in loss across training samples.
  arXiv  Detail & Related papers  (2026-06-03T14:21:53Z)
• Nonparametric Data Attribution for Diffusion Models [57.820618036556084]
  Data attribution for generative models seeks to quantify the influence of individual training examples on model outputs.<n>We propose a nonparametric attribution method that operates entirely on data, measuring influence via patch-level similarity between generated and training images.
  arXiv  Detail & Related papers  (2025-10-16T03:37:16Z)
• Improving Out-of-Distribution Detection via Dynamic Covariance Calibration [12.001290283557466]
  Out-of-Distribution (OOD) detection is essential for the trustworthiness of AI systems.<n>We argue that the influence of ill-distributed samples can be corrected by dynamically adjusting the prior geometry.<n>Our approach significantly enhances OOD detection across various models.
  arXiv  Detail & Related papers  (2025-06-11T05:05:26Z)
• Capturing the Temporal Dependence of Training Data Influence [100.91355498124527]
  We formalize the concept of trajectory-specific leave-one-out influence, which quantifies the impact of removing a data point during training.<n>We propose data value embedding, a novel technique enabling efficient approximation of trajectory-specific LOO.<n>As data value embedding captures training data ordering, it offers valuable insights into model training dynamics.
  arXiv  Detail & Related papers  (2024-12-12T18:28:55Z)
• Impact of Noisy Supervision in Foundation Model Learning [91.56591923244943]
  This paper is the first work to comprehensively understand and analyze the nature of noise in pre-training datasets.<n>We propose a tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise and improve generalization.
  arXiv  Detail & Related papers  (2024-03-11T16:22:41Z)
• Imputation-Free Learning from Incomplete Observations [73.15386629370111]
  We introduce the importance of guided gradient descent (IGSGD) method to train inference from inputs containing missing values without imputation. We employ reinforcement learning (RL) to adjust the gradients used to train the models via back-propagation. Our imputation-free predictions outperform the traditional two-step imputation-based predictions using state-of-the-art imputation methods.
  arXiv  Detail & Related papers  (2021-07-05T12:44:39Z)
• Gradient-guided Loss Masking for Neural Machine Translation [27.609155878513334]
  In this paper, we explore strategies that dynamically optimize data usage during the training process. Our algorithm calculates the gradient alignment between the training data and the clean data to mask out data with negative alignment. Experiments on three WMT language pairs show that our method brings significant improvement over strong baselines.
  arXiv  Detail & Related papers  (2021-02-26T15:41:48Z)
• HYDRA: Hypergradient Data Relevance Analysis for Interpreting Deep Neural Networks [63.02441733983715]
  We propose Hypergradient Data Relevance Analysis, or HYDRA, which interprets predictions made by deep neural networks (DNNs) as effects of their training data.<n> HYDRA assesses the contribution of training data toward test data points throughout the training trajectory.<n>In addition, we quantitatively demonstrate that HYDRA outperforms influence functions in accurately estimating data contribution and detecting noisy data labels.
  arXiv  Detail & Related papers  (2021-02-04T10:00:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.