Measuring Task-Agnostic Training Data Influence Across Language Model Pretraining

• URL: http://arxiv.org/abs/2608.13515v1
• Date: Thu, 13 Aug 2026 17:36:49 GMT
• Title: Measuring Task-Agnostic Training Data Influence Across Language Model Pretraining
• Abstract summary: We propose a measure of training data influence that does not require selecting a downstream task or validation set as the attribution target.<n>We define an example's influence by how much its gradient update reduces the squared distance to the final parameters of a given pretraining run.<n>Applying the method to 18 configurations from the Pythia and PolyPythia suites, we find systematic temporal changes in influential data.
• Score: 35.37816139054677
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Measuring training data influence consistently across language model pretraining is challenging. It is difficult to select downstream tasks or validation sets representative of a model's general capabilities, and reliance on task performance at intermediate checkpoints complicates comparisons across training. We propose a measure of training data influence that does not require selecting a downstream task or validation set as the attribution target. Specifically, we define an example's influence by how much its gradient update reduces the squared distance to the final parameters of a given pretraining run, and estimate this quantity from intermediate checkpoints without retraining. Applying the method to 18 configurations from the Pythia and PolyPythia suites, we find systematic temporal changes in influential data. Early in training, literature-related data are more strongly aligned with the trajectory toward the final parameters, whereas STEM data become more strongly aligned in later stages. This qualitative crossover is broadly consistent across model configurations. Our results provide a tractable trajectory-level view of how influential data change throughout pretraining, complementing influence analyses defined with respect to specific downstream tasks or validation sets.

Related papers

• STRIDE: Training Data Attribution via Sparse Recovery from Subset Perturbations [46.963298896594814]
  Training Data Attribution seeks to trace a model's predictions back to its training data.<n>Most approaches approximate this effect in the parameter space using gradients.<n>We model the functional effect of training data in the activation space.
  arXiv  Detail & Related papers  (2026-06-03T17:59:36Z)
• Learning Multi-Indicator Weights for Data Selection: A Joint Task-Model Adaptation Framework with Efficient Proxies [50.39041754816285]
  We propose a framework for learning multi-validation weights that jointly adapts data selection to both the downstream task and the specific model.<n>Our approach achieves performance comparable to or exceeding full-dataset tuning while using only 30% of the training samples on GSM8K.
  arXiv  Detail & Related papers  (2026-05-10T17:30:16Z)
• Gradient-Weight Alignment as a Train-Time Proxy for Generalization in Classification Tasks [32.61771956544867]
  We introduce Gradient-Weight Alignment (GWA), quantifying the coherence between per-sample gradients and model weights.<n>We show that effective learning corresponds to coherent alignment, while misalignment indicates deteriorating generalization.<n>Experiments show that GWA accurately predicts optimal early stopping, enables principled model comparisons, and identifies influential training samples.
  arXiv  Detail & Related papers  (2025-10-29T13:04:17Z)
• Quantifying the Importance of Data Alignment in Downstream Model Performance [9.929939035590076]
  We use the Task2Vec-based alignment coefficient to quantify the impact of alignment between training data and evaluation data on downstream performance.<n>We find a strong, predictable negative correlation between the alignment coefficient of a model's training and evaluation data and the model's loss/perplexity on the respective downstream task.
  arXiv  Detail & Related papers  (2025-01-14T23:59:23Z)
• Capturing the Temporal Dependence of Training Data Influence [100.91355498124527]
  We formalize the concept of trajectory-specific leave-one-out influence, which quantifies the impact of removing a data point during training.<n>We propose data value embedding, a novel technique enabling efficient approximation of trajectory-specific LOO.<n>As data value embedding captures training data ordering, it offers valuable insights into model training dynamics.
  arXiv  Detail & Related papers  (2024-12-12T18:28:55Z)
• ROSE: A Reward-Oriented Data Selection Framework for LLM Task-Specific Instruction Tuning [26.069986067128912]
  We introduce ROSE, a Reward-Oriented inStruction data sElection method to optimize data selection for task-specific instruction tuning.<n> ROSE adapts an influence formulation to approximate the influence of training data points relative to a few-shot preference validation set to select the most task-related training data points.
  arXiv  Detail & Related papers  (2024-12-01T01:01:09Z)
• Distilled Datamodel with Reverse Gradient Matching [74.75248610868685]
  We introduce an efficient framework for assessing data impact, comprising offline training and online evaluation stages. Our proposed method achieves comparable model behavior evaluation while significantly speeding up the process compared to the direct retraining method.
  arXiv  Detail & Related papers  (2024-04-22T09:16:14Z)
• Unsupervised neural adaptation model based on optimal transport for spoken language identification [54.96267179988487]
  Due to the mismatch of statistical distributions of acoustic speech between training and testing sets, the performance of spoken language identification (SLID) could be drastically degraded. We propose an unsupervised neural adaptation model to deal with the distribution mismatch problem for SLID.
  arXiv  Detail & Related papers  (2020-12-24T07:37:19Z)
• How Training Data Impacts Performance in Learning-based Control [67.7875109298865]
  This paper derives an analytical relationship between the density of the training data and the control performance. We formulate a quality measure for the data set, which we refer to as $rho$-gap. We show how the $rho$-gap can be applied to a feedback linearizing control law.
  arXiv  Detail & Related papers  (2020-05-25T12:13:49Z)
• Improving Multi-Turn Response Selection Models with Complementary Last-Utterance Selection by Instance Weighting [84.9716460244444]
  We consider utilizing the underlying correlation in the data resource itself to derive different kinds of supervision signals. We conduct extensive experiments in two public datasets and obtain significant improvement in both datasets.
  arXiv  Detail & Related papers  (2020-02-18T06:29:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.