Step-resolved data attribution for looped transformers

• URL: http://arxiv.org/abs/2602.10097v1
• Date: Tue, 10 Feb 2026 18:57:53 GMT
• Title: Step-resolved data attribution for looped transformers
• Abstract summary: We study how individual training examples shape the internal computation of looped transformers, where a shared block is applied for $$ recurrent iterations.<n>We introduce textStep-De Influence (Sketch), which decomposes TracIn into a length-$$ influence trajectory by unrolling the recurrent graph and attributing influence to specific loop iterations.<n>Experiments on looped GPT-style models and algorithmic tasks show that SDI scales excellently, matches full-gradient baselines with low error and supports a broad range of data attribution and interpretability tasks with per-step insights into the latent reasoning process
• Score: 15.546254897542113
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study how individual training examples shape the internal computation of looped transformers, where a shared block is applied for $τ$ recurrent iterations to enable latent reasoning. Existing training-data influence estimators such as TracIn yield a single scalar score that aggregates over all loop iterations, obscuring when during the recurrent computation a training example matters. We introduce \textit{Step-Decomposed Influence (SDI)}, which decomposes TracIn into a length-$τ$ influence trajectory by unrolling the recurrent computation graph and attributing influence to specific loop iterations. To make SDI practical at transformer scale, we propose a TensorSketch implementation that never materialises per-example gradients. Experiments on looped GPT-style models and algorithmic reasoning tasks show that SDI scales excellently, matches full-gradient baselines with low error and supports a broad range of data attribution and interpretability tasks with per-step insights into the latent reasoning process.

Related papers

• Accelerating Diffusion Transformers with Gaussian Process Rectified Feature Cache [17.39547918764814]
  Diffusion Transformers have become the dominant paradigm in generative AI, but their high computational costs hinder real-time applications.<n>Prediction-based feature caching is widely used to accelerate diffusion transformers; however, as the number of steps increases, the deviation between its predictions and the reference full-compute trajectory gradually grows.<n>An intuitive idea is to use an online regression model to dynamically correct this deviation, but it faces the issue of label data being unavailable during the acceleration process.<n>This paper presents a statistical observation that the residuals between the features of full computation steps using caching methods and reference full-compute trajectory locally exhibit a
  arXiv  Detail & Related papers  (2026-09-05T08:54:54Z)
• Stabilizing Extrapolation in Looped Transformers via Learned Stochastic Stopping [56.14767235650558]
  Looped Transformers, which repeatedly apply a shared transformer block, are an architecturally natural fit for variable-length computation tasks.<n>We trace this variance to the spurious correlation in simple algorithmic tasks between sequence length and number of loops.<n>Our work suggests that "when to stop" should be treated as a training design choice, not merely an inference-time-allocation rule.
  arXiv  Detail & Related papers  (2026-06-29T08:58:09Z)
• STRIDE: Training Data Attribution via Sparse Recovery from Subset Perturbations [46.963298896594814]
  Training Data Attribution seeks to trace a model's predictions back to its training data.<n>Most approaches approximate this effect in the parameter space using gradients.<n>We model the functional effect of training data in the activation space.
  arXiv  Detail & Related papers  (2026-06-03T17:59:36Z)
• A Bayesian Information-Theoretic Approach to Data Attribution [7.908887001497405]
  Training Data Attribution seeks to trace model predictions back to influential training examples.<n>We approximate information loss using a Gaussian Process surrogate built from tangent features.<n>Experiments show our method scales to modern architectures while bridging principled measures with practice.
  arXiv  Detail & Related papers  (2026-04-04T20:42:05Z)
• Dual Path Attribution: Efficient Attribution for SwiGLU-Transformers through Layer-Wise Target Propagation [5.186807923082922]
  We introduce Dual Path Attribution (DPA), a novel framework that faithfully traces information flow on the frozen transformer in one forward and one backward pass.<n>DPA achieves state-of-the-art faithfulness and unprecedented efficiency compared to existing baselines.
  arXiv  Detail & Related papers  (2026-03-20T08:28:24Z)
• Learning with Locally Private Examples by Inverse Weierstrass Private Stochastic Gradient Descent [9.706390554730126]
  We use the Weierstrass transform to characterize this bias in binary classification.<n>We build a novel gradient descent algorithm called Inverse Weierstrass Private SGD.<n>We empirically validate IWP-SGD on binary classification tasks using synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2026-02-18T13:13:43Z)
• Unbiased Gradient Estimation for Event Binning via Functional Backpropagation [64.88399635309918]
  We propose a novel framework for unbiased gradient estimation of arbitrary binning functions by synthesizing weak derivatives during backpropagation.<n>We achieve 9.4% lower EPE in self-supervised optical flow, and 5.1% lower RMS error in SLAM, demonstrating broad benefits for event-based visual perception.
  arXiv  Detail & Related papers  (2026-02-13T04:05:03Z)
• Scalable Data Attribution via Forward-Only Test-Time Inference [3.5466521714943138]
  Data attribution seeks to trace model behavior back to the training examples that shaped it.<n>We propose a data attribution method that preserves the same first-order counterfactual target.<n>Our method provides a theoretical framework for practical, real-time data attribution in large pretrained models.
  arXiv  Detail & Related papers  (2025-11-25T00:11:39Z)
• Nonparametric Data Attribution for Diffusion Models [57.820618036556084]
  Data attribution for generative models seeks to quantify the influence of individual training examples on model outputs.<n>We propose a nonparametric attribution method that operates entirely on data, measuring influence via patch-level similarity between generated and training images.
  arXiv  Detail & Related papers  (2025-10-16T03:37:16Z)
• Capturing the Temporal Dependence of Training Data Influence [100.91355498124527]
  We formalize the concept of trajectory-specific leave-one-out influence, which quantifies the impact of removing a data point during training.<n>We propose data value embedding, a novel technique enabling efficient approximation of trajectory-specific LOO.<n>As data value embedding captures training data ordering, it offers valuable insights into model training dynamics.
  arXiv  Detail & Related papers  (2024-12-12T18:28:55Z)
• Streaming Factor Trajectory Learning for Temporal Tensor Decomposition [33.18423605559094]
  We propose Streaming Factor Trajectory Learning for temporal tensor decomposition. We use Gaussian processes (GPs) to model the trajectory of factors so as to flexibly estimate their temporal evolution. We have shown the advantage of SFTL in both synthetic tasks and real-world applications.
  arXiv  Detail & Related papers  (2023-10-25T21:58:52Z)
• Gradient-Based Feature Learning under Structured Data [57.76552698981579]
  In the anisotropic setting, the commonly used spherical gradient dynamics may fail to recover the true direction. We show that appropriate weight normalization that is reminiscent of batch normalization can alleviate this issue. In particular, under the spiked model with a suitably large spike, the sample complexity of gradient-based training can be made independent of the information exponent.
  arXiv  Detail & Related papers  (2023-09-07T16:55:50Z)
• Winner-Take-All Column Row Sampling for Memory Efficient Adaptation of Language Model [89.8764435351222]
  We propose a new family of unbiased estimators called WTA-CRS, for matrix production with reduced variance. Our work provides both theoretical and experimental evidence that, in the context of tuning transformers, our proposed estimators exhibit lower variance compared to existing ones.
  arXiv  Detail & Related papers  (2023-05-24T15:52:08Z)
• Simfluence: Modeling the Influence of Individual Training Examples by Simulating Training Runs [27.314239745883967]
  Training data attribution (TDA) methods trace a model's prediction on any given example back to specific influential training examples. We propose Simfluence, a new paradigm for TDA where the goal is not to produce a single influence score per example, but instead a training run simulator. Simfluence captures non-additive interactions and is often able to predict the spiky trajectory of individual example losses with surprising fidelity.
  arXiv  Detail & Related papers  (2023-03-14T17:47:25Z)
• On gradient descent training under data augmentation with on-line noisy copies [0.0]
  We consider descent of linear regression under DA using noisy copies of datasets, in which noise is injected into inputs. We show that, in all cases, training for DA with on-line copies is approximately equivalent to a ridge regression training. We experimentally investigated the training process of neural networks under DA with off-line noisy copies.
  arXiv  Detail & Related papers  (2022-06-08T08:20:00Z)
• Dynamic Scale Training for Object Detection [111.33112051962514]
  We propose a Dynamic Scale Training paradigm (abbreviated as DST) to mitigate scale variation challenge in object detection. Experimental results demonstrate the efficacy of our proposed DST towards scale variation handling. It does not introduce inference overhead and could serve as a free lunch for general detection configurations.
  arXiv  Detail & Related papers  (2020-04-26T16:48:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.