Right Direction, Wrong Step: Geometric Analysis of Finite-Step Failure in Looped Transformers

• URL: http://arxiv.org/abs/2609.16665v1
• Date: Tue, 15 Sep 2026 05:42:02 GMT
• Title: Right Direction, Wrong Step: Geometric Analysis of Finite-Step Failure in Looped Transformers
• Abstract summary: Looped Transformers offer a parameter-efficient route to test-time scaling by reusing shared layers for iterative latent reasoning.<n>Additional iterations can reduce support for a reference answer, leaving unclear whether an update's direction is locally unhelpful or its full displacement moves too far.<n>We study this distinction by analysing reference utility, which measures this support, along the model's own update direction.
• Score: 40.43356191604634
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Looped Transformers offer a parameter-efficient route to test-time scaling by reusing shared layers for iterative latent reasoning. However, additional iterations can reduce support for a reference answer, leaving unclear whether an update's direction is locally unhelpful or its full displacement moves too far. We study this distinction by analysing reference utility, which measures this support, along the model's own update direction, varying the fraction of the proposed displacement supplied to the readout. This reveals finite-step failures in which a locally improving direction produces a harmful full update. A pathwise curvature decomposition characterises how initial progress is lost, while a local quadratic model predicts full-step gains and useful step scales. Bounds based on accumulated curvature variation characterise the approximation error of these predictions. Experiments across two model families reveal this separation on mathematical and commonsense tasks. A fixed quarter step produces positive gains in reference utility for 72.2--83.2% of selected failures across four settings. These findings identify a mismatch between update direction and step scale as a mechanism of lost progress, explaining how some harmful updates retain useful computation.

Related papers

• Accelerating Diffusion Transformers with Gaussian Process Rectified Feature Cache [17.39547918764814]
  Diffusion Transformers have become the dominant paradigm in generative AI, but their high computational costs hinder real-time applications.<n>Prediction-based feature caching is widely used to accelerate diffusion transformers; however, as the number of steps increases, the deviation between its predictions and the reference full-compute trajectory gradually grows.<n>An intuitive idea is to use an online regression model to dynamically correct this deviation, but it faces the issue of label data being unavailable during the acceleration process.<n>This paper presents a statistical observation that the residuals between the features of full computation steps using caching methods and reference full-compute trajectory locally exhibit a
  arXiv  Detail & Related papers  (2026-09-05T08:54:54Z)
• Geometric and Behavioral Stratification in Transformer Residual Streams [0.0]
  Trained transformer models develop privileged bases: coordinate axes whose statistics differ from the rest of the residual stream.<n>We investigate the prediction direction, the unembedding direction of the token a model currently predicts, and find that it functions as a content-defined privileged anchor.
  arXiv  Detail & Related papers  (2026-08-12T17:42:20Z)
• When Do Surrogate Updates Improve Decisions? A Local Theory of Trajectory-Wise Transfer [0.0]
  We ask when one step of trajectory training reduces both population surrogate loss and decision risk, and how transfer accumulates along repeated updates.<n>Controlled gridworld and LLM post-training experiments yield results consistent with our predictions.
  arXiv  Detail & Related papers  (2026-08-02T10:12:42Z)
• Stabilizing Extrapolation in Looped Transformers via Learned Stochastic Stopping [56.14767235650558]
  Looped Transformers, which repeatedly apply a shared transformer block, are an architecturally natural fit for variable-length computation tasks.<n>We trace this variance to the spurious correlation in simple algorithmic tasks between sequence length and number of loops.<n>Our work suggests that "when to stop" should be treated as a training design choice, not merely an inference-time-allocation rule.
  arXiv  Detail & Related papers  (2026-06-29T08:58:09Z)
• Controlling Transient Amplification Improves Long-horizon Rollouts [33.85418321772749]
  We identify transient perturbations around rollout trajectories as a mechanism driving rollout error.<n>We propose commutativity regularization: a combination of two penalties designed to reduce the normality defect of individual Jacobians.<n>We show that the method improves FourNet climate forecasts on ERA5 without using any new data.
  arXiv  Detail & Related papers  (2026-05-09T10:10:30Z)
• ResAD: Normalized Residual Trajectory Modeling for End-to-End Autonomous Driving [64.42138266293202]
  ResAD is a Normalized Residual Trajectory Modeling framework.<n>It reframes the learning task to predict the residual deviation from an inertial reference.<n>On the NAVSIM benchmark, ResAD achieves a state-of-the-art PDMS of 88.6 using a vanilla diffusion policy.
  arXiv  Detail & Related papers  (2025-10-09T17:59:36Z)
• Laplacian Autoencoders for Learning Stochastic Representations [0.6999740786886537]
  We present a Bayesian autoencoder for unsupervised representation learning, which is trained using a novel variational lower-bound of the autoencoder evidence. We show that our Laplacian autoencoder estimates well-calibrated uncertainties in both latent and output space.
  arXiv  Detail & Related papers  (2022-06-30T07:23:16Z)
• Differentiable Annealed Importance Sampling and the Perils of Gradient Noise [68.44523807580438]
  Annealed importance sampling (AIS) and related algorithms are highly effective tools for marginal likelihood estimation. Differentiability is a desirable property as it would admit the possibility of optimizing marginal likelihood as an objective. We propose a differentiable algorithm by abandoning Metropolis-Hastings steps, which further unlocks mini-batch computation.
  arXiv  Detail & Related papers  (2021-07-21T17:10:14Z)
• Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift [81.74795324629712]
  We call prediction-time batch normalization, which significantly improves model accuracy and calibration under covariate shift. We show that prediction-time batch normalization provides complementary benefits to existing state-of-the-art approaches for improving robustness. The method has mixed results when used alongside pre-training, and does not seem to perform as well under more natural types of dataset shift.
  arXiv  Detail & Related papers  (2020-06-19T05:08:43Z)
• Path Sample-Analytic Gradient Estimators for Stochastic Binary Networks [78.76880041670904]
  In neural networks with binary activations and or binary weights the training by gradient descent is complicated. We propose a new method for this estimation problem combining sampling and analytic approximation steps. We experimentally show higher accuracy in gradient estimation and demonstrate a more stable and better performing training in deep convolutional models.
  arXiv  Detail & Related papers  (2020-06-04T21:51:21Z)
• Explore Aggressively, Update Conservatively: Stochastic Extragradient Methods with Variable Stepsize Scaling [34.35013145885164]
  Extragradient methods have become a staple for solving large-scale saddlepoint problems in machine learning. We show in this paper that running vanilla extragradient with gradients may jeopardize its convergence, even in simple bilinear models. We show that this modification allows the method to converge even with gradients, and we derive sharp convergence rates under an error bound condition.
  arXiv  Detail & Related papers  (2020-03-23T10:24:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.