Looped Transformers with Source-Centered State Evolution

• URL: http://arxiv.org/abs/2607.27656v1
• Date: Thu, 30 Jul 2026 04:13:58 GMT
• Title: Looped Transformers with Source-Centered State Evolution
• Abstract summary: Looped Transformers create a useful train- and test-time compute axis by reusing the same Transformer block over recurrent depth.<n>In additive-injection looped Transformers, an input-conditioned signal is reintroduced at every recurrent step.<n>We propose Source-Centered State Evolution to reconcile input conditioning with reference-preserving shared recurrence.
• Score: 38.624504791098545
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Looped Transformers create a useful train- and test-time compute axis by reusing the same Transformer block over recurrent depth, increasing effective depth at a fixed parameter count. However, that shared block must then govern an entire trajectory of varying hidden states over trained and extrapolated depths. Furthermore, in additive-injection looped Transformers, an input-conditioned signal is reintroduced at every recurrent step, so applying the shared transition at an input-conditioned reference can still move the hidden state. In this paper, we propose Source-Centered State Evolution (SCSE), which is designed to reconcile input conditioning with reference-preserving shared recurrence. Specifically, SCSE retains input dependence through its learned anchor and initial deviation, allows nonzero deviations to drive recurrent computation while mapping zero deviation to zero, and guarantees exact anchor invariance through its zero-deviation mask. The designated anchor is thereby a one-step fixed point by construction. The zero-deviation forcing bias is the next deviation produced from the anchor itself and vanishes in SCSE, while nonzero deviations remain active and support state-dependent recurrent computation. Our theory shows that the zero-deviation forcing bias is a design degree of freedom whose task effect can be harmful, neutral, or beneficial; SCSE resolves this choice in favor of exact anchor invariance by setting the bias to zero. Across WikiText-2, WikiText-103, direct web-corpus pretraining, held-out web-text transfer, and LAMBADA completion, SCSE improves the controlled recurrent quality frontier. Ablation studies identify the learned anchor and the anchor-coordinate deviation recurrence as the primary contributors to the gain, and a trained-model case study grounds the anchor-response diagnostic in observed recurrent motion.

Related papers

• Scale-Consistent Posterior Dynamics for Diffusion Inverse Problems [16.685352002333225]
  Posterior sampling with a pretrained diffusion prior is governed by a conditional score whose likelihood component is generally intractable.<n>We show that a noise-conditioned path whose targets approach the clean posterior produces a continuous surrogate SDE.<n>Experiments on FFHQ and ImageNet with 100 score evaluations demonstrate competitive reconstruction fidelity for superresolution and deblurring.
  arXiv  Detail & Related papers  (2026-08-15T09:38:41Z)
• Convex-Hull-Neighborhood Smooth Dual Generalization: Controlling Local Correction Propagation in Offline RL [7.803284875836737]
  bootstrapping can amplify estimation errors in offline reinforcement learning.<n>We propose Convex Hull Neighborhood Smooth Dual Generalization, which expresses the Bellman backup as an in-sample value target plus a CHN-local correction.<n> Experiments on Gym-MuJoCo and AntMaze show strong aggregate performance and stable value estimation.
  arXiv  Detail & Related papers  (2026-08-04T04:30:02Z)
• An Adjoint-Sensitivity Framework for Lost-in-the-Middle Phenomena in Causal Residual Transformers [1.4180331276028662]
  We develop an adjoint-sensitivity framework for positional influence in causal residual Transformers.<n>We state boundary advantages under independently checkable energy, correlation, and local-channel bounds.
  arXiv  Detail & Related papers  (2026-07-20T08:44:31Z)
• Branch-resolved Pauli-block spectroscopy of residual conditional phase in two-qubit gates [0.0]
  We introduce branch-resolved Pauli-block spectroscopy to estimate the per-cycle residual ZZ-rotation angle theta_c with its sign.<n>The approach yields a low-overhead, signed per-cycle estimate of residual conditional phase that standard fidelity benchmarks underresolve at leading order.
  arXiv  Detail & Related papers  (2026-07-11T10:32:05Z)
• Stabilizing Extrapolation in Looped Transformers via Learned Stochastic Stopping [56.14767235650558]
  Looped Transformers, which repeatedly apply a shared transformer block, are an architecturally natural fit for variable-length computation tasks.<n>We trace this variance to the spurious correlation in simple algorithmic tasks between sequence length and number of loops.<n>Our work suggests that "when to stop" should be treated as a training design choice, not merely an inference-time-allocation rule.
  arXiv  Detail & Related papers  (2026-06-29T08:58:09Z)
• A Doeblin-Anchored Contrastive Chart for Learning Markov Transition Kernels [1.2691047660244335]
  This paper introduces a statistical-to-dynamical coordinate framework for learning transition kernels from contrastive objectives.<n>We prove that the anchored contrastive risk identifies the anchored transition density and calibrates excess risk to density error.
  arXiv  Detail & Related papers  (2026-06-01T13:26:44Z)
• Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior [107.2098567818173]
  Latent Recurrent Transformer (LRT) is a lightweight augmentation of autoregressive transformers.<n>LRT reuses a high-level source-layer hidden state from the previous token as recurrent memory for the next token.
  arXiv  Detail & Related papers  (2026-05-26T10:10:26Z)
• Learning from Emptiness: De-biasing Listwise Rerankers with Content-Agnostic Probability Calibration [76.08899010904652]
  CapCal is a training-free framework that mechanically decouples positional bias from ranking decisions.<n>It achieves superior performance among training-free methods while preserving single-pass efficiency.
  arXiv  Detail & Related papers  (2026-04-11T10:47:22Z)
• Diffusion Controller: Framework, Algorithms and Parameterization [54.82539154511621]
  We present a unified control-theoretic view that casts reverse diffusion sampling as state-only control within (generalized) linearly-solvable Markov Decision Processes.<n>Under this framework, control acts by reweighting the pretrained reverse-time transition kernels, balancing terminal objectives against an $f$-divergence cost.<n>Experiments on Stable Diffusion v1.4 show consistent gains in preference-alignment win rates and improved quality-efficiency trade-offs.
  arXiv  Detail & Related papers  (2026-03-07T01:49:59Z)
• Deterministic Continuous Replacement: Fast and Stable Module Replacement in Pretrained Transformers [0.7829352305480287]
  Cold-start reinitialization destabilizes frozen backbones. We isolate this core stability challenge in a controlled study.<n>In a single-seed study, DCR attains faster convergence and stronger alignment than gating and distillation baselines on controlled attention replacement, establishing a foundation for heterogeneous operator swaps.
  arXiv  Detail & Related papers  (2025-11-24T00:55:14Z)
• PseudoNeg-MAE: Self-Supervised Point Cloud Learning using Conditional Pseudo-Negative Embeddings [55.55445978692678]
  PseudoNeg-MAE enhances global feature representation of point cloud masked autoencoders by making them both discriminative and sensitive to transformations.<n>We propose a novel loss that explicitly penalizes invariant collapse, enabling the network to capture richer transformation cues while preserving discriminative representations.
  arXiv  Detail & Related papers  (2024-09-24T07:57:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.