Adaptive Depth in Looped Transformers: Diagnosing Learned Halting Gates and Trajectory Readouts

• URL: http://arxiv.org/abs/2607.20519v1
• Date: Wed, 08 Jul 2026 02:54:03 GMT
• Title: Adaptive Depth in Looped Transformers: Diagnosing Learned Halting Gates and Trajectory Readouts
• Abstract summary: We study adaptive depth in looped Transformers through a trajectory--readout lens.<n>We find that fixed-prior depth supervision produces difficulty-aware trajectories whose intermediate states expose useful stopping signals.<n>Our systematic diagnostic evaluation reframes adaptive depth in looped Transformers as a joint problem of trajectory formation and exit readout.
• Score: 16.36140240070991
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Looped Transformers increase test-time computation by repeatedly applying a shared recurrent block. Learned halting objectives in looped Transformers typically use a single exit distribution both as the inference-time stopping rule and as the training-time weighting of per-depth losses. This entangles exit selection with trajectory formation: the gate not only chooses which recurrent state to use, but also determines how strongly each intermediate state is supervised. Consequently, poor adaptive-compute performance can arise from the readout, the induced trajectory, or their interaction. We study adaptive depth in looped Transformers through this trajectory--readout lens, across controlled synthetic tasks (modular arithmetic and binary parity) and large-scale Ouro-1.4B and 2.6B checkpoints. We find that fixed-prior depth supervision, which shapes the trajectory without an input-dependent halting policy, produces difficulty-aware trajectories whose intermediate states expose useful stopping signals, and that simple post-hoc confidence readouts often match or outperform learned linear and MLP gates. Fitting gates on frozen trajectories localizes the failure: it appears to stem mainly from the trajectory induced by joint gate training rather than from limited gate expressivity. The same pattern is present in Ouro evaluations, where pretrained ponder gates are competitive but not uniformly Pareto-optimal, and measured latency confirms that the resulting reductions in average exit depth translate into practical inference-time savings. Our systematic diagnostic evaluation reframes adaptive depth in looped Transformers as a joint problem of trajectory formation and exit readout, rather than gate learning alone, highlighting a distinction that prior learned-halting work has often left implicit.

Related papers

• UniCycleFlow: Bidirectional Unpaired Image Translation with a Shared Rectified Flow [14.60117153247222]
  Bidirectional unpaired image translation must preserve source-specific structure while learning coherent transformations in both directions without paired supervision.<n>We propose UniCycleFlow, a rectified-flow framework that represents bidirectional translation as forward and reverse integration of a single time-conditioned velocity field.
  arXiv  Detail & Related papers  (2026-08-07T04:05:04Z)
• Stabilizing Extrapolation in Looped Transformers via Learned Stochastic Stopping [56.14767235650558]
  Looped Transformers, which repeatedly apply a shared transformer block, are an architecturally natural fit for variable-length computation tasks.<n>We trace this variance to the spurious correlation in simple algorithmic tasks between sequence length and number of loops.<n>Our work suggests that "when to stop" should be treated as a training design choice, not merely an inference-time-allocation rule.
  arXiv  Detail & Related papers  (2026-06-29T08:58:09Z)
• Same Signal, Opposite Meaning: Direction-Informed Adaptive Learning for LLM Agents [24.71914991820339]
  LLM agents aim to invoke extra computation only when it improves performance.<n>Existing methods typically use confidence-, uncertainty-, or difficulty-based gates, assuming a fixed direction from the gating signal through compute need to the value of computation.<n>We show that this alignment is unstable, with reversals across environments and backbones even when the task is fixed.<n>We propose DIAL (Direction-Informed Adaptive Learning), a sparse gate trained from signal-agnostic counterfactual exploration to learn the utility direction of state features per.
  arXiv  Detail & Related papers  (2026-05-07T20:15:51Z)
• Temporal Attention for Adaptive Control of Euler-Lagrange Systems with Unobservable Memory [1.2891210250935148]
  The paper proposes a meta-control architecture in which the gains of a computed-torque controller are generated by a self-attention block.<n>The number of attention heads is selected before policy training through a surrogate analysis of the memory-state gradient.<n>The approach is tested on a 2-DOF manipulator with nonlinear friction and variable payload.
  arXiv  Detail & Related papers  (2026-05-07T19:25:18Z)
• Meta-Adaptive Beam Search Planning for Transformer-Based Reinforcement Learning Control of UAVs with Overhead Manipulators under Flight Disturbances [8.618483849755604]
  A drone equipped with overhead manipulators offer unique capabilities for inspection, maintenance, and contact-based interaction.<n>The motion of the drone and its manipulator is tightly linked, and even small attitude changes caused by wind or control imperfections shift the end-effector away from its intended path.<n>We develop a reinforcement-learning framework with a transformer-based double deep Q learning (DDQN)<n>This allows the controller to anticipate the end-effector's motion through simulated rollouts rather than executing those actions directly on the actual model.
  arXiv  Detail & Related papers  (2026-03-27T17:08:40Z)
• Verifying Closed-Loop Contractivity of Learning-Based Controllers via Partitioning [52.23804865017831]
  We address the problem of verifying closed-loop contraction in nonlinear control systems whose controller and contraction metric are both parameterized by neural networks.<n>We derive a tractable and scalable sufficient condition for closed-loop contractivity that reduces to checking that the dominant eigenvalue of a symmetric Metzler matrix is nonpositive.
  arXiv  Detail & Related papers  (2025-12-01T23:06:56Z)
• Directional Optimization Asymmetry in Transformers: A Synthetic Stress Test [0.15229257192293197]
  Transformers are theoretically reversal-invariant: their function class does not prefer left-to-right over right-to-left mappings.<n>Recent work on temporal asymmetry in LLMs suggests that real-world corpora carry their own arrow of time.<n>This leaves an unresolved question: do directional failures stem from linguistic statistics, or from the architecture itself?
  arXiv  Detail & Related papers  (2025-11-25T07:03:20Z)
• How Transformers Learn Regular Language Recognition: A Theoretical Study on Training Dynamics and Implicit Bias [48.9399496805422]
  We focus on two representative tasks in the category of regular language recognition, known as even pairs' and parity check'<n>Our goal is to explore how a one-layer transformer, consisting of an attention layer followed by a linear layer, learns to solve these tasks.
  arXiv  Detail & Related papers  (2025-05-02T00:07:35Z)
• Time-series Generation by Contrastive Imitation [87.51882102248395]
  We study a generative framework that seeks to combine the strengths of both: Motivated by a moment-matching objective to mitigate compounding error, we optimize a local (but forward-looking) transition policy. At inference, the learned policy serves as the generator for iterative sampling, and the learned energy serves as a trajectory-level measure for evaluating sample quality.
  arXiv  Detail & Related papers  (2023-11-02T16:45:25Z)
• Semi-DETR: Semi-Supervised Object Detection with Detection Transformers [105.45018934087076]
  We analyze the DETR-based framework on semi-supervised object detection (SSOD) We present Semi-DETR, the first transformer-based end-to-end semi-supervised object detector. Our method outperforms all state-of-the-art methods by clear margins.
  arXiv  Detail & Related papers  (2023-07-16T16:32:14Z)
• Accurate methods for the analysis of strong-drive effects in parametric gates [94.70553167084388]
  We show how to efficiently extract gate parameters using exact numerics and a perturbative analytical approach. We identify optimal regimes of operation for different types of gates including $i$SWAP, controlled-Z, and CNOT.
  arXiv  Detail & Related papers  (2021-07-06T02:02:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.