A Mechanistic Analysis of Looped Reasoning Language Models

• URL: http://arxiv.org/abs/2604.11791v1
• Date: Mon, 13 Apr 2026 17:55:36 GMT
• Title: A Mechanistic Analysis of Looped Reasoning Language Models
• Abstract summary: We conduct a mechanistic analysis of the latent states in looped language models.<n>We show that for many of the studied models each layer in the cycle converges to a distinct fixed point.<n>We discover that recurrent blocks learn stages of inference that closely mirror those of feedforward models.
• Score: 41.640597313586525
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reasoning has become a central capability in large language models. Recent research has shown that reasoning performance can be improved by looping an LLM's layers in the latent dimension, resulting in looped reasoning language models. Despite promising results, few works have investigated how their internal dynamics differ from those of standard feedforward models. In this paper, we conduct a mechanistic analysis of the latent states in looped language models, focusing in particular on how the stages of inference observed in feedforward models compare to those observed in looped ones. To this end, we analyze cyclic recurrence and show that for many of the studied models each layer in the cycle converges to a distinct fixed point; consequently, the recurrent block follows a consistent cyclic trajectory in the latent space. We provide evidence that as these fixed points are reached, attention-head behavior stabilizes, leading to constant behavior across recurrences. Empirically, we discover that recurrent blocks learn stages of inference that closely mirror those of feedforward models, repeating these stages in depth with each iteration. We study how recurrent block size, input injection, and normalization influence the emergence and stability of these cyclic fixed points. We believe these findings help translate mechanistic insights into practical guidance for architectural design.

Related papers

• Answer-Distribution Trajectories: A Stochastic-Dynamics View of LLM Reasoning [29.58875226713171]
  Chain-of-thought reasoning provides a structured computation between a model's input and final answer.<n> entropy profiles track how uncertainty evolves over the reasoning process but do not reveal which competing hypotheses account for that uncertainty.<n>We introduce answer-distribution trajectories, a representation that tracks the model's full predictive distribution over answers as reasoning unfolds.
  arXiv  Detail & Related papers  (2026-09-08T16:58:40Z)
• Tracking Representation Dynamics in Large Language Models with Persistent Homology [5.665477411284105]
  We study alignment dynamics using persistent homology.<n>We find that the majority of topological reorganization occurs during the earliest stages of training.<n>Different alignment objectives induce distinguishable topological trajectories, while instruction-tuned and pretrained models exhibit qualitatively different patterns of evolution.
  arXiv  Detail & Related papers  (2026-06-17T19:35:10Z)
• Grokking of Diffusion Models: Case Study on Modular Addition [20.821116485381154]
  We show that diffusion models trained with flow-matching objectives exhibit grokking--delayed generalization after overfitting--on modular addition.<n>Our work provides the mechanistic decomposition of algorithmic learning in diffusion models.
  arXiv  Detail & Related papers  (2026-04-20T00:02:00Z)
• Demystifing Video Reasoning [71.53763299316041]
  We show that reasoning in video models instead primarily emerges along the diffusion denoising steps.<n>We identify several emergent reasoning behaviors critical to model performance.<n>Motivated by these insights, we present a training-free strategy as a proof-of-concept.
  arXiv  Detail & Related papers  (2026-03-17T17:59:55Z)
• Dynamics Within Latent Chain-of-Thought: An Empirical Study of Causal Structure [58.89643769707751]
  We study latent chain-of-thought as a manipulable causal process in representation space.<n>We find that latent-step budgets behave less like homogeneous extra depth and more like staged functionality with non-local routing.<n>These results motivate mode-conditional and stability-aware analyses as more reliable tools for interpreting and improving latent reasoning systems.
  arXiv  Detail & Related papers  (2026-02-09T15:25:12Z)
• Circular Reasoning: Understanding Self-Reinforcing Loops in Large Reasoning Models [66.11277323593475]
  Circular Reasoning is a self-reinforcing trap where generated content acts as a logical premise for its own recurrence.<n>Mechanistically, we characterize circular reasoning as a state collapse exhibiting distinct boundaries.<n>We reveal that reasoning impasses trigger the loop onset, which subsequently persists as an inescapable cycle driven by a self-reinforcing V-shaped attention mechanism.
  arXiv  Detail & Related papers  (2026-01-09T10:23:55Z)
• Schoenfeld's Anatomy of Mathematical Reasoning by Language Models [56.656180566692946]
  We adopt Schoenfeld's Episode Theory as an inductive, intermediate-scale lens and introduce ThinkARM (Anatomy of Reasoning in Models)<n>ThinkARM explicitly abstracts reasoning traces into functional reasoning steps such as Analysis, Explore, Implement, verify, etc.<n>We show that episode-level representations make reasoning steps explicit, enabling systematic analysis of how reasoning is structured, stabilized, and altered in modern language models.
  arXiv  Detail & Related papers  (2025-12-23T02:44:25Z)
• A Closer Look at Bias and Chain-of-Thought Faithfulness of Large (Vision) Language Models [58.32070787537946]
  Chain-of-thought (CoT) reasoning enhances performance of large language models.<n>We present the first comprehensive study of CoT faithfulness in large vision-language models.
  arXiv  Detail & Related papers  (2025-05-29T18:55:05Z)
• Towards Efficient Modelling of String Dynamics: A Comparison of State Space and Koopman based Deep Learning Methods [8.654571696634825]
  State Space Models (SSM) and Koopman-based deep learning methods for modelling the dynamics of both linear and non-linear stiff strings. Our findings indicate that our proposed Koopman-based model performs as well as or better than other existing approaches in non-linear cases for long-sequence modelling. This research contributes insights into the physical modelling of dynamical systems by offering a comparative overview of these and previous methods and introducing innovative strategies for model improvement.
  arXiv  Detail & Related papers  (2024-08-29T15:55:27Z)
• Neural Persistence Dynamics [8.197801260302642]
  We consider the problem of learning the dynamics in the topology of time-evolving point clouds. Our proposed model - $textitNeural Persistence Dynamics$ - substantially outperforms the state-of-the-art across a diverse set of parameter regression tasks.
  arXiv  Detail & Related papers  (2024-05-24T17:20:18Z)
• A Fixed-Point Approach for Causal Generative Modeling [20.88890689294816]
  We propose a novel formalism for describing Structural Causal Models (SCMs) as fixed-point problems on causally ordered variables.<n>We establish the weakest known conditions for their unique recovery given the topological ordering (TO)
  arXiv  Detail & Related papers  (2024-04-10T12:29:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.