The Topological Trouble With Transformers

• URL: http://arxiv.org/abs/2604.17121v1
• Date: Sat, 18 Apr 2026 19:46:30 GMT
• Title: The Topological Trouble With Transformers
• Abstract summary: Transformers encode structure in sequences via an expanding contextual history.<n>Their purely feedforward architecture fundamentally limits dynamic state tracking.<n>We argue that temporally extended cognition requires refocusing from explicit thought traces to implicit activation dynamics.
• Score: 12.229239113414236
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformers encode structure in sequences via an expanding contextual history. However, their purely feedforward architecture fundamentally limits dynamic state tracking. State tracking -- the iterative updating of latent variables reflecting an evolving environment -- involves inherently sequential dependencies that feedforward networks struggle to maintain. Consequently, feedforward models push evolving state representations deeper into their layer stack with each new input step, rendering information inaccessible in shallow layers and ultimately exhausting the model's depth. While this depth limit can be bypassed by dynamic depth models and by explicit or latent thinking that externalizes state representations, these solutions are computationally and memory inefficient. In this article, we argue that temporally extended cognition requires refocusing from explicit thought traces to implicit activation dynamics via recurrent architectures. We introduce a taxonomy of recurrent and continuous-thought transformer architectures, categorizing them by their recurrence axis (depth versus step) and their ratio of input tokens to recurrence steps. Finally, we outline promising research directions, including enhanced state-space models and coarse-grained recurrence, to better integrate state tracking into modern foundation models.

Related papers

• ARDepth: Auto-regressive Monocular Depth Estimation with Progressive Visual Conditioning [66.42476564611607]
  Diffusion models assume that depth can be recovered as a globally smooth field through iterative denoising.<n>We introduce ARDepth, which formulates depth estimation as structured auto-regressive generation.<n>We show that our approach achieves strong performance and produces structurally consistent depth predictions across scales.
  arXiv  Detail & Related papers  (2026-07-14T07:06:46Z)
• Do Agents Think Deeper? A Mechanistic Investigation of Layer-Wise Dynamics in Sequential Planning [6.5668779922726666]
  Large language models (LLMs) may utilize their depth inefficiently in standard single-turn tasks.<n>We show that agentic reasoning exhibits a distinct depth profile from static tasks.
  arXiv  Detail & Related papers  (2026-05-27T04:08:08Z)
• Thinking While Listening: Fast-Slow Recurrence for Long-Horizon Sequential Modeling [39.56521605534737]
  We extend the recent latent recurrent modeling to sequential input streams.<n>By interleaving fast, recurrent latent updates with self-organizational ability, our method facilitates the learning of stable internal structures that evolve alongside the input.
  arXiv  Detail & Related papers  (2026-04-02T03:45:13Z)
• RainDiff: End-to-end Precipitation Nowcasting Via Token-wise Attention Diffusion [64.49056527678606]
  We propose a Token-wise Attention integrated into not only the U-Net diffusion model but also the radar-temporal encoder.<n>Unlike prior approaches, our method integrates attention into the architecture without incurring the high resource cost typical of pixel-space diffusion.<n>Our experiments and evaluations demonstrate that the proposed method significantly outperforms state-of-the-art approaches, robustness local fidelity, generalization, and superior in complex precipitation forecasting scenarios.
  arXiv  Detail & Related papers  (2025-10-16T17:59:13Z)
• Learning by Steering the Neural Dynamics: A Statistical Mechanics Perspective [0.0]
  We study how neural dynamics can support fully local, distributed learning.<n>We propose a biologically plausible algorithm for supervised learning with any binary recurrent network.
  arXiv  Detail & Related papers  (2025-10-13T22:28:34Z)
• Neural Network Reprogrammability: A Unified Theme on Model Reprogramming, Prompt Tuning, and Prompt Instruction [57.19302613163439]
  We introduce neural network reprogrammability as a unifying framework for model adaptation.<n>We present a taxonomy that categorizes such information manipulation approaches across four key dimensions.<n>We also analyze remaining technical challenges and ethical considerations.
  arXiv  Detail & Related papers  (2025-06-05T05:42:27Z)
• ReMatching Dynamic Reconstruction Flow [55.272357926111454]
  This work introduces the ReMatching framework, designed to improve reconstruction quality by incorporating deformation priors into dynamic reconstruction models.<n>Our approach advocates for velocity-field based priors, for which we suggest a matching procedure that can seamlessly supplement existing dynamic reconstruction pipelines.<n>Our evaluations on popular benchmarks involving both synthetic and real-world dynamic scenes demonstrate that augmenting current state-of-the-art methods with our approach leads to a clear improvement in reconstruction accuracy.
  arXiv  Detail & Related papers  (2024-11-01T16:09:33Z)
• State-Space Modeling in Long Sequence Processing: A Survey on Recurrence in the Transformer Era [59.279784235147254]
  This survey provides an in-depth summary of the latest approaches that are based on recurrent models for sequential data processing.<n>The emerging picture suggests that there is room for exploring novel routes, constituted by learning algorithms that depart from the standard Backpropagation Through Time.
  arXiv  Detail & Related papers  (2024-06-13T12:51:22Z)
• Todyformer: Towards Holistic Dynamic Graph Transformers with Structure-Aware Tokenization [6.799413002613627]
  Todyformer is a novel Transformer-based neural network tailored for dynamic graphs. It unifies the local encoding capacity of Message-Passing Neural Networks (MPNNs) with the global encoding of Transformers. We show that Todyformer consistently outperforms the state-of-the-art methods for downstream tasks.
  arXiv  Detail & Related papers  (2024-02-02T23:05:30Z)
• Latent Traversals in Generative Models as Potential Flows [113.4232528843775]
  We propose to model latent structures with a learned dynamic potential landscape. Inspired by physics, optimal transport, and neuroscience, these potential landscapes are learned as physically realistic partial differential equations. Our method achieves both more qualitatively and quantitatively disentangled trajectories than state-of-the-art baselines.
  arXiv  Detail & Related papers  (2023-04-25T15:53:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.