Kinetic theory for Transformers and the lost-in-the-middle phenomenon

• URL: http://arxiv.org/abs/2605.09213v1
• Date: Sat, 09 May 2026 23:16:19 GMT
• Title: Kinetic theory for Transformers and the lost-in-the-middle phenomenon
• Abstract summary: We study causal self-attention dynamics -- a toy model for decoder Transformers.<n>For iid uniformly distributed tokens, the limiting correlation equation can be solved in closed form.
• Score: 1.0705399532413615
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study causal self-attention dynamics -- a toy model for decoder Transformers -- which we interpret as a non-exchangeable interacting particle system. Adapting cumulant expansions to the triangular causal dependency structure of the model, and appealing to non-hierarchical methods to estimate correlations using Glauber calculus, we prove a quantitative mean-field limit result and a next-order characterization of correlations. For iid uniformly distributed tokens, the limiting correlation equation can be solved in closed form and we obtain a rigorous explanation of the empirically observed \emph{lost-in-the-middle} phenomenon: the token retrieval profile, as a function of the source position in the prompt, is $\mathsf{U}$-shaped, with primacy, recency, and a unique interior minimum under an explicit smallness condition.

Related papers

• A Free Probabilistic Framework for Denoising Diffusion Models: Entropy, Transport, and Reverse Processes [22.56299060022639]
  This paper builds on Voiculescu's theory of free entropy and free Fisher information.<n>We formulate diffusion and quantify reverse processes governed by operator-valued dynamics.<n>The resulting dynamics admit a gradient-flow structure in the noncommutative Wasserstein space.
  arXiv  Detail & Related papers  (2025-10-26T18:03:54Z)
• Loss-Complexity Landscape and Model Structure Functions [53.92822954974537]
  We develop a framework for dualizing the Kolmogorov structure function $h_x(alpha)$.<n>We establish a mathematical analogy between information-theoretic constructs and statistical mechanics.<n>We explicitly prove the Legendre-Fenchel duality between the structure function and free energy.
  arXiv  Detail & Related papers  (2025-07-17T21:31:45Z)
• Overcoming Dimensional Factorization Limits in Discrete Diffusion Models through Quantum Joint Distribution Learning [79.65014491424151]
  We propose a quantum Discrete Denoising Diffusion Probabilistic Model (QD3PM)<n>It enables joint probability learning through diffusion and denoising in exponentially large Hilbert spaces.<n>This paper establishes a new theoretical paradigm in generative models by leveraging the quantum advantage in joint distribution learning.
  arXiv  Detail & Related papers  (2025-05-08T11:48:21Z)
• Weak-valued correlation functions: Insights and precise readout strategies [5.3876722808329935]
  The correlation function in quantum systems plays a vital role in decoding their properties and gaining insights into physical phenomena. By defining weak-valued correlation function, we propose the basic insights and the universal methods for recording them on the apparatus through weak measurement.
  arXiv  Detail & Related papers  (2023-06-07T12:58:44Z)
• Stochastic Modified Equations and Dynamics of Dropout Algorithm [4.811269936680572]
  Dropout is a widely utilized regularization technique in the training of neural networks. Its underlying mechanism and its impact on achieving good abilities remain poorly understood.
  arXiv  Detail & Related papers  (2023-05-25T08:42:25Z)
• A statistical approach to topological entanglement: Boltzmann machine representation of high-order irreducible correlation [6.430262211852815]
  A quantum analog of high-order correlations is the topological entanglement in topologically ordered states of matter at zero temperature. In this work, we propose a statistical interpretation that unifies the two under the same information-theoretic framework.
  arXiv  Detail & Related papers  (2023-02-07T02:49:21Z)
• An Indirect Rate-Distortion Characterization for Semantic Sources: General Model and the Case of Gaussian Observation [83.93224401261068]
  Source model is motivated by the recent surge of interest in the semantic aspect of information. intrinsic state corresponds to the semantic feature of the source, which in general is not observable. Rate-distortion function is the semantic rate-distortion function of the source.
  arXiv  Detail & Related papers  (2022-01-29T02:14:24Z)
• Convex Analysis of the Mean Field Langevin Dynamics [49.66486092259375]
  convergence rate analysis of the mean field Langevin dynamics is presented. $p_q$ associated with the dynamics allows us to develop a convergence theory parallel to classical results in convex optimization.
  arXiv  Detail & Related papers  (2022-01-25T17:13:56Z)
• Emergent fractal phase in energy stratified random models [0.0]
  We study the effects of partial correlations in kinetic hopping terms of long-range random matrix models on their localization properties. We show that any deviation from the completely correlated case leads to the emergent non-ergodic delocalization in the system.
  arXiv  Detail & Related papers  (2021-06-07T18:00:01Z)
• Out-of-time-order correlations and the fine structure of eigenstate thermalisation [58.720142291102135]
  Out-of-time-orderors (OTOCs) have become established as a tool to characterise quantum information dynamics and thermalisation. We show explicitly that the OTOC is indeed a precise tool to explore the fine details of the Eigenstate Thermalisation Hypothesis (ETH) We provide an estimation of the finite-size scaling of $omega_textrmGOE$ for the general class of observables composed of sums of local operators in the infinite-temperature regime.
  arXiv  Detail & Related papers  (2021-03-01T17:51:46Z)
• Structural Causal Models Are (Solvable by) Credal Networks [70.45873402967297]
  Causal inferences can be obtained by standard algorithms for the updating of credal nets. This contribution should be regarded as a systematic approach to represent structural causal models by credal networks. Experiments show that approximate algorithms for credal networks can immediately be used to do causal inference in real-size problems.
  arXiv  Detail & Related papers  (2020-08-02T11:19:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.