Scaling Limits of Long-Context Transformers

• URL: http://arxiv.org/abs/2605.08505v1
• Date: Fri, 08 May 2026 21:39:31 GMT
• Title: Scaling Limits of Long-Context Transformers
• Abstract summary: We study the long-context limit of softmax self-attention with a fixed query and a random context.<n>We show that the critical scale at which selectivity emerges is determined by the local exponent of the distance-to-query distribution near zero.
• Score: 24.95451131347537
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study the long-context limit of softmax self-attention with a fixed query and a random context of $n$ i.i.d. keys on the sphere, viewing the inverse temperature $β_n$ as the scaling parameter that decides whether attention degenerates into uniform averaging or collapses onto the single closest key. We show that the critical scale at which selectivity emerges is determined by the local exponent of the distance-to-query distribution near zero rather than by global features of the context, and scales like $β_n^\ast \asymp n^{2/(d-1)}$ for uniform keys on $\mathbb{S}^{d-1}$. Furthermore, we characterize the limiting laws of the ordered attention weights and of the attention output across all regimes of $β_n$: a subcritical regime in which the output reduces to a local average around $q$ with explicit deterministic bias and Gaussian fluctuations; a critical regime in which a finite collection of nearest keys retains macroscopic mass without single-key collapse; and a supercritical regime in which all mass concentrates on the closest key. Of notable interest is the subcritical case with identity value matrix where the attention map approximately implements a backward heat equation.

Related papers

• Robust Average-Reward Markov Decision Processes: Minimax-Optimal Learning via Plug-in Reductions [51.50375419691955]
  Distributionally robust Markov decision processes provide a principled framework for sequential decision making under model uncertainty.<n>We study how many samples are necessary and sufficient to learn an $varepsilon$-optimal robust policy under the average-reward criterion.
  arXiv  Detail & Related papers  (2026-08-06T19:49:48Z)
• Bayesian Best-Arm Identification with Abstention: A Polynomial-to-Exponential Phase Transition [51.253617466579335]
  We study the Bayesian fixed-budget best-arm identification problem in which a learner can abstain from making a terminal recommendation.<n>We analyze the probability of undetected error--the risk of recommending a suboptimal arm without abstaining.
  arXiv  Detail & Related papers  (2026-06-28T05:07:27Z)
• A Unified Framework for Critical Scaling of Inverse Temperature in Self-Attention [15.584040836686043]
  We show that the desirable scale is determined by the gap-counting function $N_n$ of each attention row.<n> Counting how many competitors lie within each gap from the maximum, we define an upper-tail accumulation scale.<n>This framework unifies prior scaling laws as different $N_n$ and yields a direct diagnostic for attention-score families.
  arXiv  Detail & Related papers  (2026-05-12T19:48:36Z)
• Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime [0.9974630621313314]
  We study the evolution of tokens in deep encoder-only transformers at inference time.<n>We prove that the token distribution rapidly concentrates onto the push-forward of the initial distribution.
  arXiv  Detail & Related papers  (2026-05-11T17:58:14Z)
• Extreme-Value Criticality and Gain Decomposition at the Integer Quantum Hall Transition [6.416957991576607]
  We show that the maximal wave-function amplitude separates into a global gain and an intrinsic extreme component.<n>Extreme observables provide a robust probe of correlated criticality in open quantum systems.
  arXiv  Detail & Related papers  (2026-03-16T13:52:41Z)
• Regularized Online RLHF with Generalized Bilinear Preferences [68.44113000390544]
  We consider the problem of contextual online RLHF with general preferences.<n>We adopt the Generalized Bilinear Preference Model to capture preferences via low-rank, skew-symmetric matrices.<n>We prove that the dual gap of the greedy policy is bounded by the square of the estimation error.
  arXiv  Detail & Related papers  (2026-02-26T15:27:53Z)
• Near-Optimal Clustering in Mixture of Markov Chains [74.3828414695655]
  We study the problem of clustering $T$ trajectories of length $H$, each generated by one of $K$ unknown ergodic Markov chains over a finite state space of size $S$.<n>We derive an instance-dependent, high-probability lower bound on the clustering error rate, governed by the weighted KL divergence between the transition kernels of the chains.<n>We then present a novel two-stage clustering algorithm.
  arXiv  Detail & Related papers  (2025-06-02T05:10:40Z)
• Quantum Supercritical Crossover with Dynamical Singularity [2.9659182523095047]
  We extend this notable concept of supercriticality from classical to quantum systems near the quantum critical point.<n>We reveal the existence of quantum supercritical (QSC) crossover lines, determined by not only response functions but also quantum information quantities.<n>Our work paves the way for exploring QSC crossovers in and out of equilibrium in quantum many-body systems.
  arXiv  Detail & Related papers  (2024-07-07T17:52:02Z)
• KPZ scaling from the Krylov space [83.88591755871734]
  Recently, a superdiffusion exhibiting the Kardar-Parisi-Zhang scaling in late-time correlators and autocorrelators has been reported. Inspired by these results, we explore the KPZ scaling in correlation functions using their realization in the Krylov operator basis.
  arXiv  Detail & Related papers  (2024-06-04T20:57:59Z)
• Quantum-critical properties of the one- and two-dimensional random transverse-field Ising model from large-scale quantum Monte Carlo simulations [0.0]
  We study the ferromagnetic transverse-field Ising model with quenched disorder at $T = 0$ in one and two dimensions. The emphasis on effective zero-temperature simulations resolves several inconsistencies in existing literature.
  arXiv  Detail & Related papers  (2024-03-08T11:20:42Z)
• Average entanglement entropy of midspectrum eigenstates of quantum-chaotic interacting Hamiltonians [0.0]
  We show that the magnitude of the negative $O(1)$ correction is only slightly greater than the one predicted for random pure states. We derive a simple expression that describes the numerically observed $nu$ dependence of the $O(1)$ deviation from the prediction for random pure states.
  arXiv  Detail & Related papers  (2023-03-23T18:00:02Z)
• Optimal and instance-dependent guarantees for Markovian linear stochastic approximation [47.912511426974376]
  We show a non-asymptotic bound of the order $t_mathrmmix tfracdn$ on the squared error of the last iterate of a standard scheme. We derive corollaries of these results for policy evaluation with Markov noise.
  arXiv  Detail & Related papers  (2021-12-23T18:47:50Z)
• Optimal policy evaluation using kernel-based temporal difference methods [78.83926562536791]
  We use kernel Hilbert spaces for estimating the value function of an infinite-horizon discounted Markov reward process. We derive a non-asymptotic upper bound on the error with explicit dependence on the eigenvalues of the associated kernel operator. We prove minimax lower bounds over sub-classes of MRPs.
  arXiv  Detail & Related papers  (2021-09-24T14:48:20Z)
• On Linear Stochastic Approximation: Fine-grained Polyak-Ruppert and Non-Asymptotic Concentration [115.1954841020189]
  We study the inequality and non-asymptotic properties of approximation procedures with Polyak-Ruppert averaging. We prove a central limit theorem (CLT) for the averaged iterates with fixed step size and number of iterations going to infinity.
  arXiv  Detail & Related papers  (2020-04-09T17:54:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.