Clustered Attractor Manifolds and Dynamical Condensation in Self-Attention

• URL: http://arxiv.org/abs/2608.08922v1
• Date: Sun, 09 Aug 2026 21:28:01 GMT
• Title: Clustered Attractor Manifolds and Dynamical Condensation in Self-Attention
• Abstract summary: Transformer layers generate state-dependent interaction networks.<n> token representations determine the attention matrix, which in turn updates the representations.<n>We study this feedback in a minimal normalized self-attention dynamics.
• Score: 2.4792581096761404
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transformer layers generate state-dependent interaction networks: token representations determine the attention matrix, which in turn updates the representations. We study this feedback in a minimal normalized self-attention dynamics and identify the overlap gap as the central quantity governing its attractor structure in the thermodynamic limit. When tokens form internally aligned clusters and their similarity to members of the same cluster exceeds that to every other cluster by a nonvanishing amount, inter-cluster attention is exponentially suppressed as the dimension increases. This mechanism produces a high-dimensional manifold of clustered fixed points, ranging from a few macroscopic clusters to extensive microscopic fragmentation, and also controls their stability against perturbations. Starting from an unstructured Gaussian state, we find that clustered states nucleate from the diffuse background only above a finite threshold in attention sharpness, giving rise to a dynamical attention-condensation transition.

Related papers

• The Structural Origin of Attention Sink: Variance Discrepancy, Super Neurons, and Dimension Disparity [22.040532283910522]
  This work provides a textitmechanistic explanation for the phenomenon.<n>We trace its root to the value aggregation process inherent in self-attention.<n>As a proof of concept, we propose textithead-wise RMSNorm, an architectural modification that stabilizes value aggregation outputs during pre-training.
  arXiv  Detail & Related papers  (2026-05-07T17:28:55Z)
• Gradient Flow Structure and Quantitative Dynamics of Multi-Head Self-Attention [0.0]
  Transformer self-attention can be interpreted as a gradient flow on the unit sphere.<n>We develop a theoretical framework for multi-head self-attention dynamics.
  arXiv  Detail & Related papers  (2026-05-05T20:31:11Z)
• Introducing the Correlation Concentration Ratio (CCR): Quantitative Framework for Comparing Quantum Cluster States [0.0]
  The process of generating cluster states with linear, square, and T-shaped topologies has been systematically modeled.<n>The simulation results show that the cluster graph structure is directly reflected in the pattern of quadrature correlations.<n>A metric titled CCR is introduced that quantifies the concentration of effective correlations on the graph edges relative to the total correlations of the system.
  arXiv  Detail & Related papers  (2026-04-25T11:50:43Z)
• Weight-Informed Self-Explaining Clustering for Mixed-Type Tabular Data [63.62853416081748]
  WISE is a framework that unifies representation, feature weighting, clustering, and interpretation.<n>It produces faithful, human-interpretable explanations grounded in the same primitives that drive clustering.
  arXiv  Detail & Related papers  (2026-04-07T13:18:31Z)
• Krause Synchronization Transformers [63.8469912831803]
  Self-attention in Transformers relies on globally normalized softmax weights, causing all tokens to compete for influence at every layer.<n>We introduce Krause Attention, a principled attention mechanism inspired by bounded-confidence consensus dynamics.
  arXiv  Detail & Related papers  (2026-02-12T03:47:53Z)
• Random-Matrix-Induced Simplicity Bias in Over-parameterized Variational Quantum Circuits [72.0643009153473]
  We show that expressive variational ansatze enter a Haar-like universality class in which both observable expectation values and parameter gradients concentrate exponentially with system size.<n>As a consequence, the hypothesis class induced by such circuits collapses with high probability to a narrow family of near-constant functions.<n>We further show that this collapse is not unavoidable: tensor-structured VQCs, including tensor-network-based and tensor-hypernetwork parameterizations, lie outside the Haar-like universality class.
  arXiv  Detail & Related papers  (2026-01-05T08:04:33Z)
• The Mean-Field Dynamics of Transformers [6.008788032203683]
  By idealizing attention on the sphere, we connect Transformer dynamics to Wasserstein gradient flows (Kuramoto), and mean-shift clustering.<n>Results highlight both the mechanisms that drive representation collapse and the regimes that preserve expressive, multi-cluster structure in deep attention architectures.
  arXiv  Detail & Related papers  (2025-12-01T16:51:00Z)
• Self-Supervised Graph Embedding Clustering [70.36328717683297]
  K-means one-step dimensionality reduction clustering method has made some progress in addressing the curse of dimensionality in clustering tasks. We propose a unified framework that integrates manifold learning with K-means, resulting in the self-supervised graph embedding framework.
  arXiv  Detail & Related papers  (2024-09-24T08:59:51Z)
• Calibrating Undisciplined Over-Smoothing in Transformer for Weakly Supervised Semantic Segmentation [51.14107156747967]
  Weakly supervised semantic segmentation (WSSS) has attracted considerable attention because it requires fewer annotations than fully supervised approaches.<n>We propose an Adaptive Re-Activation Mechanism (AReAM) to control deep-level attention to undisciplined over-smoothing.<n>AReAM substantially improves segmentation performance compared with existing WSSS methods, reducing noise while sharpening focus on relevant semantic regions.
  arXiv  Detail & Related papers  (2023-05-04T19:11:33Z)
• Quantum scars and bulk coherence in a symmetry-protected topological phase [0.0]
  We show the existence of many-body scars and their implications on bulk coherence in certain protected topological (SPT) phases. We show that eigenstates with volume-law entanglement coexist with area-law entangled eigenstates throughout the spectrum. Our work sheds light on the role of quantum many-body scars in preserving SPT order at finite temperature and the possibility of coherent bulk dynamics in models with SPT order beyond the existence of long-lived edge modes.
  arXiv  Detail & Related papers  (2021-03-29T18:35:35Z)
• Out-of-time-order correlations and the fine structure of eigenstate thermalisation [58.720142291102135]
  Out-of-time-orderors (OTOCs) have become established as a tool to characterise quantum information dynamics and thermalisation. We show explicitly that the OTOC is indeed a precise tool to explore the fine details of the Eigenstate Thermalisation Hypothesis (ETH) We provide an estimation of the finite-size scaling of $omega_textrmGOE$ for the general class of observables composed of sums of local operators in the infinite-temperature regime.
  arXiv  Detail & Related papers  (2021-03-01T17:51:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.