Attention to Order: Transformers Discover Phase Transitions via Learnability

• URL: http://arxiv.org/abs/2510.07401v1
• Date: Wed, 08 Oct 2025 18:00:59 GMT
• Title: Attention to Order: Transformers Discover Phase Transitions via Learnability
• Abstract summary: We introduce learnability as a universal criterion, defined as the ability of a transformer model containing attention mechanism to extract structure from microscopic states.<n>Using self-supervised learning and Monte Carlo generated configurations of the two-dimensional Ising model, we show that ordered phases correspond to enhanced learnability.<n>Our results establish learnability as a data-driven marker of phase transitions and highlight deep parallels between long-range order in condensed matter and the emergence of structure in modern language models.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Phase transitions mark qualitative reorganizations of collective behavior, yet identifying their boundaries remains challenging whenever analytic solutions are absent and conventional simulations fail. Here we introduce learnability as a universal criterion, defined as the ability of a transformer model containing attention mechanism to extract structure from microscopic states. Using self-supervised learning and Monte Carlo generated configurations of the two-dimensional Ising model, we show that ordered phases correspond to enhanced learnability, manifested in both reduced training loss and structured attention patterns, while disordered phases remain resistant to learning. Two unsupervised diagnostics, the sharp jump in training loss and the rise in attention entropy, recover the critical temperature in excellent agreement with the exact value. Our results establish learnability as a data-driven marker of phase transitions and highlight deep parallels between long-range order in condensed matter and the emergence of structure in modern language models.

Related papers

• Emergent Capabilities Arise Randomly from Learning Sparse Attention Patterns [48.062049009807446]
  downstream capabilities such as in-context learning are known to emerge abruptly past a certain model scale.<n>We demonstrate that the emergence of capabilities such as pattern completion and indirect object identification corresponds to the abrupt learning of task-relevant attention patterns.<n>We train transformer models on synthetic linear map and cellular automata datasets, and we show that the difficulty of learning attention patterns depends on context length and pattern sparsity.
  arXiv  Detail & Related papers  (2026-06-23T17:51:10Z)
• SurgVista: Long-Horizon Surgical World Modeling with Plausible Instrument-Tissue Dynamics [49.771521708359955]
  SurgVista is a surgical world model that mitigates spatial interaction incoherence and temporal fidelity collapse.<n>It consistently outperforms state-of-the-art methods across visual quality, temporal consistency, and interaction fidelity.
  arXiv  Detail & Related papers  (2026-06-18T07:47:28Z)
• An Interpretable Generative Framework for Anomaly Detection in High-Dimensional Financial Time Series [0.0]
  We propose ReGEN-TAD, an interpretable generative framework that integrates modern machine learning with econometric diagnostics for anomaly detection.<n> Experiments on synthetic and financial panels demonstrate improved robustness to structured deviations while enabling economically coherent factor-level attribution.
  arXiv  Detail & Related papers  (2026-03-09T00:36:19Z)
• Transformer Learning of Chaotic Collective Dynamics in Many-Body Systems [0.0]
  We show that a self-attention-based transformer framework provides an effective approach for modeling chaotic collective dynamics.<n>We study the one-dimensional semiclassical Holstein model, where interaction quenches induce strongly nonlinear and chaotic dynamics.<n>Our results establish self-attention as a powerful mechanism for learning effective reduced dynamics in chaotic many-body systems.
  arXiv  Detail & Related papers  (2026-01-27T01:33:33Z)
• Continual Learning with Query-Only Attention [0.0]
  Continual learning involves learning from a stream of data without repetition of data points.<n>We propose a query-only attention mechanism that discards keys and values, yet preserves the core inductive bias of transformer architectures.
  arXiv  Detail & Related papers  (2025-10-01T00:14:34Z)
• Provable In-Context Learning of Nonlinear Regression with Transformers [66.99048542127768]
  In-context learning (ICL) is the ability to perform unseen tasks using task specific prompts without updating parameters.<n>Recent research has actively explored the training dynamics behind ICL, with much of the focus on relatively simple tasks.<n>This paper investigates more complex nonlinear regression tasks, aiming to uncover how transformers acquire in-context learning capabilities.
  arXiv  Detail & Related papers  (2025-07-28T00:09:28Z)
• Multiple Descents in Deep Learning as a Sequence of Order-Chaos Transitions [4.424479010649862]
  We observe a novel'multiple-descent' phenomenon during the training process of LSTM, in which the test loss goes through long cycles of up and down trend multiple times after the model is overtrained.<n>By carrying out stability analysis of the models, we found that the cycles in test loss are closely associated with the phase transition process between order and chaos.
  arXiv  Detail & Related papers  (2025-05-26T14:18:22Z)
• Interpreting Affine Recurrence Learning in GPT-style Transformers [54.01174470722201]
  In-context learning allows GPT-style transformers to generalize during inference without modifying their weights. This paper focuses specifically on their ability to learn and predict affine recurrences as an ICL task. We analyze the model's internal operations using both empirical and theoretical approaches.
  arXiv  Detail & Related papers  (2024-10-22T21:30:01Z)
• Training Dynamics of Transformers to Recognize Word Co-occurrence via Gradient Flow Analysis [97.54180451650122]
  We study the dynamics of training a shallow transformer on a task of recognizing co-occurrence of two designated words. We analyze the gradient flow dynamics of simultaneously training three attention matrices and a linear layer. We prove a novel property of the gradient flow, termed textitautomatic balancing of gradients, which enables the loss values of different samples to decrease almost at the same rate and further facilitates the proof of near minimum training loss.
  arXiv  Detail & Related papers  (2024-10-12T17:50:58Z)
• Connectivity Shapes Implicit Regularization in Matrix Factorization Models for Matrix Completion [2.8948274245812335]
  We investigate the implicit regularization of matrix factorization for solving matrix completion problems.<n>We empirically discover that the connectivity of observed data plays a crucial role in the implicit bias.<n>Our work reveals the intricate interplay between data connectivity, training dynamics, and implicit regularization in matrix factorization models.
  arXiv  Detail & Related papers  (2024-05-22T15:12:14Z)
• In-Context Convergence of Transformers [63.04956160537308]
  We study the learning dynamics of a one-layer transformer with softmax attention trained via gradient descent. For data with imbalanced features, we show that the learning dynamics take a stage-wise convergence process.
  arXiv  Detail & Related papers  (2023-10-08T17:55:33Z)
• Learning Multiscale Consistency for Self-supervised Electron Microscopy Instance Segmentation [48.267001230607306]
  We propose a pretraining framework that enhances multiscale consistency in EM volumes. Our approach leverages a Siamese network architecture, integrating strong and weak data augmentations. It effectively captures voxel and feature consistency, showing promise for learning transferable representations for EM analysis.
  arXiv  Detail & Related papers  (2023-08-19T05:49:13Z)
• Latent Traversals in Generative Models as Potential Flows [113.4232528843775]
  We propose to model latent structures with a learned dynamic potential landscape. Inspired by physics, optimal transport, and neuroscience, these potential landscapes are learned as physically realistic partial differential equations. Our method achieves both more qualitatively and quantitatively disentangled trajectories than state-of-the-art baselines.
  arXiv  Detail & Related papers  (2023-04-25T15:53:45Z)
• Stabilizing Transformer Training by Preventing Attention Entropy Collapse [56.45313891694746]
  We investigate the training dynamics of Transformers by examining the evolution of the attention layers. We show that $sigma$Reparam successfully prevents entropy collapse in the attention layers, promoting more stable training. We conduct experiments with $sigma$Reparam on image classification, image self-supervised learning, machine translation, speech recognition, and language modeling tasks.
  arXiv  Detail & Related papers  (2023-03-11T03:30:47Z)
• Inferring topological transitions in pattern-forming processes with self-supervised learning [25.90630151217217]
  We use a self-supervised approach to predict process parameters from observed microstructures using neural networks. We show that the difficulty of performing this prediction task is related to the goal of discovering microstructure regimes. This approach opens a promising path forward for discovering and understanding unseen or hard-to-detect transition regimes.
  arXiv  Detail & Related papers  (2022-03-19T00:47:50Z)
• Learning perturbation sets for robust machine learning [97.6757418136662]
  We use a conditional generator that defines the perturbation set over a constrained region of the latent space. We measure the quality of our learned perturbation sets both quantitatively and qualitatively. We leverage our learned perturbation sets to train models which are empirically and certifiably robust to adversarial image corruptions and adversarial lighting variations.
  arXiv  Detail & Related papers  (2020-07-16T16:39:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.