Latent Object Permanence: Topological Phase Transitions, Free-Energy Principles, and Renormalization Group Flows in Deep Transformer Manifolds

• URL: http://arxiv.org/abs/2601.19942v1
• Date: Fri, 16 Jan 2026 23:11:02 GMT
• Title: Latent Object Permanence: Topological Phase Transitions, Free-Energy Principles, and Renormalization Group Flows in Deep Transformer Manifolds
• Authors: Faruk Alpay, Bugra Kilictas,
• Abstract summary: We study the emergence of multi-step reasoning in deep Transformer language models through a geometric and statistical-physics lens.<n>We formalize the forward pass as a discrete coarse-graining map and relate the appearance of stable "concept basins" to fixed points of this renormalization-like dynamics.<n>The resulting low-entropy regime is characterized by a spectral tail collapse and by the formation of transient, reusable object-like structures in representation space.
• Score: 0.5729426778193398
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study the emergence of multi-step reasoning in deep Transformer language models through a geometric and statistical-physics lens. Treating the hidden-state trajectory as a flow on an implicit Riemannian manifold, we analyze the layerwise covariance spectrum of activations, where $C^{(\ell)}=\mathbb{E}[h^{(\ell)}h^{(\ell)\top}]$, and track deviations from a random-matrix bulk. Across model scales (1.5B--30B), we observe a sharp reduction in effective dimensionality consistent with a phase transition: an order parameter based on sparsity/localization, $Ω(h)=1-\|h\|_1/(\sqrt{d}\|h\|_2)$, exhibits a discontinuity near a critical normalized depth $γ_c\approx 0.42$ in sufficiently large models. We formalize the forward pass as a discrete coarse-graining map and relate the appearance of stable "concept basins" to fixed points of this renormalization-like dynamics. The resulting low-entropy regime is characterized by a spectral tail collapse and by the formation of transient, reusable object-like structures in representation space, which we call Transient Class Objects (TCOs). We provide theoretical conditions connecting logical separability to spectral decay and validate the predicted signatures with layerwise probes on multiple open-weight model families.

Related papers

• Symmetry-protected topology and deconfined solitons in a multi-link $\mathbb{Z}_2$ gauge theory [45.88028371034407]
  We study a $mathbbZ$ lattice gauge theory defined on a multi-graph with links that can be visualized as great circles of a spherical shell.<n>We show that this leads to state-dependent tunneling amplitudes underlying a phenomenon analogous to the Peierls instability.<n>By performining a detailed analysis based on matrix product states, we prove that charge deconfinement emerges as a consequence of charge-fractionalization.
  arXiv  Detail & Related papers  (2026-03-02T22:59:25Z)
• Rethinking Diffusion Models with Symmetries through Canonicalization with Applications to Molecular Graph Generation [56.361076943802594]
  CanonFlow achieves state-of-the-art performance on the challenging GEOM-DRUG dataset, and the advantage remains large in few-step generation.
  arXiv  Detail & Related papers  (2026-02-16T18:58:55Z)
• Parallel Complex Diffusion for Scalable Time Series Generation [50.01609741902786]
  PaCoDi is a spectral-native architecture that decouples generative modeling in the frequency domain.<n>We show that PaCoDi outperforms existing baselines in both generation quality and inference speed.
  arXiv  Detail & Related papers  (2026-02-10T14:31:53Z)
• Block-Recurrent Dynamics in Vision Transformers [42.261020313952976]
  We argue that trained ViTs admit a block-recurrent depth structure such that the computation of the original $L$ blocks can be accurately rewritten using only $k ll L$ distinct blocks applied recurrently.<n>We train a Raptor model to recover $96%$ of DINOv2 ImageNet-1k linear probe accuracy in only 2 blocks at equivalent computational cost.
  arXiv  Detail & Related papers  (2025-12-23T00:18:23Z)
• Spiral renormalization group flow and universal entanglement spectrum of the non-Hermitian 5-state Potts model [0.06597195879147556]
  We show that tensor network algorithms are still capable of simulating non-Hermitian theories.<n>We reconstruct the full boundary CCFT spectrum through the entanglement Hamiltonian encoded in the ground state.
  arXiv  Detail & Related papers  (2025-07-19T19:46:16Z)
• Generalized Linear Mode Connectivity for Transformers [87.32299363530996]
  A striking phenomenon is linear mode connectivity (LMC), where independently trained models can be connected by low- or zero-loss paths.<n>Prior work has predominantly focused on neuron re-ordering through permutations, but such approaches are limited in scope.<n>We introduce a unified framework that captures four symmetry classes: permutations, semi-permutations, transformations, and general invertible maps.<n>This generalization enables, for the first time, the discovery of low- and zero-barrier linear paths between independently trained Vision Transformers and GPT-2 models.
  arXiv  Detail & Related papers  (2025-06-28T01:46:36Z)
• Hessian Geometry of Latent Space in Generative Models [41.94295877935867]
  We present a novel method for analyzing the latent space geometry of generative models.<n>The method approximates the posterior distribution of latent variables given generated samples.<n>It is validated on the Ising and TASEP models, outperforming existing baselines in reconstructing thermodynamic quantities.
  arXiv  Detail & Related papers  (2025-06-12T12:17:40Z)
• From Chern to Winding: Topological Invariant Correspondence in the Reduced Haldane Model [0.4249842620609682]
  We present an exact analytical investigation of the topological properties and edge states of the Haldane model defined on a honeycomb lattice with zigzag edges.<n>We show that the $nu$ exactly reproduces the Chern number of the parent model in the topologically nontrivial phase.<n>Our analysis further reveals the critical momentum $ k_c $ where edge states traverse the bulk energy gap.
  arXiv  Detail & Related papers  (2025-05-26T19:11:43Z)
• Global Convergence in Training Large-Scale Transformers [43.3685424966098]
  This paper rigorously analyzes the convergence properties of gradient flow in training Transformers with weight decay regularization. Our analysis is based on a series of novel mean-field techniques that adapt to Transformers.
  arXiv  Detail & Related papers  (2024-10-31T03:51:39Z)
• Message-Passing Neural Quantum States for the Homogeneous Electron Gas [41.94295877935867]
  We introduce a message-passing-neural-network-based wave function Ansatz to simulate extended, strongly interacting fermions in continuous space. We demonstrate its accuracy by simulating the ground state of the homogeneous electron gas in three spatial dimensions.
  arXiv  Detail & Related papers  (2023-05-12T04:12:04Z)
• Nonlinear sigma models for monitored dynamics of free fermions [0.0]
  We derive descriptions for measurement-induced phase transitions in free fermion systems. We use the replica trick to map the dynamics to the imaginary time evolution of an effective spin chain. This is a nonlinear sigma model for an $Ntimes N$ matrix, in the replica limit $Nto 1$.
  arXiv  Detail & Related papers  (2023-02-24T18:56:37Z)
• Equivariant Discrete Normalizing Flows [10.867162810786361]
  We focus on building equivariant normalizing flows using discrete layers. We introduce two new equivariant flows: $G$-coupling Flows and $G$-Residual Flows. Our construction of $G$-Residual Flows are also universal, in the sense that we prove an $G$-equivariant diffeomorphism can be exactly mapped by a $G$-residual flow.
  arXiv  Detail & Related papers  (2021-10-16T20:16:00Z)
• Boundary theories of critical matchgate tensor networks [59.433172590351234]
  Key aspects of the AdS/CFT correspondence can be captured in terms of tensor network models on hyperbolic lattices. For tensors fulfilling the matchgate constraint, these have previously been shown to produce disordered boundary states. We show that these Hamiltonians exhibit multi-scale quasiperiodic symmetries captured by an analytical toy model.
  arXiv  Detail & Related papers  (2021-10-06T18:00:03Z)
• Discrete truncated Wigner approach to dynamical phase transitions in Ising models after a quantum quench [0.0]
  We study dynamical phase transitions arising in the steady state of transverse-field Ising models after a quantum quench. We find identical exponents for $alpha lesssim 0.5$, suggesting that the dynamical transitions in this regime fall into the same universality class as the nonergodic mean-field limit.
  arXiv  Detail & Related papers  (2020-04-21T08:20:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.