A Geometric Perspective on Next-Token Prediction in Large Language Models: Three Emerging Phases

• URL: http://arxiv.org/abs/2605.09011v1
• Date: Sat, 09 May 2026 15:51:59 GMT
• Title: A Geometric Perspective on Next-Token Prediction in Large Language Models: Three Emerging Phases
• Abstract summary: We investigate the geometry of predictive information across the layers of large language models (LLMs)<n>Rather than asking what the model predicts at each layer, we ask where predictive information resides and how it evolves across depth.
• Score: 1.157930969832493
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We investigate the geometry of predictive information across the layers of large language models (LLMs). We repurpose representation lenses-learned affine maps trained to predict the next token from intermediate residual streams-as geometric diagnostic tools. Rather than asking what the model predicts at each layer, we ask where predictive information resides and how it evolves across depth. We define at each layer a predictive readout subspace as the dominant k-dimensional singular subspace of such a map on the d-dimensional residual stream (where k is a resolution parameter), and track its trajectory on the Grassmann manifold as a similarity profile across layers. The profile is well described by unimodal distributions exhibiting a rise, near-plateau, and descent; varying k from 1% to 50% of d traces a Pareto frontier between visibility and energy retention, yet the same structure emerges at all scales. Across eight models from two families (Qwen2.5 and OLMo2, 1B-32B), we identify three geometric phases. Updates are approximately orthogonal to the residual stream throughout; what distinguishes the phases is their effect on the effective rank, which expands, stabilizes, and concentrates. In the first, Seeding Multiplexing, feed-forward memories and attention layers seed a candidate set in superposition in family-specific proportions, with the final token rising as leading candidate from 20% to 35% of positions across this phase. In the second, Hoisting Overriding, updates override existing subspaces to concentrate the candidate distribution without expanding the rank. In the third, Focal Convergence, high-energy low-rank updates write the winner into a form aligned with the unembedding direction. Phases 1 and 3 grow slowly with model depth, while Phase 2 expands linearly. The additional capacity of deeper LLMs is largely absorbed by candidate disambiguation.

Related papers

• A*-Thought-V2: Efficient Latent Reasoning via Geometric Dynamics of LLM [40.07171947823021]
  Chain-of-Thought (CoT) improves the reasoning ability of Large Language Models (LLMs) but incurs substantial computation and context costs.<n>We present A*-Thought-V2, a geometric dynamics of LLM guided framework that models CoT as a hidden-state trajectory and replaces hard deletion with an explicit-implicit interleaved latent architecture.
  arXiv  Detail & Related papers  (2026-09-07T17:56:20Z)
• Hyper-Fold: Exploring the Expressive Limit of Sequence-Geometry Learning for Proteins via Hypergraph Modeling [55.36837305276515]
  We show that the complete bilinear operator over content-geometry outer products--the sufficient statistic of all second-order interactions--is the expressive ceiling.<n>We then introduce Hyper-Fold, a rank-K separable convolutional backbone approaching this ceiling at message-passing cost.
  arXiv  Detail & Related papers  (2026-08-29T11:43:25Z)
• From Extrinsic to Intrinsic: Geodesic-Guided Representation Learning for 3D Geometric Data [56.41479248637621]
  We introduce a novel 3D representation learning paradigm, namely textbfPRISM, for textbfPre-training.<n>PRISM learns embeddings by textbfRecovering the textbfIntrinsic textbfSurface geodesic textbfMetric.<n> Experiments demonstrate that our approach shows satisfactory accuracy, robustness, and high efficiency in geodesic distance prediction.
  arXiv  Detail & Related papers  (2026-06-01T13:54:26Z)
• Learn from your own latents and not from tokens: A sample-complexity theory [53.5821824211418]
  We study networks trained to predict their emphown latent representations of related views or masked regions.<n>We show that latent prediction achieves this with a number of samples emphconstant in $L$, up to logarithmic factors.<n>This suggests that explicit stacking such as H-JEPA is largely redundant.
  arXiv  Detail & Related papers  (2026-05-26T22:16:42Z)
• The Geometric Wall: Manifold Structure Predicts Layerwise Sparse Autoencoder Scaling Laws [13.054357482525505]
  We conduct the first cross-layer SAE scaling study, fitting and regressing on 68 layers of Gemma 2 2B and 9B.<n>We find that manifold geometry predicts the per-layer width exponent in both models.
  arXiv  Detail & Related papers  (2026-05-11T02:24:46Z)
• Posterior Augmented Flow Matching [64.1559809786948]
  Posterior-Augmented Flow Matching (PAFM) is a generalization of flow matching (FM)<n>We show that PAFM improves over FM by up to 3.4 FID50K across different model scales.
  arXiv  Detail & Related papers  (2026-05-01T17:59:59Z)
• HERMES++: Toward a Unified Driving World Model for 3D Scene Understanding and Generation [83.31948299340782]
  HERMES++ is a unified driving world model that integrates 3D scene understanding and future geometry prediction within a single framework.<n>Hermes++ achieves strong performance, outperforming specialist approaches in both future point cloud prediction and 3D scene understanding tasks.
  arXiv  Detail & Related papers  (2026-04-30T17:59:58Z)
• The Junction Law for Multipartite Entanglement in Confining Holographic Backgrounds [2.477977081568645]
  We first study an AdS$_3$ hard-wall toy model as an analytic benchmark, where multi-way cuts and junction geometries can be analyzed explicitly.<n>We then move beyond the hard-wall benchmark to smooth confining geometries, focusing on the D4-soliton and D3-soliton backgrounds.<n>In the smooth-cap examples, we find that the junction picture persists, while the detailed phase structure differs from the hard-wall case.
  arXiv  Detail & Related papers  (2026-04-12T11:02:36Z)
• Latent Object Permanence: Topological Phase Transitions, Free-Energy Principles, and Renormalization Group Flows in Deep Transformer Manifolds [0.5729426778193398]
  We study the emergence of multi-step reasoning in deep Transformer language models through a geometric and statistical-physics lens.<n>We formalize the forward pass as a discrete coarse-graining map and relate the appearance of stable "concept basins" to fixed points of this renormalization-like dynamics.<n>The resulting low-entropy regime is characterized by a spectral tail collapse and by the formation of transient, reusable object-like structures in representation space.
  arXiv  Detail & Related papers  (2026-01-16T23:11:02Z)
• Lotus-2: Advancing Geometric Dense Prediction with Powerful Image Generative Model [32.831576387973875]
  We propose a two-stage deterministic framework for stable, accurate and fine-grained geometric dense prediction.<n>Specifically, in the first stage, the core predictor employs a single-step deterministic formulation with a clean-data objective.<n>In the second stage, the detail sharpener performs a constrained multi-step rectified-flow refinement within the manifold defined by the core predictor.
  arXiv  Detail & Related papers  (2025-11-30T18:57:25Z)
• GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image [94.56927147492738]
  We introduce GeoWizard, a new generative foundation model designed for estimating geometric attributes from single images. We show that leveraging diffusion priors can markedly improve generalization, detail preservation, and efficiency in resource usage. We propose a simple yet effective strategy to segregate the complex data distribution of various scenes into distinct sub-distributions.
  arXiv  Detail & Related papers  (2024-03-18T17:50:41Z)
• One at a Time: Progressive Multi-step Volumetric Probability Learning for Reliable 3D Scene Perception [59.37727312705997]
  This paper proposes to decompose the complicated 3D volume representation learning into a sequence of generative steps. Considering the recent advances achieved by strong generative diffusion models, we introduce a multi-step learning framework, dubbed as VPD. For the SSC task, our work stands out as the first to surpass LiDAR-based methods on the Semantic KITTI dataset.
  arXiv  Detail & Related papers  (2023-06-22T05:55:53Z)
• Manifold Topology Divergence: a Framework for Comparing Data Manifolds [109.0784952256104]
  We develop a framework for comparing data manifold, aimed at the evaluation of deep generative models. Based on the Cross-Barcode, we introduce the Manifold Topology Divergence score (MTop-Divergence) We demonstrate that the MTop-Divergence accurately detects various degrees of mode-dropping, intra-mode collapse, mode invention, and image disturbance.
  arXiv  Detail & Related papers  (2021-06-08T00:30:43Z)
• You say Normalizing Flows I see Bayesian Networks [11.23030807455021]
  We show that normalizing flows reduce to Bayesian networks with a pre-defined topology and a learnable density at each node. We show that stacking multiple transformations in a normalizing flow relaxes independence assumptions and entangles the model distribution. We prove the non-universality of the affine normalizing flow, regardless of its depth.
  arXiv  Detail & Related papers  (2020-06-01T11:54:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.