Structure Before Collapse: Transient semantic geometry in next-token prediction

• URL: http://arxiv.org/abs/2606.26749v1
• Date: Thu, 25 Jun 2026 08:33:34 GMT
• Title: Structure Before Collapse: Transient semantic geometry in next-token prediction
• Abstract summary: Neural Collapse predicts that balanced one-hot classification pushes model representations to be equally far from each other.<n>This creates a puzzle: next-token prediction language models are trained predominantly with one-hot labels.<n>We find that semantic geometry emerges early in training, and that representations cluster by shared attributes.
• Score: 33.942301586178914
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural Collapse predicts that balanced one-hot classification pushes model representations to be equally far from each other; a symmetric configuration that depends only on the output label and ignores any semantic similarity in the inputs. This creates a puzzle: next-token prediction language models are trained predominantly (as context length increases) with one-hot labels: the same context is very unlikely to appear twice in training with different labels. However, they clearly learn latent structural features. That is, despite the one-hot training regime, a language model's contextual embeddings represent the fact that the next word in ''Mary broke the ___'' is likely to be filled by tokens in the latent classes of a) medium-sized, b) rigid, c) inanimate nouns. How does gradient descent find such categorical semantic structure when co-occurrence statistics collapse to one-hot sparsity, eliminating any shared next-tokens among different contexts? To investigate this tension we identify three synthetic controlled settings where inputs have latent semantic factors but are mapped to distinct one-hot labels. We find that semantic geometry emerges early in training, and that representations cluster by shared attributes despite receiving no explicit supervision to do so. This structure is transient: with sufficient capacity and time, the model eventually reaches the predicted symmetric state where all representations are equally separated. We study this phase transition through Gram matrix analysis and propose a preliminary modification to the commonly used unconstrained features model to capture the emergent semantic geometry.

Related papers

• Does Syntax Matter? A Graph-Augmented Variational Topic Model for Computational Social Sciences [51.56484100374058]
  This paper introduces the Structural Contextual Probabilistic Topic Model ( SCPTM), an architecture that incorporates syntactic dependency relations into topic inference.<n>We evaluate seven topic modeling techniques across four corpora differing in register and discourse structure.<n>Results show that SCPTM's neural architecture yields substantial gains in document-topic alignment over generative baselines.
  arXiv  Detail & Related papers  (2026-09-07T17:39:06Z)
• Learn from your own latents and not from tokens: A sample-complexity theory [53.5821824211418]
  We study networks trained to predict their emphown latent representations of related views or masked regions.<n>We show that latent prediction achieves this with a number of samples emphconstant in $L$, up to logarithmic factors.<n>This suggests that explicit stacking such as H-JEPA is largely redundant.
  arXiv  Detail & Related papers  (2026-05-26T22:16:42Z)
• Feature Lottery? A Bifurcation Theory of Concept Emergence [0.0]
  We show the onset of structure corresponds to a supercritical pitchfork bifurcation driven by the loss Hessian.<n>We empirically validate four distinct transition regimes predicted by this coordinate across diverse settings.<n>We term this the "feature lottery" in SAE training: a feature's terminal interpretability becomes predictable remarkably early.
  arXiv  Detail & Related papers  (2026-05-22T02:08:49Z)
• Semantic Shift: the Fundamental Challenge in Text Embedding and Retrieval [39.73839943827917]
  Transformer-based embedding models rely on pooling to map variable-length text into a single vector.<n>We argue that the missing causal factor is emphsemantic shift: the intrinsic, structured evolution and dispersion of semantics within a text.
  arXiv  Detail & Related papers  (2026-03-22T22:58:11Z)
• Geometry of Semantics in Next-Token Prediction: How Optimization Implicitly Organizes Linguistic Representations [34.88156871518115]
  Next-token prediction (NTP) optimization leads language models to extract and organize semantic structure from text.<n>We demonstrate that concepts corresponding to larger singular values are learned earlier during training, yielding a natural semantic hierarchy.<n>This insight motivates orthant-based clustering, a method that combines concept signs to identify interpretable semantic categories.
  arXiv  Detail & Related papers  (2025-05-13T08:46:04Z)
• Bridging Continuous and Discrete Tokens for Autoregressive Visual Generation [85.82112629564942]
  We propose TokenBridge, which maintains the strong representation capacity of continuous tokens while preserving the modeling simplicity of discrete tokens.<n>We introduce a dimension-wise quantization strategy that independently discretizes each feature dimension, paired with a lightweight autoregressive prediction mechanism.<n>Our approach achieves reconstruction and generation quality on par with continuous methods while using standard categorical prediction.
  arXiv  Detail & Related papers  (2025-03-20T17:59:59Z)
• Human-in-the-loop: Towards Label Embeddings for Measuring Classification Difficulty [14.452983136429967]
  In supervised learning, uncertainty can already occur in the first stage of the training process, the annotation phase. The main idea of this work is to drop the assumption of a ground truth label and instead embed the annotations into a multidimensional space. The methods developed in this paper readily extend to various situations where multiple annotators independently label instances.
  arXiv  Detail & Related papers  (2023-11-15T11:23:15Z)
• Robustness of the Random Language Model [0.0]
  The model suggests a simple picture of first language learning as a type of annealing in the vast space of potential languages. It implies a single continuous transition to grammatical syntax, at which the symmetry among potential words and categories is spontaneously broken. Results are discussed in light of theory of first-language acquisition in linguistics, and recent successes in machine learning.
  arXiv  Detail & Related papers  (2023-09-26T13:14:35Z)
• A Multi-Grained Self-Interpretable Symbolic-Neural Model For Single/Multi-Labeled Text Classification [29.075766631810595]
  We propose a Symbolic-Neural model that can learn to explicitly predict class labels of text spans from a constituency tree. As the structured language model learns to predict constituency trees in a self-supervised manner, only raw texts and sentence-level labels are required as training data. Our experiments demonstrate that our approach could achieve good prediction accuracy in downstream tasks.
  arXiv  Detail & Related papers  (2023-03-06T03:25:43Z)
• Understanding Imbalanced Semantic Segmentation Through Neural Collapse [81.89121711426951]
  We show that semantic segmentation naturally brings contextual correlation and imbalanced distribution among classes. We introduce a regularizer on feature centers to encourage the network to learn features closer to the appealing structure. Our method ranks 1st and sets a new record on the ScanNet200 test leaderboard.
  arXiv  Detail & Related papers  (2023-01-03T13:51:51Z)
• Unsupervised Distillation of Syntactic Information from Contextualized Word Representations [62.230491683411536]
  We tackle the task of unsupervised disentanglement between semantics and structure in neural language representations. To this end, we automatically generate groups of sentences which are structurally similar but semantically different. We demonstrate that our transformation clusters vectors in space by structural properties, rather than by lexical semantics.
  arXiv  Detail & Related papers  (2020-10-11T15:13:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.