Symmetry in language statistics shapes the geometry of model representations

• URL: http://arxiv.org/abs/2602.15029v1
• Date: Mon, 16 Feb 2026 18:59:55 GMT
• Title: Symmetry in language statistics shapes the geometry of model representations
• Abstract summary: We show that the statistics of language exhibit a translation symmetry.<n>We prove that the latter governs the aforementioned geometric structures in high-dimensional word embedding models.<n>We show that this robustness naturally emerges if the co-occurrence statistics are collectively controlled by an underlying continuous latent variable.
• Score: 8.371687575566714
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Although learned representations underlie neural networks' success, their fundamental properties remain poorly understood. A striking example is the emergence of simple geometric structures in LLM representations: for example, calendar months organize into a circle, years form a smooth one-dimensional manifold, and cities' latitudes and longitudes can be decoded by a linear probe. We show that the statistics of language exhibit a translation symmetry -- e.g., the co-occurrence probability of two months depends only on the time interval between them -- and we prove that the latter governs the aforementioned geometric structures in high-dimensional word embedding models. Moreover, we find that these structures persist even when the co-occurrence statistics are strongly perturbed (for example, by removing all sentences in which two months appear together) and at moderate embedding dimension. We show that this robustness naturally emerges if the co-occurrence statistics are collectively controlled by an underlying continuous latent variable. We empirically validate this theoretical framework in word embedding models, text embedding models, and large language models.

Related papers

• Structure Before Collapse: Transient semantic geometry in next-token prediction [33.942301586178914]
  Neural Collapse predicts that balanced one-hot classification pushes model representations to be equally far from each other.<n>This creates a puzzle: next-token prediction language models are trained predominantly with one-hot labels.<n>We find that semantic geometry emerges early in training, and that representations cluster by shared attributes.
  arXiv  Detail & Related papers  (2026-06-25T08:33:34Z)
• The Shape of Addition: Geometric Structures of Arithmetic in Large Language Models [11.106770083374288]
  Large Language Models exhibit paradoxical hallucination in fundamental arithmetic, implying a disconnect between internal computation and discrete output.<n>We propose the Noisy Quantization Model to explain this geometry, framing arithmetic errors as Geometric Slippages caused by internal neural noise pushing a continuous, latent Carry Potential across quantization thresholds.<n>We validate these insights through a geometric consistency check method that effectively detects and corrects these quantization failures during inference.
  arXiv  Detail & Related papers  (2026-05-29T10:55:50Z)
• The Shape of Beliefs: Geometry, Dynamics, and Interventions along Representation Manifolds of Language Models' Posteriors [24.477029700560113]
  Large language models (LLMs) represent prompt-conditioned beliefs (posteriors over answers and claims)<n>We study a controlled setting in which Llama-3.2 generates samples from a normal distribution by implicitly inferring its parameters.<n>We find representations of curved "belief manifold" for these parameters form with sufficient in-context learning.
  arXiv  Detail & Related papers  (2026-02-02T16:45:05Z)
• Steering Embedding Models with Geometric Rotation: Mapping Semantic Relationships Across Languages and Models [2.3204178451683264]
  We introduce Rotor-Invariant Shift Estimation (RISE), a geometric approach that represents semantic transformations as consistent rotational operations in embedding space.<n>RISE operations have the ability to operate across both languages and models with high transfer of performance.<n>This work provides the first systematic demonstration that discourse-level semantic transformations correspond to consistent geometric operations in multilingual embedding spaces.
  arXiv  Detail & Related papers  (2025-10-10T18:51:32Z)
• Event2Vec: A Geometric Approach to Learning Composable Representations of Event Sequences [0.15229257192293197]
  We introduce Event2Vec, a novel framework for learning representations of discrete event sequences.<n>We provide a theoretical analysis demonstrating that our model's learned representations in a Euclidean space converge to an ideal additive structure.<n>To address the limitations of Euclidean geometry for hierarchical data, we also introduce a variant of our model in hyperbolic space.
  arXiv  Detail & Related papers  (2025-09-15T17:51:02Z)
• TokenBlowUp: Resolving Representational Singularities in LLM Token Spaces via Monoidal Transformations [0.9179857807576733]
  Recent work has provided compelling evidence challenging the foundational manifold hypothesis for the token embedding spaces of Large Language Models.<n>We formalize this problem in the language of scheme theory and propose a rigorous resolution by applying the scheme-theoretic blow-up at each singular point.<n>We prove a formal theorem guaranteeing the geometric regularization of this new space, showing that the original pathologies are resolved.
  arXiv  Detail & Related papers  (2025-07-26T02:39:54Z)
• The language of time: a language model perspective on time-series foundation models [7.113398204739559]
  We study the representation learning mechanisms and generalization capabilities of patch-based time series foundation models.<n>Our work provides a rigorous theoretical cornerstone for understanding, evaluating, and improving the safety and reliability of large-scale time series foundation models.
  arXiv  Detail & Related papers  (2025-06-29T14:03:34Z)
• Shared Global and Local Geometry of Language Model Embeddings [46.33317507982751]
  We find numerous geometric similarities across the token embeddings of large language models.<n>We characterize local geometry in two ways: (1) by using Locally Linear Embeddings, and (2) by defining a simple measure for the intrinsic dimension of each embedding.
  arXiv  Detail & Related papers  (2025-03-27T01:17:06Z)
• (Deep) Generative Geodesics [57.635187092922976]
  We introduce a newian metric to assess the similarity between any two data points. Our metric leads to the conceptual definition of generative distances and generative geodesics. Their approximations are proven to converge to their true values under mild conditions.
  arXiv  Detail & Related papers  (2024-07-15T21:14:02Z)
• Latent Semantic Consensus For Deterministic Geometric Model Fitting [109.44565542031384]
  We propose an effective method called Latent Semantic Consensus (LSC) LSC formulates the model fitting problem into two latent semantic spaces based on data points and model hypotheses. LSC is able to provide consistent and reliable solutions within only a few milliseconds for general multi-structural model fitting.
  arXiv  Detail & Related papers  (2024-03-11T05:35:38Z)
• On the Origins of Linear Representations in Large Language Models [51.88404605700344]
  We introduce a simple latent variable model to formalize the concept dynamics of the next token prediction. Experiments show that linear representations emerge when learning from data matching the latent variable model. We additionally confirm some predictions of the theory using the LLaMA-2 large language model.
  arXiv  Detail & Related papers  (2024-03-06T17:17:36Z)
• Arrows of Time for Large Language Models [3.5678162800263307]
  We study the probabilistic modeling performed by Autoregressive Large Language Models (LLMs) through the angle of time directionality. For large enough models, we empirically find a time asymmetry in their ability to learn natural language: a difference in the average log-perplexity when trying to predict the next token versus when trying to predict the previous one.
  arXiv  Detail & Related papers  (2024-01-30T23:46:35Z)
• Geometric Neural Diffusion Processes [55.891428654434634]
  We extend the framework of diffusion models to incorporate a series of geometric priors in infinite-dimension modelling. We show that with these conditions, the generative functional model admits the same symmetry.
  arXiv  Detail & Related papers  (2023-07-11T16:51:38Z)
• GELATO: Geometrically Enriched Latent Model for Offline Reinforcement Learning [54.291331971813364]
  offline reinforcement learning approaches can be divided into proximal and uncertainty-aware methods. In this work, we demonstrate the benefit of combining the two in a latent variational model. Our proposed metrics measure both the quality of out of distribution samples as well as the discrepancy of examples in the data.
  arXiv  Detail & Related papers  (2021-02-22T19:42:40Z)
• Geometry of Similarity Comparisons [51.552779977889045]
  We show that the ordinal capacity of a space form is related to its dimension and the sign of its curvature. More importantly, we show that the statistical behavior of the ordinal spread random variables defined on a similarity graph can be used to identify its underlying space form.
  arXiv  Detail & Related papers  (2020-06-17T13:37:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.