Local and Global Regimes of Geometric Complexity in Language Model Representations

• URL: http://arxiv.org/abs/2608.14361v1
• Date: Fri, 14 Aug 2026 15:01:24 GMT
• Title: Local and Global Regimes of Geometric Complexity in Language Model Representations
• Abstract summary: This paper focuses on how lexical diversity, the number of unique last-token items present in a dataset, affects ID estimates of that dataset.<n>We find a scale-dependent transition between two regimes: at low lexical diversity, conditions with fewer unique final words produce higher ID, while at high lexical diversity, this ordering reverses.
• Score: 5.972083596296453
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Intrinsic dimensionality (ID) is widely used to probe the representational complexity of language models, but it remains unclear whether ID differences reflect properties of language itself or artefacts of how the underlying dataset was constructed. In this paper, we focus specifically on how lexical diversity, the number of unique last-token items present in a dataset, affects ID estimates of that dataset. We find a scale-dependent transition between two regimes: at low lexical diversity, conditions with fewer unique final words produce higher ID, while at high lexical diversity, this ordering reverses, and conditions with more unique words produce higher ID. We derive an exact, parameter-free formula for the point at which this reversal occurs, which matches the observed transition point at every scale tested. On the one hand, our results highlight how care must be taken when interpreting the intrinsic dimensionality of a set of representations as a straightforward cue of their complexity. On the other hand, our discovery of the two ID regimes reveals a general principle of organisation of linguistic data in LLMs that sheds new light on their inner manifold structures.

Related papers

• Tracing the complexity profiles of different linguistic phenomena through the intrinsic dimension of LLM representations [8.540350401981277]
  We ask if different ID profiles across LLM layers differentially characterize formal and functional complexity.<n>We find the formal contrast between sentences with multiple coordinated or subordinated clauses to be reflected in ID differences.<n>The functional contrasts between sentences characterized by right branching vs. center embedding or unambiguous vs. ambiguous relative clause attachment are also picked up by ID.
  arXiv  Detail & Related papers  (2026-01-07T10:16:59Z)
• Disentangling Linguistic Features with Dimension-Wise Analysis of Vector Embeddings [0.0]
  This paper proposes a framework for uncovering the specific dimensions of vector embeddings that encode distinct linguistic properties (LPs)<n>We introduce the Linguistically Distinct Sentence Pairs dataset, which isolates ten key linguistic features such as synonymy, negation, tense, and quantity.<n>Using this dataset, we analyze BERT embeddings with various methods to identify the most influential dimensions for each LP.<n>Our findings show that certain properties, such as negation and polarity, are robustly encoded in specific dimensions, while others, like synonymy, exhibit more complex patterns.
  arXiv  Detail & Related papers  (2025-04-20T23:38:16Z)
• Order-agnostic Identifier for Large Language Model-based Generative Recommendation [94.37662915542603]
  Items are assigned identifiers for Large Language Models (LLMs) to encode user history and generate the next item.<n>Existing approaches leverage either token-sequence identifiers, representing items as discrete token sequences, or single-token identifiers, using ID or semantic embeddings.<n>We propose SETRec, which leverages semantic tokenizers to obtain order-agnostic multi-dimensional tokens.
  arXiv  Detail & Related papers  (2025-02-15T15:25:38Z)
• Geometric Signatures of Compositionality Across a Language Model's Lifetime [47.25475802128033]
  We study whether contemporary language models reflect intrinsic simplicity of language enabled by compositionality.<n>We find that the relationship between compositionality and geometric complexity arises due to learned linguistic features over training.<n>Our analyses reveal a striking contrast between nonlinear and linear dimensionality, showing they respectively encode semantic and superficial aspects of linguistic composition.
  arXiv  Detail & Related papers  (2024-10-02T11:54:06Z)
• A Measure for Transparent Comparison of Linguistic Diversity in Multilingual NLP Data Sets [1.1647644386277962]
  Typologically diverse benchmarks are increasingly created to track the progress achieved in multilingual NLP. We propose assessing linguistic diversity of a data set against a reference language sample.
  arXiv  Detail & Related papers  (2024-03-06T18:14:22Z)
• Language Models As Semantic Indexers [78.83425357657026]
  We introduce LMIndexer, a self-supervised framework to learn semantic IDs with a generative language model. We show the high quality of the learned IDs and demonstrate their effectiveness on three tasks including recommendation, product search, and document retrieval.
  arXiv  Detail & Related papers  (2023-10-11T18:56:15Z)
• A Latent-Variable Model for Intrinsic Probing [93.62808331764072]
  We propose a novel latent-variable formulation for constructing intrinsic probes.<n>We find empirical evidence that pre-trained representations develop a cross-lingually entangled notion of morphosyntax.
  arXiv  Detail & Related papers  (2022-01-20T15:01:12Z)
• A Simple and Effective Method To Eliminate the Self Language Bias in Multilingual Representations [7.571549274473274]
  Language agnostic and semantic-language information isolation is an emerging research direction for multilingual representations models. "Language Information Removal (LIR)" factors out language identity information from semantic related components in multilingual representations pre-trained on multi-monolingual data. LIR reveals that for weak-alignment multilingual systems, the principal components of semantic spaces primarily encodes language identity information.
  arXiv  Detail & Related papers  (2021-09-10T08:15:37Z)
• Learning Universal Representations from Word to Sentence [89.82415322763475]
  This work introduces and explores the universal representation learning, i.e., embeddings of different levels of linguistic unit in a uniform vector space. We present our approach of constructing analogy datasets in terms of words, phrases and sentences. We empirically verify that well pre-trained Transformer models incorporated with appropriate training settings may effectively yield universal representation.
  arXiv  Detail & Related papers  (2020-09-10T03:53:18Z)
• BURT: BERT-inspired Universal Representation from Twin Structure [89.82415322763475]
  BURT (BERT inspired Universal Representation from Twin Structure) is capable of generating universal, fixed-size representations for input sequences of any granularity. Our proposed BURT adopts the Siamese network, learning sentence-level representations from natural language inference dataset and word/phrase-level representations from paraphrasing dataset. We evaluate BURT across different granularities of text similarity tasks, including STS tasks, SemEval2013 Task 5(a) and some commonly used word similarity tasks.
  arXiv  Detail & Related papers  (2020-04-29T04:01:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.