Hierarchical Concept Geometry in Language Models Emerges from Word Co-occurrence

• URL: http://arxiv.org/abs/2605.23821v1
• Date: Fri, 22 May 2026 16:24:30 GMT
• Title: Hierarchical Concept Geometry in Language Models Emerges from Word Co-occurrence
• Abstract summary: We propose a distributional theory of how hypernymy is encoded geometrically in language representations.<n>We characterize theoretically the spectrum of the embedding Gram matrix of word2vec embeddings.<n>Our results indicate that hierarchical concept geometry in LLMs need not reflect a hierarchy-specific functional mechanism.
• Score: 9.453554184019108
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose a distributional theory of how hypernymy -- the ``is-a'' relation between general and specific concepts -- is encoded geometrically in language representations. Starting from the empirically verified assumption that words closer on the WordNet hypernym graph co-occur more often, we characterize theoretically the spectrum of the resulting embedding Gram matrix of word2vec embeddings. Under mild positivity and decay conditions on the co-occurrence kernel, we prove that the leading eigenvectors first separate broad taxonomic branches and then progressively finer sub-branches, producing a \emph{hierarchical splitting geometry} with a coarse-to-fine spectral organization that mirrors the tree. We confirm these predictions in word2vec embeddings across many sampled WordNet subtrees, and show that the same signature extends strikingly well to Gemma 2B unembeddings. Our results indicate that hierarchical concept geometry in LLMs need not reflect a hierarchy-specific functional mechanism, but emerges from the spectral structure of pairwise word statistics.

Related papers

• H$^2$EDL: Hyper Evidential Deep Learning for Hierarchical Classification [19.611209640933435]
  A new uncertainty representation, H$2$EDL, can capture both fine-grained classes and intermediate concepts.<n>On FGVC-Aircraft and DERM12345, H$2$EDL reduces calibration error by approximately half compared with cross-entropy baselines.
  arXiv  Detail & Related papers  (2026-08-18T08:42:32Z)
• Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Models [47.868429337792314]
  Large multimodal models (LMMs) often lack taxonomic knowledge, leading to low hierarchical visual recognition (HVR) consistency.<n>We propose Hierarchical Representation Regularization ($HiR2$), a simple plug-and-play regularizer that improves hierarchical consistency in LMMs.<n>$HiR2$ effectively captures taxonomic structures across diverse LMMs and fine-tuning methods.
  arXiv  Detail & Related papers  (2026-07-03T03:16:41Z)
• An expressivity analysis of hierarchical modelling in deep transformers via bounded-depth grammars [54.11540943172608]
  Deep neural networks are widely believed to derive their expressive power from their ability to form textbfhierarchical representations<n>In language modeling, textbftransformers have emerged as the dominant architecture, with early layers capturing local syntactic patterns and later layers encoding more complex clause-level dependencies.<n>We argue that these architectures possess the structural capacity to encode abstract grammatical states into low-dimensional, linearly separable subspaces within the residual stream.
  arXiv  Detail & Related papers  (2026-06-16T05:02:13Z)
• OpenGaFF: Open-Vocabulary Gaussian Feature Field with Codebook Attention [80.51557267896938]
  We present OpenGaFF, a novel framework for open-vocabulary 3D scene understanding built upon 3D Gaussian Splatting.<n>At the core of our method is a Gaussian Feature Field that models semantics as a continuous function of Gaussian geometry and appearance.<n>To further enforce object-level semantic consistency, we introduce a structured codebook that serves as a set of shared semantic primitives.
  arXiv  Detail & Related papers  (2026-05-07T12:10:07Z)
• The Lattice Representation Hypothesis of Large Language Models [18.00499182102749]
  We show that linear attribute directions with separating thresholds induce a concept lattice via half-space intersections.<n>This geometry enables symbolic reasoning through geometric meet (intersection) and join (union) operations.
  arXiv  Detail & Related papers  (2026-03-01T18:42:59Z)
• Hierarchical Mamba Meets Hyperbolic Geometry: A New Paradigm for Structured Language Embeddings [1.4183971140167244]
  We propose Hierarchical Mamba (HiM) to learn hierarchy-aware language embeddings.<n>HiM integrates efficient Mamba2 with exponential growth and curved nature of hyperbolic geometry.<n>We show that both HiM models effectively capture hierarchical relationships for four ontological datasets.
  arXiv  Detail & Related papers  (2025-05-25T04:45:06Z)
• The Geometry of Meaning: Perfect Spacetime Representations of Hierarchical Structures [0.0]
  We show that there is a fast algorithm that embeds hierarchical structures in three-dimensional Minkowski spacetime.<n>Our results seem to indicate that all discrete data has a perfect geometrical representation that is three-dimensional.
  arXiv  Detail & Related papers  (2025-05-07T20:41:06Z)
• Integrating Hierarchical Semantic into Iterative Generation Model for Entailment Tree Explanation [7.5496857647335585]
  We propose an architecture of integrating the Hierarchical Semantics of sentences under the framework of Controller-Generator (HiSCG) to explain answers. The proposed method achieves comparable performance on all three settings of the EntailmentBank dataset.
  arXiv  Detail & Related papers  (2024-09-26T11:46:58Z)
• Linear Spaces of Meanings: Compositional Structures in Vision-Language Models [110.00434385712786]
  We investigate compositional structures in data embeddings from pre-trained vision-language models (VLMs) We first present a framework for understanding compositional structures from a geometric perspective. We then explain what these structures entail probabilistically in the case of VLM embeddings, providing intuitions for why they arise in practice.
  arXiv  Detail & Related papers  (2023-02-28T08:11:56Z)
• Latent Topology Induction for Understanding Contextualized Representations [84.7918739062235]
  We study the representation space of contextualized embeddings and gain insight into the hidden topology of large language models. We show there exists a network of latent states that summarize linguistic properties of contextualized representations.
  arXiv  Detail & Related papers  (2022-06-03T11:22:48Z)
• Spectral embedding and the latent geometry of multipartite networks [47.18054692085814]
  Many networks are multipartite, meaning that they contain nodes of fundamentally different types.<n>This paper demonstrates that the node representations obtained via spectral embedding lie near type-specific low-dimensional subspaces of a higher-dimensional ambient space.<n>We propose a follow-on step after spectral embedding, to recover node representations in their intrinsic rather than ambient dimension.
  arXiv  Detail & Related papers  (2022-02-08T15:52:03Z)
• Modeling Heterogeneous Hierarchies with Relation-specific Hyperbolic Cones [64.75766944882389]
  We present ConE (Cone Embedding), a KG embedding model that is able to simultaneously model multiple hierarchical as well as non-hierarchical relations in a knowledge graph. In particular, ConE uses cone containment constraints in different subspaces of the hyperbolic embedding space to capture multiple heterogeneous hierarchies. Our approach yields new state-of-the-art Hits@1 of 45.3% on WN18RR and 16.1% on DDB14 (0.231 MRR)
  arXiv  Detail & Related papers  (2021-10-28T07:16:08Z)
• The Low-Dimensional Linear Geometry of Contextualized Word Representations [27.50785941238007]
  We study the linear geometry of contextualized word representations in ELMO and BERT. We show that a variety of linguistic features are encoded in low-dimensional subspaces.
  arXiv  Detail & Related papers  (2021-05-15T00:58:08Z)
• Dual-constrained Deep Semi-Supervised Coupled Factorization Network with Enriched Prior [80.5637175255349]
  We propose a new enriched prior based Dual-constrained Deep Semi-Supervised Coupled Factorization Network, called DS2CF-Net. To ex-tract hidden deep features, DS2CF-Net is modeled as a deep-structure and geometrical structure-constrained neural network. Our network can obtain state-of-the-art performance for representation learning and clustering.
  arXiv  Detail & Related papers  (2020-09-08T13:10:21Z)
• Embedding Graph Auto-Encoder for Graph Clustering [90.8576971748142]
  Graph auto-encoder (GAE) models are based on semi-supervised graph convolution networks (GCN) We design a specific GAE-based model for graph clustering to be consistent with the theory, namely Embedding Graph Auto-Encoder (EGAE) EGAE consists of one encoder and dual decoders.
  arXiv  Detail & Related papers  (2020-02-20T09:53:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.