The Lattice Representation Hypothesis of Large Language Models

• URL: http://arxiv.org/abs/2603.01227v1
• Date: Sun, 01 Mar 2026 18:42:59 GMT
• Title: The Lattice Representation Hypothesis of Large Language Models
• Authors: Bo Xiong,
• Abstract summary: We show that linear attribute directions with separating thresholds induce a concept lattice via half-space intersections.<n>This geometry enables symbolic reasoning through geometric meet (intersection) and join (union) operations.
• Score: 18.00499182102749
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose the Lattice Representation Hypothesis of large language models: a symbolic backbone that grounds conceptual hierarchies and logical operations in embedding geometry. Our framework unifies the Linear Representation Hypothesis with Formal Concept Analysis (FCA), showing that linear attribute directions with separating thresholds induce a concept lattice via half-space intersections. This geometry enables symbolic reasoning through geometric meet (intersection) and join (union) operations, and admits a canonical form when attribute directions are linearly independent. Experiments on WordNet sub-hierarchies provide empirical evidence that LLM embeddings encode concept lattices and their logical structure, revealing a principled bridge between continuous geometry and symbolic abstraction.

Related papers

• The Representational Geometry of Number [1.5994376682356057]
  We show that number representations preserve a stable relational structure across tasks.<n>We find that task-specific representations are embedded in distinct subspaces, with low-level features like magnitude encoded along separable linear directions.<n>It suggests that understanding arises when task-specific transformations are applied to a shared underlying relational structure of conceptual representations.
  arXiv  Detail & Related papers  (2026-02-06T16:35:22Z)
• Native Logical and Hierarchical Representations with Subspace Embeddings [25.274936769664098]
  We introduce a novel paradigm: embedding concepts as linear subspaces.<n>It naturally supports set-theoretic operations like intersection (conjunction) and linear sum (disjunction)<n>Our method achieves state-of-the-art results in reconstruction and link prediction on WordNet.
  arXiv  Detail & Related papers  (2025-08-21T18:29:17Z)
• TokenBlowUp: Resolving Representational Singularities in LLM Token Spaces via Monoidal Transformations [0.9179857807576733]
  Recent work has provided compelling evidence challenging the foundational manifold hypothesis for the token embedding spaces of Large Language Models.<n>We formalize this problem in the language of scheme theory and propose a rigorous resolution by applying the scheme-theoretic blow-up at each singular point.<n>We prove a formal theorem guaranteeing the geometric regularization of this new space, showing that the original pathologies are resolved.
  arXiv  Detail & Related papers  (2025-07-26T02:39:54Z)
• Why Neural Network Can Discover Symbolic Structures with Gradient-based Training: An Algebraic and Geometric Foundation for Neurosymbolic Reasoning [73.18052192964349]
  We develop a theoretical framework that explains how discrete symbolic structures can emerge naturally from continuous neural network training dynamics.<n>By lifting neural parameters to a measure space and modeling training as Wasserstein gradient flow, we show that under geometric constraints, the parameter measure $mu_t$ undergoes two concurrent phenomena.
  arXiv  Detail & Related papers  (2025-06-26T22:40:30Z)
• The Origins of Representation Manifolds in Large Language Models [52.68554895844062]
  We show that cosine similarity in representation space may encode the intrinsic geometry of a feature through shortest, on-manifold paths.<n>The critical assumptions and predictions of the theory are validated on text embeddings and token activations of large language models.
  arXiv  Detail & Related papers  (2025-05-23T13:31:22Z)
• The Geometry of Categorical and Hierarchical Concepts in Large Language Models [15.126806053878855]
  We show how to extend the formalization of the linear representation hypothesis to represent features (e.g., is_animal) as vectors.<n>We use the formalization to prove a relationship between the hierarchical structure of concepts and the geometry of their representations.<n>We validate these theoretical results on the Gemma and LLaMA-3 large language models, estimating representations for 900+ hierarchically related concepts using data from WordNet.
  arXiv  Detail & Related papers  (2024-06-03T16:34:01Z)
• Learning Visual-Semantic Subspace Representations [49.17165360280794]
  We introduce a nuclear norm-based loss function, grounded in the same information theoretic principles that have proved effective in self-supervised learning.<n>We present a theoretical characterization of this loss, demonstrating that, in addition to promoting classity, it encodes the spectral geometry of the data within a subspace lattice.
  arXiv  Detail & Related papers  (2024-05-25T12:51:38Z)
• On the Origins of Linear Representations in Large Language Models [51.88404605700344]
  We introduce a simple latent variable model to formalize the concept dynamics of the next token prediction. Experiments show that linear representations emerge when learning from data matching the latent variable model. We additionally confirm some predictions of the theory using the LLaMA-2 large language model.
  arXiv  Detail & Related papers  (2024-03-06T17:17:36Z)
• An Encoding of Abstract Dialectical Frameworks into Higher-Order Logic [57.24311218570012]
  This approach allows for the computer-assisted analysis of abstract dialectical frameworks. Exemplary applications include the formal analysis and verification of meta-theoretical properties.
  arXiv  Detail & Related papers  (2023-12-08T09:32:26Z)
• A Formalisation of Abstract Argumentation in Higher-Order Logic [77.34726150561087]
  We present an approach for representing abstract argumentation frameworks based on an encoding into classical higher-order logic. This provides a uniform framework for computer-assisted assessment of abstract argumentation frameworks using interactive and automated reasoning tools.
  arXiv  Detail & Related papers  (2021-10-18T10:45:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.