Native Logical and Hierarchical Representations with Subspace Embeddings

• URL: http://arxiv.org/abs/2508.16687v1
• Date: Thu, 21 Aug 2025 18:29:17 GMT
• Title: Native Logical and Hierarchical Representations with Subspace Embeddings
• Authors: Gabriel Moreira, Zita Marinho, Manuel Marques, João Paulo Costeira, Chenyan Xiong,
• Abstract summary: We introduce a novel paradigm: embedding concepts as linear subspaces.<n>It naturally supports set-theoretic operations like intersection (conjunction) and linear sum (disjunction)<n>Our method achieves state-of-the-art results in reconstruction and link prediction on WordNet.
• Score: 25.274936769664098
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Traditional neural embeddings represent concepts as points, excelling at similarity but struggling with higher-level reasoning and asymmetric relationships. We introduce a novel paradigm: embedding concepts as linear subspaces. This framework inherently models generality via subspace dimensionality and hierarchy through subspace inclusion. It naturally supports set-theoretic operations like intersection (conjunction), linear sum (disjunction) and orthogonal complements (negations), aligning with classical formal semantics. To enable differentiable learning, we propose a smooth relaxation of orthogonal projection operators, allowing for the learning of both subspace orientation and dimension. Our method achieves state-of-the-art results in reconstruction and link prediction on WordNet. Furthermore, on natural language inference benchmarks, our subspace embeddings surpass bi-encoder baselines, offering an interpretable formulation of entailment that is both geometrically grounded and amenable to logical operations.

Related papers

• The Lattice Representation Hypothesis of Large Language Models [18.00499182102749]
  We show that linear attribute directions with separating thresholds induce a concept lattice via half-space intersections.<n>This geometry enables symbolic reasoning through geometric meet (intersection) and join (union) operations.
  arXiv  Detail & Related papers  (2026-03-01T18:42:59Z)
• The Representational Geometry of Number [1.5994376682356057]
  We show that number representations preserve a stable relational structure across tasks.<n>We find that task-specific representations are embedded in distinct subspaces, with low-level features like magnitude encoded along separable linear directions.<n>It suggests that understanding arises when task-specific transformations are applied to a shared underlying relational structure of conceptual representations.
  arXiv  Detail & Related papers  (2026-02-06T16:35:22Z)
• Bhargava Cube--Inspired Quadratic Regularization for Structured Neural Embeddings [0.0]
  We present a novel approach to neural representation learning that incorporates algebraic constraints inspired by Bhargava cubes from number theory.<n>Our framework maps input data to constrained 3-dimensional latent spaces where embeddings are regularized to satisfy learned quadratic relationships.<n>We evaluate on MNIST, achieving 99.46% accuracy while producing interpretable 3D embeddings that naturally cluster by digit class.
  arXiv  Detail & Related papers  (2025-12-12T09:05:11Z)
• The Origins of Representation Manifolds in Large Language Models [52.68554895844062]
  We show that cosine similarity in representation space may encode the intrinsic geometry of a feature through shortest, on-manifold paths.<n>The critical assumptions and predictions of the theory are validated on text embeddings and token activations of large language models.
  arXiv  Detail & Related papers  (2025-05-23T13:31:22Z)
• Manifold-Constrained Sentence Embeddings via Triplet Loss: Projecting Semantics onto Spheres, Tori, and Möbius Strips [0.0]
  We introduce a novel framework that constrains sentence embeddings to lie on continuous manifold.<n>By enforcing differential geometric constraints on the output space, our approach encourages the learning of embeddings that are both discriminative and topologically structured.<n>Our results demonstrate that manifold-constrained embeddings, particularly those projected onto spheres and M"obius strips, significantly outperform traditional approaches in both clustering quality and classification performance.
  arXiv  Detail & Related papers  (2025-04-22T19:20:53Z)
• Learning Visual-Semantic Subspace Representations [49.17165360280794]
  We introduce a nuclear norm-based loss function, grounded in the same information theoretic principles that have proved effective in self-supervised learning.<n>We present a theoretical characterization of this loss, demonstrating that, in addition to promoting classity, it encodes the spectral geometry of the data within a subspace lattice.
  arXiv  Detail & Related papers  (2024-05-25T12:51:38Z)
• A Geometric Notion of Causal Probing [85.49839090913515]
  The linear subspace hypothesis states that, in a language model's representation space, all information about a concept such as verbal number is encoded in a linear subspace.<n>We give a set of intrinsic criteria which characterize an ideal linear concept subspace.<n>We find that, for at least one concept across two languages models, the concept subspace can be used to manipulate the concept value of the generated word with precision.
  arXiv  Detail & Related papers  (2023-07-27T17:57:57Z)
• Linear Spaces of Meanings: Compositional Structures in Vision-Language Models [110.00434385712786]
  We investigate compositional structures in data embeddings from pre-trained vision-language models (VLMs) We first present a framework for understanding compositional structures from a geometric perspective. We then explain what these structures entail probabilistically in the case of VLM embeddings, providing intuitions for why they arise in practice.
  arXiv  Detail & Related papers  (2023-02-28T08:11:56Z)
• HyperMiner: Topic Taxonomy Mining with Hyperbolic Embedding [54.52651110749165]
  We present a novel framework that introduces hyperbolic embeddings to represent words and topics. With the tree-likeness property of hyperbolic space, the underlying semantic hierarchy can be better exploited to mine more interpretable topics.
  arXiv  Detail & Related papers  (2022-10-16T02:54:17Z)
• Cross-Lingual BERT Contextual Embedding Space Mapping with Isotropic and Isometric Conditions [7.615096161060399]
  We investigate a context-aware and dictionary-free mapping approach by leveraging parallel corpora. Our findings unfold the tight relationship between isotropy, isometry, and isomorphism in normalized contextual embedding spaces.
  arXiv  Detail & Related papers  (2021-07-19T22:57:36Z)
• The Low-Dimensional Linear Geometry of Contextualized Word Representations [27.50785941238007]
  We study the linear geometry of contextualized word representations in ELMO and BERT. We show that a variety of linguistic features are encoded in low-dimensional subspaces.
  arXiv  Detail & Related papers  (2021-05-15T00:58:08Z)
• Neural Operator: Graph Kernel Network for Partial Differential Equations [57.90284928158383]
  This work is to generalize neural networks so that they can learn mappings between infinite-dimensional spaces (operators) We formulate approximation of the infinite-dimensional mapping by composing nonlinear activation functions and a class of integral operators. Experiments confirm that the proposed graph kernel network does have the desired properties and show competitive performance compared to the state of the art solvers.
  arXiv  Detail & Related papers  (2020-03-07T01:56:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.