Relation Geometry in Semantic Space of Language Models

• URL: http://arxiv.org/abs/2607.26762v1
• Date: Wed, 29 Jul 2026 11:02:47 GMT
• Title: Relation Geometry in Semantic Space of Language Models
• Abstract summary: We study the relation geometry of semantic spaces from three perspectives.<n>We first examine whether words standing in a particular relation to a target word occupy the same region in semantic space.<n>We then verify to what extent semantic spaces reflect certain well-known properties of relations, such as symmetry, asymmetry, and transitivity.
• Score: 35.765722891808586
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: When it comes to generating vector representations of words, current language models are achieving high-quality results. However, what is not known is the extent to which knowledge about semantic relations is represented in the geometry of the semantic spaces created in this way. In order to answer this question, we study the relation geometry of such semantic spaces from three perspectives. We first examine whether words standing in a particular relation to a target word~(called relata) occupy the same region in semantic space, and whether the regions corresponding to different relations are distinct from each other. We then verify to what extent semantic spaces reflect certain well-known properties of relations, such as symmetry, asymmetry, and transitivity. Finally, we consider which information about the target words and relata is more important for relation geometry: their surface forms, or their contexts. We conduct experiments on six semantic relations using causal, masked, and diffusion language models. The results show that relata in asymmetric relations relatively clearly occupy a distinct region in semantic space. Asymmetric relations' properties are only moderately well encoded in the semantic space, yet better than those of symmetric ones. Furthermore, when considering the question which information source has the strongest impact on results amongst the models we evaluated, we find that lexical information tends to be more important for the causal language model, whereas contextual information is more important for the masked and diffusion language models. Our results empirically show that relation geometry is not equally well-represented for all relations in semantic space, suggesting that there is a difference in how well semantic relations might be learned from distributional information alone.

Related papers

• See More, Match Better: Multi-Source Feature Fusion for Two-View Correspondence Learning [52.52121389921707]
  Two-view correspondence learning aims to distinguish true correspondences (inliers) from false ones (outliers) in image pairs.<n>Existing methods mainly rely on coordinate-based geometric consistency.<n>We propose TriMatch, a multi-source feature fusion framework for two-view correspondence learning.
  arXiv  Detail & Related papers  (2026-06-08T09:33:01Z)
• Semantic Structure of Feature Space in Large Language Models [0.0]
  We show that the geometric relations between semantic features in large language models closely mirror human psychological associations.<n>We construct feature vectors corresponding to 360 words and project them on 32 semantic axes.<n>We find that these projections correlate highly with human ratings of those words on the respective semantic scales.
  arXiv  Detail & Related papers  (2026-04-29T20:17:43Z)
• Semantic Structure in Large Language Model Embeddings [0.0]
  Psychological research consistently finds that human ratings of words can be reduced to a low-dimensional form with relatively little information loss.<n>We show that the projections of words on semantic directions defined by antonym pairs correlate highly with human ratings.<n>We find that shifting tokens along one semantic direction causes off-target effects on geometrically aligned features proportional to their cosine similarity.
  arXiv  Detail & Related papers  (2025-08-04T20:21:50Z)
• Skews in the Phenomenon Space Hinder Generalization in Text-to-Image Generation [59.138470433237615]
  We introduce statistical metrics that quantify both the linguistic and visual skew of a dataset for relational learning. We show that systematically controlled metrics are strongly predictive of generalization performance. This work informs an important direction towards quality-enhancing the data diversity or balance to scaling up the absolute size.
  arXiv  Detail & Related papers  (2024-03-25T03:18:39Z)
• Discovering Fine-Grained Semantics in Knowledge Graph Relations [5.619233302594469]
  Polysemous relations between different types of entities represent multiple semantics. For numerous use cases, such as entity type classification, question answering and knowledge graph completion, the correct semantic interpretation is necessary. We propose a strategy for discovering the different semantics associated with abstract relations and deriving many sub-relations with fine-grained meaning.
  arXiv  Detail & Related papers  (2022-02-17T22:05:41Z)
• Understanding Spatial Relations through Multiple Modalities [78.07328342973611]
  spatial relations between objects can either be explicit -- expressed as spatial prepositions, or implicit -- expressed by spatial verbs such as moving, walking, shifting, etc. We introduce the task of inferring implicit and explicit spatial relations between two entities in an image. We design a model that uses both textual and visual information to predict the spatial relations, making use of both positional and size information of objects and image embeddings.
  arXiv  Detail & Related papers  (2020-07-19T01:35:08Z)
• Geometry of Similarity Comparisons [51.552779977889045]
  We show that the ordinal capacity of a space form is related to its dimension and the sign of its curvature. More importantly, we show that the statistical behavior of the ordinal spread random variables defined on a similarity graph can be used to identify its underlying space form.
  arXiv  Detail & Related papers  (2020-06-17T13:37:42Z)
• Synonymy = Translational Equivalence [6.198307677263333]
  Synonymy and translational equivalence are the relations of sameness of meaning within and across languages. This paper proposes a unifying treatment of these two relations, which is validated by experiments on existing resources.
  arXiv  Detail & Related papers  (2020-04-28T23:15:02Z)
• Multiplex Word Embeddings for Selectional Preference Acquisition [70.33531759861111]
  We propose a multiplex word embedding model, which can be easily extended according to various relations among words. Our model can effectively distinguish words with respect to different relations without introducing unnecessary sparseness.
  arXiv  Detail & Related papers  (2020-01-09T04:47:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.