Semantic Structure of Feature Space in Large Language Models

• URL: http://arxiv.org/abs/2604.27169v1
• Date: Wed, 29 Apr 2026 20:17:43 GMT
• Title: Semantic Structure of Feature Space in Large Language Models
• Abstract summary: We show that the geometric relations between semantic features in large language models closely mirror human psychological associations.<n>We construct feature vectors corresponding to 360 words and project them on 32 semantic axes.<n>We find that these projections correlate highly with human ratings of those words on the respective semantic scales.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We show that the geometric relations between semantic features in large language models' hidden states closely mirror human psychological associations. We construct feature vectors corresponding to 360 words and project them on 32 semantic axes (e.g. beautiful-ugly, soft-hard), and find that these projections correlate highly with human ratings of those words on the respective semantic scales. Second, we find that the cosine similarities between the semantic axes themselves are highly predictive of the correlations between these scales in the survey. Third, we show that substantial variance across the 32 semantic axes lies on a low-dimensional subspace, reproducing patterns typical of human semantic associations. Finally, we demonstrate that steering a word on one semantic axis causes spillover effects on the model's rating of that word on other semantic scales proportionate to the cosine similarity between those semantic axes. These findings suggest that features should be understood not only in isolation but through their geometric relations and the meaningful subspaces they form.

Related papers

• Relation Geometry in Semantic Space of Language Models [35.765722891808586]
  We study the relation geometry of semantic spaces from three perspectives.<n>We first examine whether words standing in a particular relation to a target word occupy the same region in semantic space.<n>We then verify to what extent semantic spaces reflect certain well-known properties of relations, such as symmetry, asymmetry, and transitivity.
  arXiv  Detail & Related papers  (2026-07-29T11:02:47Z)
• See More, Match Better: Multi-Source Feature Fusion for Two-View Correspondence Learning [52.52121389921707]
  Two-view correspondence learning aims to distinguish true correspondences (inliers) from false ones (outliers) in image pairs.<n>Existing methods mainly rely on coordinate-based geometric consistency.<n>We propose TriMatch, a multi-source feature fusion framework for two-view correspondence learning.
  arXiv  Detail & Related papers  (2026-06-08T09:33:01Z)
• Psychological Constructs in Shared Semantic Space [0.0]
  Psychological constructs are often measured in separate instruments, datasets, and research traditions.<n>This paper proposes a framework for making such constructs semantically commensurate by representing and comparing them as directions in a shared word-embedding space.
  arXiv  Detail & Related papers  (2026-05-26T10:16:24Z)
• AffordGrasp: Cross-Modal Diffusion for Affordance-Aware Grasp Synthesis [50.793806818677716]
  AffordGrasp produces physically stable and semantically faithful human grasps with high precision.<n>AffordGrasp integrates an affordance-aware latent representation of hand poses with a dual-conditioning diffusion process.<n>We evaluate AffordGrasp across four instruction-augmented benchmarks derived from HO-3D, OakInk, GRAB, and AffordPose.
  arXiv  Detail & Related papers  (2026-03-09T06:56:35Z)
• Semantic Structure in Large Language Model Embeddings [0.0]
  Psychological research consistently finds that human ratings of words can be reduced to a low-dimensional form with relatively little information loss.<n>We show that the projections of words on semantic directions defined by antonym pairs correlate highly with human ratings.<n>We find that shifting tokens along one semantic direction causes off-target effects on geometrically aligned features proportional to their cosine similarity.
  arXiv  Detail & Related papers  (2025-08-04T20:21:50Z)
• The Shape of Word Embeddings: Quantifying Non-Isometry With Topological Data Analysis [10.242373477945376]
  We use persistent homology from topological data analysis to measure the distances between language pairs from the shape of their unlabeled embeddings. To distinguish whether these differences are random training errors or capture real information about the languages, we use the computed distance matrices to construct language phylogenetic trees over 81 Indo-European languages.
  arXiv  Detail & Related papers  (2024-03-30T23:51:25Z)
• Spoken Word2Vec: Learning Skipgram Embeddings from Speech [0.8901073744693314]
  We show how shallow skipgram-like algorithms fail to encode distributional semantics when the input units are acoustically correlated. We illustrate the potential of an alternative deep end-to-end variant of the model and examine the effects on the resulting embeddings.
  arXiv  Detail & Related papers  (2023-11-15T19:25:29Z)
• Agentivit\`a e telicit\`a in GilBERTo: implicazioni cognitive [77.71680953280436]
  The goal of this study is to investigate whether a Transformer-based neural language model infers lexical semantics. The semantic properties considered are telicity (also combined with definiteness) and agentivity.
  arXiv  Detail & Related papers  (2023-07-06T10:52:22Z)
• Syntax and Semantics Meet in the "Middle": Probing the Syntax-Semantics Interface of LMs Through Agentivity [68.8204255655161]
  We present the semantic notion of agentivity as a case study for probing such interactions. This suggests LMs may potentially serve as more useful tools for linguistic annotation, theory testing, and discovery.
  arXiv  Detail & Related papers  (2023-05-29T16:24:01Z)
• Lexical semantics enhanced neural word embeddings [4.040491121427623]
  hierarchy-fitting is a novel approach to modelling semantic similarity nuances inherently stored in the IS-A hierarchies. Results demonstrate the efficacy of hierarchy-fitting in specialising neural embeddings with semantic relations in late fusion.
  arXiv  Detail & Related papers  (2022-10-03T08:10:23Z)
• Evaluation of taxonomic and neural embedding methods for calculating semantic similarity [0.0]
  We study the mechanisms between taxonomic and distributional similarity measures. We find that taxonomic similarity measures can depend on the shortest path length as a prime factor to predict semantic similarity. The synergy of retrofitting neural embeddings with concept relations in similarity prediction may indicate a new trend to leverage knowledge bases on transfer learning.
  arXiv  Detail & Related papers  (2022-09-30T02:54:21Z)
• Image Synthesis via Semantic Composition [74.68191130898805]
  We present a novel approach to synthesize realistic images based on their semantic layouts. It hypothesizes that for objects with similar appearance, they share similar representation. Our method establishes dependencies between regions according to their appearance correlation, yielding both spatially variant and associated representations.
  arXiv  Detail & Related papers  (2021-09-15T02:26:07Z)
• Understanding Spatial Relations through Multiple Modalities [78.07328342973611]
  spatial relations between objects can either be explicit -- expressed as spatial prepositions, or implicit -- expressed by spatial verbs such as moving, walking, shifting, etc. We introduce the task of inferring implicit and explicit spatial relations between two entities in an image. We design a model that uses both textual and visual information to predict the spatial relations, making use of both positional and size information of objects and image embeddings.
  arXiv  Detail & Related papers  (2020-07-19T01:35:08Z)
• Human Correspondence Consensus for 3D Object Semantic Understanding [56.34297279246823]
  In this paper, we introduce a new dataset named CorresPondenceNet. Based on this dataset, we are able to learn dense semantic embeddings with a novel geodesic consistency loss. We show that CorresPondenceNet could not only boost fine-grained understanding of heterogeneous objects but also cross-object registration and partial object matching.
  arXiv  Detail & Related papers  (2019-12-29T04:24:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.