Steering Embedding Models with Geometric Rotation: Mapping Semantic Relationships Across Languages and Models

• URL: http://arxiv.org/abs/2510.09790v1
• Date: Fri, 10 Oct 2025 18:51:32 GMT
• Title: Steering Embedding Models with Geometric Rotation: Mapping Semantic Relationships Across Languages and Models
• Authors: Michael Freenor, Lauren Alvarez,
• Abstract summary: We introduce Rotor-Invariant Shift Estimation (RISE), a geometric approach that represents semantic transformations as consistent rotational operations in embedding space.<n>RISE operations have the ability to operate across both languages and models with high transfer of performance.<n>This work provides the first systematic demonstration that discourse-level semantic transformations correspond to consistent geometric operations in multilingual embedding spaces.
• Score: 2.3204178451683264
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding how language and embedding models encode semantic relationships is fundamental to model interpretability and control. While early word embeddings exhibited intuitive vector arithmetic (''king'' - ''man'' + ''woman'' = ''queen''), modern high-dimensional text representations lack straightforward interpretable geometric properties. We introduce Rotor-Invariant Shift Estimation (RISE), a geometric approach that represents semantic transformations as consistent rotational operations in embedding space, leveraging the manifold structure of modern language representations. RISE operations have the ability to operate across both languages and models with high transfer of performance, suggesting the existence of analogous cross-lingual geometric structure. We evaluate RISE across three embedding models, three datasets, and seven morphologically diverse languages in five major language groups. Our results demonstrate that RISE consistently maps discourse-level semantic transformations with distinct grammatical features (e.g., negation and conditionality) across languages and models. This work provides the first systematic demonstration that discourse-level semantic transformations correspond to consistent geometric operations in multilingual embedding spaces, empirically supporting the Linear Representation Hypothesis at the sentence level.

Related papers

• Geometric Patterns of Meaning: A PHATE Manifold Analysis of Multi-lingual Embeddings [0.0]
  We introduce a multi-level analysis framework for examining semantic geometry in multilingual embeddings, implemented through Semanscope.<n>Analysis of diverse datasets spanning sub-character components, alphabetic systems, semantic domains, and numerical concepts reveals systematic geometric patterns and critical limitations in current embedding models.<n>These findings establish PHATE manifold learning as an essential analytic tool not only for studying geometric structure of meaning in embedding space, but also for validating the effectiveness of embedding models in capturing semantic relationships.
  arXiv  Detail & Related papers  (2025-12-29T14:00:12Z)
• Hierarchical Neural Semantic Representation for 3D Semantic Correspondence [72.8101601086805]
  We design the hierarchical neural semantic representation (HNSR), which consists of a global semantic feature to capture high-level structure and multi-resolution local geometric features.<n>Second, we design a progressive global-to-local matching strategy, which establishes coarse semantic correspondence using the global semantic feature.<n>Third, our framework is training-free and broadly compatible with various pre-trained 3D generative backbones, demonstrating strong generalization across diverse shape categories.
  arXiv  Detail & Related papers  (2025-09-22T07:23:07Z)
• Geometry of Semantics in Next-Token Prediction: How Optimization Implicitly Organizes Linguistic Representations [34.88156871518115]
  Next-token prediction (NTP) optimization leads language models to extract and organize semantic structure from text.<n>We demonstrate that concepts corresponding to larger singular values are learned earlier during training, yielding a natural semantic hierarchy.<n>This insight motivates orthant-based clustering, a method that combines concept signs to identify interpretable semantic categories.
  arXiv  Detail & Related papers  (2025-05-13T08:46:04Z)
• Large Spatial Model: End-to-end Unposed Images to Semantic 3D [79.94479633598102]
  Large Spatial Model (LSM) processes unposed RGB images directly into semantic radiance fields. LSM simultaneously estimates geometry, appearance, and semantics in a single feed-forward operation. It can generate versatile label maps by interacting with language at novel viewpoints.
  arXiv  Detail & Related papers  (2024-10-24T17:54:42Z)
• Geometric Signatures of Compositionality Across a Language Model's Lifetime [47.25475802128033]
  We study whether contemporary language models reflect intrinsic simplicity of language enabled by compositionality.<n>We find that the relationship between compositionality and geometric complexity arises due to learned linguistic features over training.<n>Our analyses reveal a striking contrast between nonlinear and linear dimensionality, showing they respectively encode semantic and superficial aspects of linguistic composition.
  arXiv  Detail & Related papers  (2024-10-02T11:54:06Z)
• Emergence of a High-Dimensional Abstraction Phase in Language Transformers [47.60397331657208]
  A language model (LM) is a mapping from a linguistic context to an output token.<n>We take a high-level geometric approach to its analysis, observing, across five pre-trained transformer-based LMs and three input datasets.<n>Our results suggest that a central high-dimensionality phase underlies core linguistic processing in many common LM architectures.
  arXiv  Detail & Related papers  (2024-05-24T11:49:07Z)
• Syntax and Semantics Meet in the "Middle": Probing the Syntax-Semantics Interface of LMs Through Agentivity [68.8204255655161]
  We present the semantic notion of agentivity as a case study for probing such interactions. This suggests LMs may potentially serve as more useful tools for linguistic annotation, theory testing, and discovery.
  arXiv  Detail & Related papers  (2023-05-29T16:24:01Z)
• Variational Cross-Graph Reasoning and Adaptive Structured Semantics Learning for Compositional Temporal Grounding [143.5927158318524]
  Temporal grounding is the task of locating a specific segment from an untrimmed video according to a query sentence. We introduce a new Compositional Temporal Grounding task and construct two new dataset splits. We argue that the inherent structured semantics inside the videos and language is the crucial factor to achieve compositional generalization.
  arXiv  Detail & Related papers  (2023-01-22T08:02:23Z)
• Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale [31.293175512404172]
  We introduce Transformer Grammars -- a class of Transformer language models that combine expressive power, scalability, and strong performance of Transformers. We find that Transformer Grammars outperform various strong baselines on multiple syntax-sensitive language modeling evaluation metrics.
  arXiv  Detail & Related papers  (2022-03-01T17:22:31Z)
• Oracle Linguistic Graphs Complement a Pretrained Transformer Language Model: A Cross-formalism Comparison [13.31232311913236]
  We examine the extent to which, in principle, linguistic graph representations can complement and improve neural language modeling. We find that, overall, semantic constituency structures are most useful to language modeling performance.
  arXiv  Detail & Related papers  (2021-12-15T04:29:02Z)
• Learning Universal Representations from Word to Sentence [89.82415322763475]
  This work introduces and explores the universal representation learning, i.e., embeddings of different levels of linguistic unit in a uniform vector space. We present our approach of constructing analogy datasets in terms of words, phrases and sentences. We empirically verify that well pre-trained Transformer models incorporated with appropriate training settings may effectively yield universal representation.
  arXiv  Detail & Related papers  (2020-09-10T03:53:18Z)
• A Systematic Analysis of Morphological Content in BERT Models for Multiple Languages [2.345305607613153]
  This work describes experiments which probe the hidden representations of several BERT-style models for morphological content. The goal is to examine the extent to which discrete linguistic structure, in the form of morphological features and feature values, presents itself in the vector representations and attention distributions of pre-trained language models for five European languages.
  arXiv  Detail & Related papers  (2020-04-06T22:50:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.