Semantic Shift: the Fundamental Challenge in Text Embedding and Retrieval

• URL: http://arxiv.org/abs/2603.21437v1
• Date: Sun, 22 Mar 2026 22:58:11 GMT
• Title: Semantic Shift: the Fundamental Challenge in Text Embedding and Retrieval
• Abstract summary: Transformer-based embedding models rely on pooling to map variable-length text into a single vector.<n>We argue that the missing causal factor is emphsemantic shift: the intrinsic, structured evolution and dispersion of semantics within a text.
• Score: 39.73839943827917
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformer-based embedding models rely on pooling to map variable-length text into a single vector, enabling efficient similarity search but also inducing well-known geometric pathologies such as anisotropy and length-induced embedding collapse. Existing accounts largely describe \emph{what} these pathologies look like, yet provide limited insight into \emph{when} and \emph{why} they harm downstream retrieval. In this work, we argue that the missing causal factor is \emph{semantic shift}: the intrinsic, structured evolution and dispersion of semantics within a text. We first present a theoretical analysis of \emph{semantic smoothing} in Transformer embeddings: as the semantic diversity among constituent sentences increases, the pooled representation necessarily shifts away from every individual sentence embedding, yielding a smoothed and less discriminative vector. Building on this foundation, we formalize semantic shift as a computable measure integrating local semantic evolution and global semantic dispersion. Through controlled experiments across corpora and multiple embedding models, we show that semantic shift aligns closely with the severity of embedding concentration and predicts retrieval degradation, whereas text length alone does not. Overall, semantic shift offers a unified and actionable lens for understanding embedding collapse and for diagnosing when anisotropy becomes harmful.

Related papers

• Prototype-Anchored Generalized Manifold Regression for Unknown-Domain Object Detection [58.25418970608328]
  We study Single-Domain Generalized Object Detection (Single-DGOD), which aims to transfer a detector trained on a single source domain to multiple unseen domains.<n>Existing methods mainly rely on simulation-driven strategies, such as data augmentation or textual prompts, to enlarge the training distribution.<n>We propose Manifold Regression with Visual-Text Dual Chain-of-Thought (MR-DCoT), which formulates unknown-domain generalization as a manifold regression problem.
  arXiv  Detail & Related papers  (2026-07-08T09:27:15Z)
• Structure Before Collapse: Transient semantic geometry in next-token prediction [33.942301586178914]
  Neural Collapse predicts that balanced one-hot classification pushes model representations to be equally far from each other.<n>This creates a puzzle: next-token prediction language models are trained predominantly with one-hot labels.<n>We find that semantic geometry emerges early in training, and that representations cluster by shared attributes.
  arXiv  Detail & Related papers  (2026-06-25T08:33:34Z)
• Asymptotic Semantic Collapse in Hierarchical Optimization [0.5729426778193398]
  Multi-agent language systems can exhibit a failure mode where a shared dominant context progressively absorbs individual semantics.<n>We study this effect under the name Asymptotic Semantic Collapse in Hierarchical Optimization.<n>We show that repeated interactions with Peripheral Agent Nodes drive an alignment that minimizes a global loss.
  arXiv  Detail & Related papers  (2026-02-01T00:02:01Z)
• One Swallow Does Not Make a Summer: Understanding Semantic Structures in Embedding Spaces [17.173074024116477]
  Embedding spaces are fundamental to modern AI, translating raw data into high-dimensional vectors that encode rich semantic relationships.<n>We introduce the Semantic Field Subspace (SFS), a geometry-preserving, context-aware representation that captures local semantic neighborhoods within the embedding space.<n>We also propose SAFARI, an unsupervised, modality-agnostic algorithm that uncovers hierarchical semantic structures using a novel metric called Semantic Shift.
  arXiv  Detail & Related papers  (2025-11-30T11:48:00Z)
• Distributional Semantics Tracing: A Framework for Explaining Hallucinations in Large Language Models [4.946483489399819]
  Large Language Models (LLMs) are prone to hallucination, the generation of factually incorrect statements.<n>This work investigates the intrinsic, architectural origins of this failure mode through three primary contributions.
  arXiv  Detail & Related papers  (2025-10-07T16:40:31Z)
• Analyzing Continuous Semantic Shifts with Diachronic Word Similarity Matrices [21.852268893122115]
  We propose a simple yet intuitive framework for how semantic shifts occur over multiple time periods.<n>We compute a diachronic word similarity matrix using fast and lightweight word embeddings.<n>We can categorize words that exhibit similar behavior of semantic shift in an unsupervised manner.
  arXiv  Detail & Related papers  (2025-01-16T13:42:09Z)
• SemFlow: Binding Semantic Segmentation and Image Synthesis via Rectified Flow [94.90853153808987]
  Semantic segmentation and semantic image synthesis are representative tasks in visual perception and generation. We propose a unified framework (SemFlow) and model them as a pair of reverse problems. Experiments show that our SemFlow achieves competitive results on semantic segmentation and semantic image synthesis tasks.
  arXiv  Detail & Related papers  (2024-05-30T17:34:40Z)
• How Do Transformers Learn Topic Structure: Towards a Mechanistic Understanding [56.222097640468306]
  We provide mechanistic understanding of how transformers learn "semantic structure" We show, through a combination of mathematical analysis and experiments on Wikipedia data, that the embedding layer and the self-attention layer encode the topical structure.
  arXiv  Detail & Related papers  (2023-03-07T21:42:17Z)
• Information Topology [6.0044467881527614]
  We introduce emphInformation Topology, a framework that unifies information theory and algebraic topology.<n>The starting point is the emphdot-cycle dichotomy, which separates pointwise, order-sensitive fluctuations (dots) from order-invariant, predictive structure (cycles)<n>We then define emphhomological capacity, the topological dual of Shannon capacity, as the number of independent informational cycles supported by a system.
  arXiv  Detail & Related papers  (2022-10-07T23:54:30Z)
• Contextualized Semantic Distance between Highly Overlapped Texts [85.1541170468617]
  Overlapping frequently occurs in paired texts in natural language processing tasks like text editing and semantic similarity evaluation. This paper aims to address the issue with a mask-and-predict strategy. We take the words in the longest common sequence as neighboring words and use masked language modeling (MLM) to predict the distributions on their positions. Experiments on Semantic Textual Similarity show NDD to be more sensitive to various semantic differences, especially on highly overlapped paired texts.
  arXiv  Detail & Related papers  (2021-10-04T03:59:15Z)
• Unsupervised Distillation of Syntactic Information from Contextualized Word Representations [62.230491683411536]
  We tackle the task of unsupervised disentanglement between semantics and structure in neural language representations. To this end, we automatically generate groups of sentences which are structurally similar but semantically different. We demonstrate that our transformation clusters vectors in space by structural properties, rather than by lexical semantics.
  arXiv  Detail & Related papers  (2020-10-11T15:13:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.