Give it Space! Explicit Disentangling of Positional and Semantic Representations in Encoders

• URL: http://arxiv.org/abs/2605.30022v1
• Date: Thu, 28 May 2026 14:42:25 GMT
• Title: Give it Space! Explicit Disentangling of Positional and Semantic Representations in Encoders
• Abstract summary: Positional encoding (PE) underpins how permutation-invariant Transformers represent sequence order.<n>PE methods such as RoPE still struggle on tasks such as long-context understanding or retrieval.<n>We build on evidence that positional and semantic signals occupy nearly confine subspaces in trained Transformers.
• Score: 8.72344410197391
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Positional encoding (PE) underpins how permutation-invariant Transformers represent sequence order, yet how positional information is processed and stored remains poorly understood. Modern PE methods such as RoPE still struggle on tasks such as long-context understanding or retrieval \cite{chen-etal-2025-hope}. Hence, a better understanding of the internal positional mechanism could help design better PE. Building on evidence that positional and semantic signals occupy nearly orthogonal subspaces in trained Transformers, we modify an encoder Transformer to process three explicitly disentangled streams: semantic, absolute positional (AP) and relative positional (RP), and confine the masked-language-modeling (MLM) objective to the semantic stream. This decoupling enables a clean mechanistic study and yields three take-aways. (1) The isolated AP subspace spontaneously collapses into a low-frequency two-dimensional manifold that captures the structure of the document; (2) Attention heads specialize into structure and semantic-oriented groups, with RP exclusively supporting the latter; (3) Standard positional encodings do not robustly retain macroscopic structure: RoPE and RP only weakly encode it, and entangled AP loses it in the final layers under MLM pressure. The disentangled approach preserves positional encoding, which improves linguistic representation on 49 of the 65 linguistic phenomena of the Flash-Holmes probing benchmark.

Related papers

• MDLMPE: Distribution Aware Positional Encoding for Masked Diffusion Language Models [10.872454179321808]
  Masked diffusion language models produce dynamic, non-contiguous configurations of revealed and masked tokens.<n>We propose MDLMPE, a positional encoding designed specifically for masked diffusion.<n>Experiments on LLaDA and DREAM demonstrate that MDLMPE generally outperforms conventional positional encoding methods.
  arXiv  Detail & Related papers  (2026-08-04T14:54:14Z)
• HeRO: Hierarchical 3D Semantic Representation for Pose-aware Object Manipulation [54.325346533275074]
  HeRO is a diffusion-based policy that couples geometry and semantics via hierarchical semantic fields.<n>In various tests, HeRO establishes a new state-of-the-art, improving success on Place Dual Shoes by 12.3% and averaging 6.5% gains across six challenging pose-aware tasks.
  arXiv  Detail & Related papers  (2026-02-21T12:29:10Z)
• Untwisting RoPE: Frequency Control for Shared Attention in DiTs [84.14005261938284]
  Positional encodings are essential to transformer-based generative models.<n>We show that Rotary Positional Embeddings (RoPE) naturally decomposes into frequency components with distinct positional sensitivities.<n>We introduce a method for selectively modulating RoPE frequency bands so that attention reflects semantic similarity rather than strict positional alignment.
  arXiv  Detail & Related papers  (2026-02-04T20:01:59Z)
• Do traveling waves make good positional encodings? [44.55744608160896]
  We propose RollPE, a novel positional encoding mechanism based on traveling waves.<n>We show it significantly outperforms traditional absolute positional embeddings.<n>We derive a mathematical equivalence of RollPE to a particular configuration of RoPE.
  arXiv  Detail & Related papers  (2025-11-11T14:32:45Z)
• SeqPE: Transformer with Sequential Position Encoding [76.22159277300891]
  SeqPE represents each $n$-dimensional position index as a symbolic sequence and employs a lightweight sequential position encoder to learn their embeddings.<n> Experiments across language modeling, long-context question answering, and 2D image classification demonstrate that SeqPE not only surpasses strong baselines in perplexity, exact match (EM) and accuracy--but also enables seamless generalization to multi-dimensional inputs without requiring manual architectural redesign.
  arXiv  Detail & Related papers  (2025-06-16T09:16:40Z)
• Revisiting LRP: Positional Attribution as the Missing Ingredient for Transformer Explainability [53.21677928601684]
  Layer-wise relevance propagation is one of the most promising approaches to explainability in deep learning.<n>We propose specialized theoretically-grounded LRP rules designed to propagate attributions across various positional encoding methods.<n>Our method significantly outperforms the state-of-the-art in both vision and NLP explainability tasks.
  arXiv  Detail & Related papers  (2025-06-02T18:07:55Z)
• PaTH Attention: Position Encoding via Accumulating Householder Transformations [56.32365080761523]
  PaTH is a flexible data-dependent position encoding scheme based on accumulated products of Householder transformations.<n>We derive an efficient parallel algorithm for training through exploiting a compact representation of products of Householder matrices.
  arXiv  Detail & Related papers  (2025-05-22T08:36:09Z)
• The Locality and Symmetry of Positional Encodings [9.246374019271938]
  We conduct a systematic study of positional encodings in textbfBi Masked Language Models (BERT-style) We uncover the core function of PEs by identifying two common properties, Locality and Symmetry. We quantify the weakness of current PEs by introducing two new probing tasks, on which current PEs perform poorly.
  arXiv  Detail & Related papers  (2023-10-19T16:15:15Z)
• Spherical Position Encoding for Transformers [0.0]
  We introduce the notion of "geotokens" which are input elements for transformer architectures. Unlike the natural language the sequential position is not important for the model but the geographical coordinates are. We formulate a position encoding mechanism based on RoPE architecture which is adjusted for spherical coordinates.
  arXiv  Detail & Related papers  (2023-10-04T09:28:59Z)
• Guiding the PLMs with Semantic Anchors as Intermediate Supervision: Towards Interpretable Semantic Parsing [57.11806632758607]
  We propose to incorporate the current pretrained language models with a hierarchical decoder network. By taking the first-principle structures as the semantic anchors, we propose two novel intermediate supervision tasks. We conduct intensive experiments on several semantic parsing benchmarks and demonstrate that our approach can consistently outperform the baselines.
  arXiv  Detail & Related papers  (2022-10-04T07:27:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.