Geometry without Position? When Positional Embeddings Help and Hurt Spatial Reasoning

• URL: http://arxiv.org/abs/2601.22231v1
• Date: Thu, 29 Jan 2026 19:04:30 GMT
• Title: Geometry without Position? When Positional Embeddings Help and Hurt Spatial Reasoning
• Abstract summary: This paper revisits the role of positional embeddings (PEs) within vision transformers (ViTs) from a geometric perspective.<n>We show that PEs are not mere token indices but effectively function as geometric priors that shape the spatial structure of the representation.<n>Our findings clarify the role of PEs as a causal mechanism that governs spatial structure in ViT representations.
• Score: 46.701926049208545
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper revisits the role of positional embeddings (PEs) within vision transformers (ViTs) from a geometric perspective. We show that PEs are not mere token indices but effectively function as geometric priors that shape the spatial structure of the representation. We introduce token-level diagnostics that measure how multi-view geometric consistency in ViT representation depends on consitent PEs. Through extensive experiments on 14 foundation ViT models, we reveal how PEs influence multi-view geometry and spatial reasoning. Our findings clarify the role of PEs as a causal mechanism that governs spatial structure in ViT representations. Our code is provided in https://github.com/shijianjian/vit-geometry-probes

Related papers

• Transformer Geometry Observatory TGO-I: Spectral Geometry Observatory [2.4900509453584156]
  We introduce Transformer Geometry Observatory (TGO), a framework to investigate the representational geometry and dynamics of Vision Transformers.<n>TGO-I, the first installment of the framework, focuses on the spectral geometry of ViT representations.<n>Our results reveal a consistent increase in dimensional utilization, accompanied by decreasing anisotropy, increasing spectral entropy, increasing participation ratio, and progressively flatter eigenspectra.
  arXiv  Detail & Related papers  (2026-06-17T16:26:59Z)
• Positional Encodings Anchor Spatial Structure in Vision Transformers: A Geometric Perspective on Robustness [0.0]
  We study how different forms of positional embedding (PEs) influence the representational geometry of Vision Transformers (ViTs)<n>We show that ViTs trained without PEs still develop non-trivial spatial structure, but this structure is driven by visual content and collapses under token permutation.<n>We find that all PEs considered (learned absolute, sinusoidal, and rotary) are associated with a consistent shift toward an index-anchored spatial organization.
  arXiv  Detail & Related papers  (2026-05-28T13:17:23Z)
• GeoWeaver: Grounding Visual Tokens with Geometric Evidence before Scene Reasoning [45.229974852899716]
  multimodal models incorporate geometric information branches, 3D-aware supervision, reasoning-stage fusion, or long-horizon memory.<n>These approaches typically treat geometric cues as a shared signal across all visual tokens.<n>We introduce GeoWeaver, a framework that treats geometry as a representational prerequisite fortemporal reasoning.
  arXiv  Detail & Related papers  (2026-05-21T14:40:03Z)
• SpatialEvo: Self-Evolving Spatial Intelligence via Deterministic Geometric Environments [75.60795462502949]
  SpatialEvo is a self-evolving framework for 3D spatial reasoning.<n>It formalizes 16 spatial reasoning task categories under explicit geometric validation rules.<n>It converts unannotated 3D scenes into zero-noise interactive oracles, replacing model consensus with objective physical feedback.
  arXiv  Detail & Related papers  (2026-04-15T17:59:12Z)
• Geoparsing: Diagram Parsing for Plane and Solid Geometry with a Unified Formal Language [60.70116693224378]
  We design a unified formal language that integrates plane and solid geometry, comprehensively covering geometric structures and semantic relations.<n>We construct GDP-29K, a large-scale dataset comprising 20k plane and 9k solid geometry samples collected from diverse real-world sources.
  arXiv  Detail & Related papers  (2026-04-13T15:09:56Z)
• Make Geometry Matter for Spatial Reasoning [62.61667611352403]
  Vision-language models (VLMs) achieve strong image and video understanding, yet their ability to perform spatial reasoning in both static scenes and dynamic videos remains limited.<n>Recent advances try to handle this limitation by injecting geometry tokens from pretrained 3D foundation models into VLMs.<n>We propose GeoSR, a framework designed to make geometry matter by encouraging VLMs to actively reason with geometry tokens.
  arXiv  Detail & Related papers  (2026-03-27T17:45:12Z)
• The Information Geometry of Softmax: Probing and Steering [18.006877307358348]
  We argue that the natural geometry of representation spaces should reflect the way models use representations to produce behavior.<n>Our focus is on the role of information geometry on semantic encoding and the linear representation hypothesis.<n>As an illustrative application, we develop "dual steering", a method for robustly steering representations to exhibit a particular concept.
  arXiv  Detail & Related papers  (2026-02-17T01:33:28Z)
• Learning Proposes, Geometry Disposes: A Modular Framework for Efficient Spatial Reasoning [3.5072793256984105]
  Spatial perception aims to estimate camera motion and scene structure from visual observations.<n>Recent learning-based methods have demonstrated strong representational capacity for geometric perception.<n>In this work, we investigate an end-to-end modular framework for effective spatial reasoning.
  arXiv  Detail & Related papers  (2026-02-16T02:26:59Z)
• On Geometric Understanding and Learned Data Priors in VGGT [38.8968170074396]
  The Visual Geometry Grounded Transformer (VGGT) is a 3D foundation model that infers camera geometry and scene structure in a single feed-forward pass.<n>We conduct a systematic analysis of VGGT's internal mechanisms to uncover whether geometric understanding emerges within its representations.
  arXiv  Detail & Related papers  (2025-12-12T12:11:57Z)
• Visualizing LLM Latent Space Geometry Through Dimensionality Reduction [0.0]
  We extract, process, and visualize latent state geometries in Transformer-based language models through dimensionality reduction.<n>We demonstrate experiments on GPT-2 and LLaMa models, where we uncover interesting geometric patterns in latent space.
  arXiv  Detail & Related papers  (2025-11-26T17:11:39Z)
• Geometry matters: insights from Ollivier Ricci Curvature and Ricci Flow into representational alignment through Ollivier-Ricci Curvature and Ricci Flow [0.014893065504013906]
  This work introduces a framework using Ollivier Ricci Curvature and Ricci Flow to analyze the fine-grained local structure of representations.<n>We apply it to compare human similarity judgments for 2D and 3D face stimuli with a baseline 2D native network (VGG-Face) and a variant of it aligned to human behavior.
  arXiv  Detail & Related papers  (2025-01-01T18:33:48Z)
• Internal structure of gauge-invariant Projected Entangled Pair States [0.0]
  Projected entangled pair states (PEPS) allow encoding symmetries, either global or local (gauge), naturally.<n> PEPS with local symmetries have increasingly been used in the study of non-perturbative regimes of lattice gauge theories.<n>We study the internal structure of projected entangled pair states with a gauge symmetry.
  arXiv  Detail & Related papers  (2024-10-24T17:37:37Z)
• A Hitchhiker's Guide to Geometric GNNs for 3D Atomic Systems [87.30652640973317]
  Recent advances in computational modelling of atomic systems represent them as geometric graphs with atoms embedded as nodes in 3D Euclidean space. Geometric Graph Neural Networks have emerged as the preferred machine learning architecture powering applications ranging from protein structure prediction to molecular simulations and material generation. This paper provides a comprehensive and self-contained overview of the field of Geometric GNNs for 3D atomic systems.
  arXiv  Detail & Related papers  (2023-12-12T18:44:19Z)
• GeoDeformer: Geometric Deformable Transformer for Action Recognition [22.536307401874105]
  Vision transformers have recently emerged as an effective alternative to convolutional networks for action recognition. This paper proposes a novel approach, GeoDeformer, designed to capture the variations inherent in action video by integrating geometric comprehension directly into the ViT architecture.
  arXiv  Detail & Related papers  (2023-11-29T16:55:55Z)
• GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformers [63.41460219156508]
  We argue that existing positional encoding schemes are suboptimal for 3D vision tasks. We propose a geometry-aware attention mechanism that encodes the geometric structure of tokens as relative transformation. We show that our attention, called Geometric Transform Attention (GTA), improves learning efficiency and performance of state-of-the-art transformer-based NVS models.
  arXiv  Detail & Related papers  (2023-10-16T13:16:09Z)
• Geometry Interaction Knowledge Graph Embeddings [153.69745042757066]
  We propose Geometry Interaction knowledge graph Embeddings (GIE), which learns spatial structures interactively between the Euclidean, hyperbolic and hyperspherical spaces. Our proposed GIE can capture a richer set of relational information, model key inference patterns, and enable expressive semantic matching across entities.
  arXiv  Detail & Related papers  (2022-06-24T08:33:43Z)
• DeepMLS: Geometry-Aware Control Point Deformation [76.51312491336343]
  We introduce DeepMLS, a space-based deformation technique, guided by a set of displaced control points. We leverage the power of neural networks to inject the underlying shape geometry into the deformation parameters. Our technique facilitates intuitive piecewise smooth deformations, which are well suited for manufactured objects.
  arXiv  Detail & Related papers  (2022-01-05T23:55:34Z)
• PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoning [135.2892665079159]
  We introduce a new large-scale diagnostic visual reasoning dataset named PTR. PTR contains around 70k RGBD synthetic images with ground truth object and part level annotations. We examine several state-of-the-art visual reasoning models on this dataset and observe that they still make many surprising mistakes.
  arXiv  Detail & Related papers  (2021-12-09T18:59:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.