Weierstrass Positional Encoding for Vision Transformers

• URL: http://arxiv.org/abs/2605.23719v1
• Date: Wed, 20 May 2026 13:35:17 GMT
• Title: Weierstrass Positional Encoding for Vision Transformers
• Abstract summary: Vision Transformers have achieved remarkable success in computer vision, but their common use of one-dimensional positional encodings weakens the inherent two-dimensional structure of images after patch flattening.<n>We propose Weierstrass elliptic Positional geometric compute (WePE), a mathematically grounded method for encoding two-dimensional coordinates in the complex domain.<n>WePE is plug-and-play and resolution-agnostic, allowing seamless integration into existing ViTs.
• Score: 13.639230128765988
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision Transformers have achieved remarkable success in computer vision, but their common use of learnable one-dimensional positional encodings weakens the inherent two-dimensional spatial structure of images after patch flattening. Existing positional encodings often lack geometric constraints and do not preserve a monotonic relationship between Euclidean spatial distances and sequential index distances, limiting ViTs' ability to exploit spatial proximity priors. Motivated by the usefulness of periodicity in positional encoding, we propose Weierstrass elliptic Positional Encoding (WePE), a mathematically grounded method for encoding two-dimensional coordinates in the complex domain. WePE maps normalized 2D patch coordinates onto the complex plane and constructs compact four-dimensional positional features using the Weierstrass elliptic function and its derivative. The double periodicity provides a principled representation of 2D positions, and its intrinsic lattice structure naturally matches the regular geometry of image patch grids. Its nonlinear geometric properties help model spatial distance relationships more faithfully, while the algebraic addition formula enables relative positional information between arbitrary patch pairs to be derived directly from their absolute encodings. WePE is plug-and-play and resolution-agnostic, allowing seamless integration into existing ViTs. Extensive experiments show that WePE brings consistent performance gains in most settings. With precomputed lookup tables, these improvements introduce no noticeable computational or memory overhead. Additional analyses and ablation studies further validate the effectiveness of the proposed method.

Related papers

• Understanding Geometric Representations in Self-Supervised Vision Transformers via Subspace Intervention [7.887469434417159]
  We introduce a controlled subspace intervention framework to investigate how self-supervised Vision Transformers (ViTs) encode dense geometric information.<n>By connecting internal encoding mechanics with downstream performance, these findings provide a basis for effective feature selection and lightweight decoder design.
  arXiv  Detail & Related papers  (2026-07-02T10:18:02Z)
• URoPE: Universal Relative Position Embedding across Geometric Spaces [55.651792747815854]
  URoPE is a universal extension of Rotary Position Embedding (RoPE) to cross-view or cross-dimensional geometric spaces.<n>For each key/value image patch, URoPE samples 3D points along the corresponding camera ray at predefined depth anchors.<n>Standard 2D RoPE can then be applied using the projected pixel coordinates.
  arXiv  Detail & Related papers  (2026-04-20T18:52:03Z)
• INST-Align: Implicit Neural Alignment for Spatial Transcriptomics via Canonical Expression Fields [1.0487944945684993]
  INST-Align couples a coordinate-based deformation network with a shared Canonical Expression Field.<n>It achieves state-of-the-art mean OT Accuracy (0.702), NN Accuracy (0.719), and Chamfer distance, with Chamfer reductions of up to 94.9% on large-deformation sections.<n>It also yields biologically meaningful spatial embeddings and coherent 3D tissue reconstruction.
  arXiv  Detail & Related papers  (2026-04-13T21:44:18Z)
• GeoPE:A Unified Geometric Positional Embedding for Structured Tensors [12.459742491179947]
  We introduce Geometric Positional Embedding (GeoPE), a framework that extends rotations to 3D Euclidean space using quaternions.<n>To overcome non-commutativity and ensure symmetry, GeoPE constructs a unified rotational operator by computing the geometric mean in the Lie algebra.<n>Experiments on image classification, object detection, and 3D semantic segmentation demonstrate that GeoPE consistently outperforms existing 2D RoPE variants.
  arXiv  Detail & Related papers  (2025-12-04T16:31:12Z)
• Beyond flattening: a geometrically principled positional encoding for vision transformers with Weierstrass elliptic functions [2.8199098530835127]
  Vision Transformers have demonstrated remarkable success in computer vision tasks.<n>Traditional positional encoding approaches fail to establish monotonic correspondence between Euclidean spatial distances and sequential index distances.<n>We propose WEF-PE, a mathematically principled approach that directly addresses embedding two-dimensional coordinates through natural complex domain representation.
  arXiv  Detail & Related papers  (2025-08-26T16:14:59Z)
• SeqPE: Transformer with Sequential Position Encoding [76.22159277300891]
  SeqPE represents each $n$-dimensional position index as a symbolic sequence and employs a lightweight sequential position encoder to learn their embeddings.<n> Experiments across language modeling, long-context question answering, and 2D image classification demonstrate that SeqPE not only surpasses strong baselines in perplexity, exact match (EM) and accuracy--but also enables seamless generalization to multi-dimensional inputs without requiring manual architectural redesign.
  arXiv  Detail & Related papers  (2025-06-16T09:16:40Z)
• Geometry-Editable and Appearance-Preserving Object Compositon [67.98806888489385]
  General object composition (GOC) aims to seamlessly integrate a target object into a background scene with desired geometric properties.<n>Recent approaches derive semantic embeddings and integrate them into advanced diffusion models to enable geometry-editable generation.<n>We introduce a Disentangled Geometry-editable and Appearance-preserving Diffusion model that first leverages semantic embeddings to implicitly capture desired geometric transformations.
  arXiv  Detail & Related papers  (2025-05-27T09:05:28Z)
• Parallel Sequence Modeling via Generalized Spatial Propagation Network [80.66202109995726]
  Generalized Spatial Propagation Network (GSPN) is a new attention mechanism for optimized vision tasks that inherently captures 2D spatial structures.<n>GSPN overcomes limitations by directly operating on spatially coherent image data and forming dense pairwise connections through a line-scan approach.<n>GSPN achieves superior spatial fidelity and state-of-the-art performance in vision tasks, including ImageNet classification, class-guided image generation, and text-to-image generation.
  arXiv  Detail & Related papers  (2025-01-21T18:56:19Z)
• Neural Isometries: Taming Transformations for Equivariant ML [8.203292895010748]
  We introduce Neural Isometries, an autoencoder framework which learns to map the observation space to a general-purpose latent space. We show that a simple off-the-shelf equivariant network operating in the pre-trained latent space can achieve results on par with meticulously-engineered, handcrafted networks.
  arXiv  Detail & Related papers  (2024-05-29T17:24:25Z)
• GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformers [63.41460219156508]
  We argue that existing positional encoding schemes are suboptimal for 3D vision tasks. We propose a geometry-aware attention mechanism that encodes the geometric structure of tokens as relative transformation. We show that our attention, called Geometric Transform Attention (GTA), improves learning efficiency and performance of state-of-the-art transformer-based NVS models.
  arXiv  Detail & Related papers  (2023-10-16T13:16:09Z)
• NDC-Scene: Boost Monocular 3D Semantic Scene Completion in Normalized Device Coordinates Space [77.6067460464962]
  Monocular 3D Semantic Scene Completion (SSC) has garnered significant attention in recent years due to its potential to predict complex semantics and geometry shapes from a single image, requiring no 3D inputs. We identify several critical issues in current state-of-the-art methods, including the Feature Ambiguity of projected 2D features in the ray to the 3D space, the Pose Ambiguity of the 3D convolution, and the Imbalance in the 3D convolution across different depth levels. We devise a novel Normalized Device Coordinates scene completion network (NDC-Scene) that directly extends the 2
  arXiv  Detail & Related papers  (2023-09-26T02:09:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.