Communicability-Inspired Positional Encoding (CIPE)

• URL: http://arxiv.org/abs/2606.25293v1
• Date: Wed, 24 Jun 2026 01:56:20 GMT
• Title: Communicability-Inspired Positional Encoding (CIPE)
• Abstract summary: Communicability-Inspired Positional size (CIPE) is a principled framework for attention-compatible graph positional encodings.<n> CIPE improves structure-agnostic Transformers by 35.5% on average across seven benchmarks.<n>It also consistently improves structure-biased graph Transformers, where competing PEs often yield only marginal benefits.
• Score: 17.120735986111217
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Positional encodings (PEs) are essential for Transformers. Yet designing effective PEs for non-Euclidean graphs remains challenging. Such encodings should ideally induce an Attention-Compatible Geometry for self-attention: not merely describing graph structure, but defining a geometry whose inner products reflect meaningful structural relatedness. To realize this geometry, we propose Communicability-Inspired Positional Encoding (CIPE), built from communicability, a measure between pairs of nodes that aggregates contributions from paths of all lengths. By construction, CIPE inner products recover communicability, converting global multi-path connectivity into an attention-ready similarity geometry. For practical Transformer training, we introduce dimensionality alignment, mapping graph-size-dependent CIPE representations to prescribed dimensions while faithfully preserving the induced geometry. Empirically, CIPE improves structure-agnostic Transformers by 35.5% on average across seven benchmarks, outperforming representative PEs; it also consistently improves structure-biased graph Transformers, where competing PEs often yield only marginal benefits. These results position CIPE as a principled framework for attention-compatible graph positional encodings.

Related papers

• Weierstrass Positional Encoding for Vision Transformers [13.639230128765988]
  Vision Transformers have achieved remarkable success in computer vision, but their common use of one-dimensional positional encodings weakens the inherent two-dimensional structure of images after patch flattening.<n>We propose Weierstrass elliptic Positional geometric compute (WePE), a mathematically grounded method for encoding two-dimensional coordinates in the complex domain.<n>WePE is plug-and-play and resolution-agnostic, allowing seamless integration into existing ViTs.
  arXiv  Detail & Related papers  (2026-05-20T13:35:17Z)
• Beyond Point-Wise Matching: Structural Representation Alignment for Accelerating Diffusion Transformers [93.3976834364707]
  We propose sREPA, a structural REPresentation Alignment framework to enforce consistency in the relational geometry of feature maps.<n>By encouraging the model to internalize holistic spatial layouts and structural correlations from pre-trained features, sREPA achieves faster and more stable convergence.
  arXiv  Detail & Related papers  (2026-05-16T12:01:04Z)
• Rethinking Positional Encoding for Neural Vehicle Routing [29.116133441162173]
  We formalize three structural properties that a routing-aware PE should respect.<n>We propose a hierarchical anisometric PE that combines a distance-indexed, circularly consistent in-route encoding with a depot-anchored angular cross-route encoding.
  arXiv  Detail & Related papers  (2026-05-12T10:22:31Z)
• LG-HCC: Local Geometry-Aware Hierarchical Context Compression for 3D Gaussian Splatting [77.81227097905865]
  Anchor-based 3DGS compression schemes reduce gaussian redundancy through some advanced context models.<n>We propose a Local Geometry-aware Hierarchical Context Compression framework for 3DGS that incorporates inter-anchor geometric correlations into anchor pruning and entropy coding for compact representation.<n>Experiments show that LG-HCC effectively alleviates structural preservation issues,achieving superior geometric integrity and rendering fidelity while reducing storage by up to 30.85x compared to the Scaffold-GS baseline on the Mip-NeRF360 dataset.
  arXiv  Detail & Related papers  (2026-03-30T13:39:35Z)
• GeoReFormer: Geometry-Aware Refinement for Lane Segment Detection and Topology Reasoning [0.7769712273444055]
  GeoReFormer is a query-based architecture that embeds geometry- and topology-aware inductive biases directly within the transformer decoder.<n>GeoReFormer achieves state-of-the-art performance with 34.5% mAP on the OpenLane-V2 benchmark.
  arXiv  Detail & Related papers  (2026-03-27T02:26:12Z)
• Tab-PET: Graph-Based Positional Encodings for Tabular Transformers [10.312868216225118]
  We propose a graph-based framework for estimating and inculcating positional encodings into embeddings.<n>Inspired by approaches that derive PEs from graph topology, we explore two paradigms for graph estimation: association-based and causality-based.<n>We empirically demonstrate that graph-derived PEs significantly improve performance across 50 classification and regression datasets for 3T. Notably, association-based graphs consistently yield more stable and pronounced gains compared to causality-driven ones.
  arXiv  Detail & Related papers  (2025-11-17T13:08:34Z)
• SeqPE: Transformer with Sequential Position Encoding [76.22159277300891]
  SeqPE represents each $n$-dimensional position index as a symbolic sequence and employs a lightweight sequential position encoder to learn their embeddings.<n> Experiments across language modeling, long-context question answering, and 2D image classification demonstrate that SeqPE not only surpasses strong baselines in perplexity, exact match (EM) and accuracy--but also enables seamless generalization to multi-dimensional inputs without requiring manual architectural redesign.
  arXiv  Detail & Related papers  (2025-06-16T09:16:40Z)
• Geometry-Editable and Appearance-Preserving Object Compositon [67.98806888489385]
  General object composition (GOC) aims to seamlessly integrate a target object into a background scene with desired geometric properties.<n>Recent approaches derive semantic embeddings and integrate them into advanced diffusion models to enable geometry-editable generation.<n>We introduce a Disentangled Geometry-editable and Appearance-preserving Diffusion model that first leverages semantic embeddings to implicitly capture desired geometric transformations.
  arXiv  Detail & Related papers  (2025-05-27T09:05:28Z)
• Toward Relative Positional Encoding in Spiking Transformers [76.72869420863749]
  Spiking neural networks (SNNs) are bio-inspired networks that mimic how neurons in the brain communicate through discrete spikes.<n>We introduce several strategies to approximate relative positional encoding in spiking Transformers while preserving the binary nature of spikes.
  arXiv  Detail & Related papers  (2025-01-28T06:42:37Z)
• 3D Geometric Shape Assembly via Efficient Point Cloud Matching [59.241448711254485]
  We introduce Proxy Match Transform (PMT), an approximate high-order feature transform layer that enables reliable matching between mating surfaces of parts. Building upon PMT, we introduce a new framework, dubbed Proxy Match TransformeR (PMTR), for the geometric assembly task. We evaluate the proposed PMTR on the large-scale 3D geometric shape assembly benchmark dataset of Breaking Bad.
  arXiv  Detail & Related papers  (2024-07-15T08:50:02Z)
• GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformers [63.41460219156508]
  We argue that existing positional encoding schemes are suboptimal for 3D vision tasks. We propose a geometry-aware attention mechanism that encodes the geometric structure of tokens as relative transformation. We show that our attention, called Geometric Transform Attention (GTA), improves learning efficiency and performance of state-of-the-art transformer-based NVS models.
  arXiv  Detail & Related papers  (2023-10-16T13:16:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.