Learnable Spatial-Temporal Positional Encoding for Link Prediction

• URL: http://arxiv.org/abs/2506.08309v2
• Date: Wed, 11 Jun 2025 03:31:52 GMT
• Title: Learnable Spatial-Temporal Positional Encoding for Link Prediction
• Authors: Katherine Tieu, Dongqi Fu, Zihao Li, Ross Maciejewski, Jingrui He,
• Abstract summary: We propose a simple temporal link prediction model named L-STEP.<n>L-STEP can preserve the graph property from the spatial-temporal spectral viewpoint.<n>L-STEP obtains the leading performance in the newest large-scale TGB benchmark.
• Score: 44.0907827498725
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Accurate predictions rely on the expressiveness power of graph deep learning frameworks like graph neural networks and graph transformers, where a positional encoding mechanism has become much more indispensable in recent state-of-the-art works to record the canonical position information. However, the current positional encoding is limited in three aspects: (1) most positional encoding methods use pre-defined, and fixed functions, which are inadequate to adapt to the complex attributed graphs; (2) a few pioneering works proposed the learnable positional encoding but are still limited to the structural information, not considering the real-world time-evolving topological and feature information; (3) most positional encoding methods are equipped with transformers' attention mechanism to fully leverage their capabilities, where the dense or relational attention is often unaffordable on large-scale structured data. Hence, we aim to develop Learnable Spatial-Temporal Positional Encoding in an effective and efficient manner and propose a simple temporal link prediction model named L-STEP. Briefly, for L-STEP, we (1) prove the proposed positional learning scheme can preserve the graph property from the spatial-temporal spectral viewpoint, (2) verify that MLPs can fully exploit the expressiveness and reach transformers' performance on that encoding, (3) change different initial positional encoding inputs to show robustness, (4) analyze the theoretical complexity and obtain less empirical running time than SOTA, and (5) demonstrate its temporal link prediction out-performance on 13 classic datasets and with 10 algorithms in both transductive and inductive settings using 3 different sampling strategies. Also, L-STEP obtains the leading performance in the newest large-scale TGB benchmark. Our code is available at https://github.com/kthrn22/L-STEP.

Related papers

• SeqPE: Transformer with Sequential Position Encoding [76.22159277300891]
  SeqPE represents each $n$-dimensional position index as a symbolic sequence and employs a lightweight sequential position encoder to learn their embeddings.<n> Experiments across language modeling, long-context question answering, and 2D image classification demonstrate that SeqPE not only surpasses strong baselines in perplexity, exact match (EM) and accuracy--but also enables seamless generalization to multi-dimensional inputs without requiring manual architectural redesign.
  arXiv  Detail & Related papers  (2025-06-16T09:16:40Z)
• Revisiting LRP: Positional Attribution as the Missing Ingredient for Transformer Explainability [53.21677928601684]
  Layer-wise relevance propagation is one of the most promising approaches to explainability in deep learning.<n>We propose specialized theoretically-grounded LRP rules designed to propagate attributions across various positional encoding methods.<n>Our method significantly outperforms the state-of-the-art in both vision and NLP explainability tasks.
  arXiv  Detail & Related papers  (2025-06-02T18:07:55Z)
• Understanding and Improving Laplacian Positional Encodings For Temporal GNNs [24.209908552258632]
  We introduce a theoretical framework that connects supra-Laplacian encodings to per-time-slice encodings.<n>We also introduce novel methods to reduce the computational overhead, achieving up to 56x faster runtimes while scaling to graphs with 50,000 active nodes.<n>Our findings reveal that while positional encodings can significantly boost performance in certain scenarios, their effectiveness varies across different models.
  arXiv  Detail & Related papers  (2025-06-02T12:30:58Z)
• SOLO: A Single Transformer for Scalable Vision-Language Modeling [74.05173379908703]
  We present SOLO, a single transformer for visiOn-Language mOdeling.<n>A unified single Transformer architecture, like SOLO, effectively addresses these scalability concerns in LVLMs.<n>In this paper, we introduce the first open-source training recipe for developing SOLO, an open-source 7B LVLM.
  arXiv  Detail & Related papers  (2024-07-08T22:40:15Z)
• DAPE: Data-Adaptive Positional Encoding for Length Extrapolation [60.18239094672938]
  Positional encoding plays a crucial role in transformers, significantly impacting model performance and generalization length. We propose a Data-Adaptive Positional (DAPE) method, which enhances model performances in terms of trained length and length generalization. We successfully train the model on sequence length 128 and achieve better performance at evaluation sequence length 8192, compared with other static positional encoding methods.
  arXiv  Detail & Related papers  (2024-05-23T15:51:24Z)
• Positional Encoding Helps Recurrent Neural Networks Handle a Large Vocabulary [1.4594704809280983]
  Positional encoding is a high-dimensional representation of time indices on input data. RNNs can encode the temporal information of data points on their own, rendering their use of positional encoding seemingly redundant/unnecessary.
  arXiv  Detail & Related papers  (2024-01-31T23:32:20Z)
• Graph Transformers without Positional Encodings [0.7252027234425334]
  We introduce Eigenformer, a Graph Transformer employing a novel spectrum-aware attention mechanism cognizant of the Laplacian spectrum of the graph. We empirically show that it achieves performance competetive with SOTA Graph Transformers on a number of standard GNN benchmarks.
  arXiv  Detail & Related papers  (2024-01-31T12:33:31Z)
• The Locality and Symmetry of Positional Encodings [9.246374019271938]
  We conduct a systematic study of positional encodings in textbfBi Masked Language Models (BERT-style) We uncover the core function of PEs by identifying two common properties, Locality and Symmetry. We quantify the weakness of current PEs by introducing two new probing tasks, on which current PEs perform poorly.
  arXiv  Detail & Related papers  (2023-10-19T16:15:15Z)
• Improving Position Encoding of Transformers for Multivariate Time Series Classification [5.467400475482668]
  We propose a new absolute position encoding method dedicated to time series data called time Absolute Position. We then propose a novel time series classification (MTSC) model combining tAPE/eRPE and convolution-based input encoding named ConvTran to improve the position and data embedding of time series data.
  arXiv  Detail & Related papers  (2023-05-26T05:30:04Z)
• Trading Positional Complexity vs. Deepness in Coordinate Networks [33.90893096003318]
  We show that alternative non-Fourier embedding functions can indeed be used for positional encoding. Their performance is entirely determined by a trade-off between the stable rank of the embedded matrix and the distance preservation between embedded coordinates. We argue that employing a more complex positional encoding -- that scales exponentially with the number of modes -- requires only a linear (rather than deep) coordinate function to achieve comparable performance.
  arXiv  Detail & Related papers  (2022-05-18T15:17:09Z)
• ZippyPoint: Fast Interest Point Detection, Description, and Matching through Mixed Precision Discretization [71.91942002659795]
  We investigate and adapt network quantization techniques to accelerate inference and enable its use on compute limited platforms. ZippyPoint, our efficient quantized network with binary descriptors, improves the network runtime speed, the descriptor matching speed, and the 3D model size. These improvements come at a minor performance degradation as evaluated on the tasks of homography estimation, visual localization, and map-free visual relocalization.
  arXiv  Detail & Related papers  (2022-03-07T18:59:03Z)
• Learnable Fourier Features for Multi-DimensionalSpatial Positional Encoding [96.9752763607738]
  We propose a novel positional encoding method based on learnable Fourier features. Our experiments show that our learnable feature representation for multi-dimensional positional encoding outperforms existing methods.
  arXiv  Detail & Related papers  (2021-06-05T04:40:18Z)
• Auto-Encoding Twin-Bottleneck Hashing [141.5378966676885]
  This paper proposes an efficient and adaptive code-driven graph. It is updated by decoding in the context of an auto-encoder. Experiments on benchmarked datasets clearly show the superiority of our framework over the state-of-the-art hashing methods.
  arXiv  Detail & Related papers  (2020-02-27T05:58:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.