Supra-Laplacian Encoding for Transformer on Dynamic Graphs

• URL: http://arxiv.org/abs/2409.17986v2
• Date: Fri, 15 Nov 2024 14:22:00 GMT
• Title: Supra-Laplacian Encoding for Transformer on Dynamic Graphs
• Authors: Yannis Karmim, Marc Lafon, Raphael Fournier S'niehotta, Nicolas Thome,
• Abstract summary: We present a new-temporal encoding for GT architecture while keeping temporal information. Specifically, we transform Time Dynamic Graphplas into multi-layer graphs and take advantage of the spectral properties of their associated supra-lacian matrix. Our second contribution explicitly model nodes pairwise with a cross-attention mechanism, providing an accurate edge representation for dynamic link prediction.
• Score: 14.293220696079919
• License:
• Abstract: Fully connected Graph Transformers (GT) have rapidly become prominent in the static graph community as an alternative to Message-Passing models, which suffer from a lack of expressivity, oversquashing, and under-reaching. However, in a dynamic context, by interconnecting all nodes at multiple snapshots with self-attention, GT loose both structural and temporal information. In this work, we introduce Supra-LAplacian encoding for spatio-temporal TransformErs (SLATE), a new spatio-temporal encoding to leverage the GT architecture while keeping spatio-temporal information. Specifically, we transform Discrete Time Dynamic Graphs into multi-layer graphs and take advantage of the spectral properties of their associated supra-Laplacian matrix. Our second contribution explicitly model nodes' pairwise relationships with a cross-attention mechanism, providing an accurate edge representation for dynamic link prediction. SLATE outperforms numerous state-of-the-art methods based on Message-Passing Graph Neural Networks combined with recurrent models (e.g LSTM), and Dynamic Graph Transformers, on 9 datasets. Code is available at: github.com/ykrmm/SLATE.

Related papers

• A Pure Transformer Pretraining Framework on Text-attributed Graphs [50.833130854272774]
  We introduce a feature-centric pretraining perspective by treating graph structure as a prior. Our framework, Graph Sequence Pretraining with Transformer (GSPT), samples node contexts through random walks. GSPT can be easily adapted to both node classification and link prediction, demonstrating promising empirical success on various datasets.
  arXiv  Detail & Related papers  (2024-06-19T22:30:08Z)
• TimeGraphs: Graph-based Temporal Reasoning [64.18083371645956]
  TimeGraphs is a novel approach that characterizes dynamic interactions as a hierarchical temporal graph. Our approach models the interactions using a compact graph-based representation, enabling adaptive reasoning across diverse time scales. We evaluate TimeGraphs on multiple datasets with complex, dynamic agent interactions, including a football simulator, the Resistance game, and the MOMA human activity dataset.
  arXiv  Detail & Related papers  (2024-01-06T06:26:49Z)
• Mending of Spatio-Temporal Dependencies in Block Adjacency Matrix [3.529869282529924]
  We propose a novel end-to-end learning architecture designed to mend the temporal dependencies, resulting in a well-connected graph. Our methodology demonstrates superior performance on benchmark datasets, such as SurgVisDom and C2D2.
  arXiv  Detail & Related papers  (2023-10-04T06:42:33Z)
• Temporal Aggregation and Propagation Graph Neural Networks for Dynamic Representation [67.26422477327179]
  Temporal graphs exhibit dynamic interactions between nodes over continuous time. We propose a novel method of temporal graph convolution with the whole neighborhood. Our proposed TAP-GNN outperforms existing temporal graph methods by a large margin in terms of both predictive performance and online inference latency.
  arXiv  Detail & Related papers  (2023-04-15T08:17:18Z)
• Self-Supervised Temporal Graph learning with Temporal and Structural Intensity Alignment [53.72873672076391]
  Temporal graph learning aims to generate high-quality representations for graph-based tasks with dynamic information. We propose a self-supervised method called S2T for temporal graph learning, which extracts both temporal and structural information. S2T achieves at most 10.13% performance improvement compared with the state-of-the-art competitors on several datasets.
  arXiv  Detail & Related papers  (2023-02-15T06:36:04Z)
• Dynamic Graph Message Passing Networks for Visual Recognition [112.49513303433606]
  Modelling long-range dependencies is critical for scene understanding tasks in computer vision. A fully-connected graph is beneficial for such modelling, but its computational overhead is prohibitive. We propose a dynamic graph message passing network, that significantly reduces the computational complexity.
  arXiv  Detail & Related papers  (2022-09-20T14:41:37Z)
• Time-aware Dynamic Graph Embedding for Asynchronous Structural Evolution [60.695162101159134]
  Existing works merely view a dynamic graph as a sequence of changes. We formulate dynamic graphs as temporal edge sequences associated with joining time of. vertex and timespan of edges. A time-aware Transformer is proposed to embed. vertex' dynamic connections and ToEs into the learned. vertex representations.
  arXiv  Detail & Related papers  (2022-07-01T15:32:56Z)
• Instant Graph Neural Networks for Dynamic Graphs [18.916632816065935]
  We propose Instant Graph Neural Network (InstantGNN), an incremental approach for the graph representation matrix of dynamic graphs. Our method avoids time-consuming, repetitive computations and allows instant updates on the representation and instant predictions. Our model achieves state-of-the-art accuracy while having orders-of-magnitude higher efficiency than existing methods.
  arXiv  Detail & Related papers  (2022-06-03T03:27:42Z)
• Dynamic Graph Learning-Neural Network for Multivariate Time Series Modeling [2.3022070933226217]
  We propose a novel framework, namely static- and dynamic-graph learning-neural network (GL) The model acquires static and dynamic graph matrices from data to model long-term and short-term patterns respectively. It achieves state-of-the-art performance on almost all datasets.
  arXiv  Detail & Related papers  (2021-12-06T08:19:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.