Todyformer: Towards Holistic Dynamic Graph Transformers with
Structure-Aware Tokenization
- URL: http://arxiv.org/abs/2402.05944v1
- Date: Fri, 2 Feb 2024 23:05:30 GMT
- Title: Todyformer: Towards Holistic Dynamic Graph Transformers with
Structure-Aware Tokenization
- Authors: Mahdi Biparva, Raika Karimi, Faezeh Faez, Yingxue Zhang
- Abstract summary: Todyformer is a novel Transformer-based neural network tailored for dynamic graphs.
It unifies the local encoding capacity of Message-Passing Neural Networks (MPNNs) with the global encoding of Transformers.
We show that Todyformer consistently outperforms the state-of-the-art methods for downstream tasks.
- Score: 6.799413002613627
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Temporal Graph Neural Networks have garnered substantial attention for their
capacity to model evolving structural and temporal patterns while exhibiting
impressive performance. However, it is known that these architectures are
encumbered by issues that constrain their performance, such as over-squashing
and over-smoothing. Meanwhile, Transformers have demonstrated exceptional
computational capacity to effectively address challenges related to long-range
dependencies. Consequently, we introduce Todyformer-a novel Transformer-based
neural network tailored for dynamic graphs. It unifies the local encoding
capacity of Message-Passing Neural Networks (MPNNs) with the global encoding of
Transformers through i) a novel patchifying paradigm for dynamic graphs to
improve over-squashing, ii) a structure-aware parametric tokenization strategy
leveraging MPNNs, iii) a Transformer with temporal positional-encoding to
capture long-range dependencies, and iv) an encoding architecture that
alternates between local and global contextualization, mitigating
over-smoothing in MPNNs. Experimental evaluations on public benchmark datasets
demonstrate that Todyformer consistently outperforms the state-of-the-art
methods for downstream tasks. Furthermore, we illustrate the underlying aspects
of the proposed model in effectively capturing extensive temporal dependencies
in dynamic graphs.
Related papers
- TCCT-Net: Two-Stream Network Architecture for Fast and Efficient Engagement Estimation via Behavioral Feature Signals [58.865901821451295]
We present a novel two-stream feature fusion "Tensor-Convolution and Convolution-Transformer Network" (TCCT-Net) architecture.
To better learn the meaningful patterns in the temporal-spatial domain, we design a "CT" stream that integrates a hybrid convolutional-transformer.
In parallel, to efficiently extract rich patterns from the temporal-frequency domain, we introduce a "TC" stream that uses Continuous Wavelet Transform (CWT) to represent information in a 2D tensor form.
arXiv Detail & Related papers (2024-04-15T06:01:48Z) - Equivariant Matrix Function Neural Networks [1.8717045355288808]
We introduce Matrix Function Neural Networks (MFNs), a novel architecture that parameterizes non-local interactions through analytic matrix equivariant functions.
MFNs is able to capture intricate non-local interactions in quantum systems, paving the way to new state-of-the-art force fields.
arXiv Detail & Related papers (2023-10-16T14:17:00Z) - Distance Weighted Trans Network for Image Completion [52.318730994423106]
We propose a new architecture that relies on Distance-based Weighted Transformer (DWT) to better understand the relationships between an image's components.
CNNs are used to augment the local texture information of coarse priors.
DWT blocks are used to recover certain coarse textures and coherent visual structures.
arXiv Detail & Related papers (2023-10-11T12:46:11Z) - Adaptive Growth: Real-time CNN Layer Expansion [0.0]
This research presents a new algorithm that allows the convolutional layer of a Convolutional Neural Network (CNN) to dynamically evolve based on data input.
Instead of a rigid architecture, our approach iteratively introduces kernels to the convolutional layer, gauging its real-time response to varying data.
Remarkably, our unsupervised method has outstripped its supervised counterparts across diverse datasets.
arXiv Detail & Related papers (2023-09-06T14:43:58Z) - Structure-reinforced Transformer for Dynamic Graph Representation Learning with Edge Temporal States [8.577434144370004]
We introduce a novel dynamic graph representation learning framework namely Recurrent Structure-reinforced Graph Transformer (RSGT)
RSGT initially models the temporal status of edges explicitly by utilizing different edge types and weights based on the differences between any two consecutive snapshots.
A structure-reinforced graph transformer is proposed to capture temporal node representations that encoding both the graph topological structure and evolving dynamics.
arXiv Detail & Related papers (2023-04-20T04:12:50Z) - Towards Long-Term Time-Series Forecasting: Feature, Pattern, and
Distribution [57.71199089609161]
Long-term time-series forecasting (LTTF) has become a pressing demand in many applications, such as wind power supply planning.
Transformer models have been adopted to deliver high prediction capacity because of the high computational self-attention mechanism.
We propose an efficient Transformerbased model, named Conformer, which differentiates itself from existing methods for LTTF in three aspects.
arXiv Detail & Related papers (2023-01-05T13:59:29Z) - Dynamic Graph Message Passing Networks for Visual Recognition [112.49513303433606]
Modelling long-range dependencies is critical for scene understanding tasks in computer vision.
A fully-connected graph is beneficial for such modelling, but its computational overhead is prohibitive.
We propose a dynamic graph message passing network, that significantly reduces the computational complexity.
arXiv Detail & Related papers (2022-09-20T14:41:37Z) - CSformer: Bridging Convolution and Transformer for Compressive Sensing [65.22377493627687]
This paper proposes a hybrid framework that integrates the advantages of leveraging detailed spatial information from CNN and the global context provided by transformer for enhanced representation learning.
The proposed approach is an end-to-end compressive image sensing method, composed of adaptive sampling and recovery.
The experimental results demonstrate the effectiveness of the dedicated transformer-based architecture for compressive sensing.
arXiv Detail & Related papers (2021-12-31T04:37:11Z) - Spectral Transform Forms Scalable Transformer [1.19071399645846]
This work learns from the philosophy of self-attention and proposes an efficient spectral-based neural unit that employs informative long-range temporal interaction.
The developed spectral window unit (SW) model predicts scalable dynamic graphs with assured efficiency.
arXiv Detail & Related papers (2021-11-15T08:46:01Z) - Anomaly Detection in Dynamic Graphs via Transformer [30.926884264054042]
We present a novel Transformer-based Anomaly Detection framework for DYnamic graph (TADDY)
Our framework constructs a comprehensive node encoding strategy to better represent each node's structural and temporal roles in an evolving graphs stream.
Our proposed TADDY framework outperforms the state-of-the-art methods by a large margin on four real-world datasets.
arXiv Detail & Related papers (2021-06-18T02:27:19Z) - Transformers Solve the Limited Receptive Field for Monocular Depth
Prediction [82.90445525977904]
We propose TransDepth, an architecture which benefits from both convolutional neural networks and transformers.
This is the first paper which applies transformers into pixel-wise prediction problems involving continuous labels.
arXiv Detail & Related papers (2021-03-22T18:00:13Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.