Spectral Transform Forms Scalable Transformer
- URL: http://arxiv.org/abs/2111.07602v1
- Date: Mon, 15 Nov 2021 08:46:01 GMT
- Title: Spectral Transform Forms Scalable Transformer
- Authors: Bingxin Zhou, Xinliang Liu, Yuehua Liu, Yunying Huang, Pietro Li\`o,
YuGuang Wang
- Abstract summary: This work learns from the philosophy of self-attention and proposes an efficient spectral-based neural unit that employs informative long-range temporal interaction.
The developed spectral window unit (SW) model predicts scalable dynamic graphs with assured efficiency.
- Score: 1.19071399645846
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Many real-world relational systems, such as social networks and biological
systems, contain dynamic interactions. When learning dynamic graph
representation, it is essential to employ sequential temporal information and
geometric structure. Mainstream work achieves topological embedding via message
passing networks (e.g., GCN, GAT). The temporal evolution, on the other hand,
is conventionally expressed via memory units (e.g., LSTM or GRU) that possess
convenient information filtration in a gate mechanism. Though, such a design
prevents large-scale input sequence due to the over-complicated encoding. This
work learns from the philosophy of self-attention and proposes an efficient
spectral-based neural unit that employs informative long-range temporal
interaction. The developed spectral window unit (SWINIT) model predicts
scalable dynamic graphs with assured efficiency. The architecture is assembled
with a few simple effective computational blocks that constitute randomized
SVD, MLP, and graph Framelet convolution. The SVD plus MLP module encodes the
long-short-term feature evolution of the dynamic graph events. A fast framelet
graph transform in the framelet convolution embeds the structural dynamics.
Both strategies enhance the model's ability on scalable analysis. In
particular, the iterative SVD approximation shrinks the computational
complexity of attention to O(Nd\log(d)) for the dynamic graph with N edges and
d edge features, and the multiscale transform of framelet convolution allows
sufficient scalability in the network training. Our SWINIT achieves
state-of-the-art performance on a variety of online continuous-time dynamic
graph learning tasks, while compared to baseline methods, the number of its
learnable parameters reduces by up to seven times.
Related papers
- LongVQ: Long Sequence Modeling with Vector Quantization on Structured Memory [63.41820940103348]
Self-attention mechanism's computational cost limits its practicality for long sequences.
We propose a new method called LongVQ to compress the global abstraction as a length-fixed codebook.
LongVQ effectively maintains dynamic global and local patterns, which helps to complement the lack of long-range dependency issues.
arXiv Detail & Related papers (2024-04-17T08:26:34Z) - Todyformer: Towards Holistic Dynamic Graph Transformers with
Structure-Aware Tokenization [6.799413002613627]
Todyformer is a novel Transformer-based neural network tailored for dynamic graphs.
It unifies the local encoding capacity of Message-Passing Neural Networks (MPNNs) with the global encoding of Transformers.
We show that Todyformer consistently outperforms the state-of-the-art methods for downstream tasks.
arXiv Detail & Related papers (2024-02-02T23:05:30Z) - TimeGraphs: Graph-based Temporal Reasoning [64.18083371645956]
TimeGraphs is a novel approach that characterizes dynamic interactions as a hierarchical temporal graph.
Our approach models the interactions using a compact graph-based representation, enabling adaptive reasoning across diverse time scales.
We evaluate TimeGraphs on multiple datasets with complex, dynamic agent interactions, including a football simulator, the Resistance game, and the MOMA human activity dataset.
arXiv Detail & Related papers (2024-01-06T06:26:49Z) - TSGCNeXt: Dynamic-Static Multi-Graph Convolution for Efficient
Skeleton-Based Action Recognition with Long-term Learning Potential [1.204694982718246]
We propose the Temporal-Spatio Graph ConvNeXt (TSGCNeXt) to explore efficient learning mechanism of long temporal skeleton sequences.
New graph learning mechanism with simple structure, Dynamic-Static Separate Multi-graph Convolution (DS-SMG) is proposed.
We construct a graph convolution training acceleration mechanism to optimize the back-propagation computing of dynamic graph learning with 55.08% speed-up.
arXiv Detail & Related papers (2023-04-23T12:10:36Z) - Piecewise-Velocity Model for Learning Continuous-time Dynamic Node
Representations [0.0]
Piecewise-Veable Model (PiVeM) for representation of continuous-time dynamic networks.
We show that PiVeM can successfully represent network structure and dynamics in ultra-low two-dimensional spaces.
It outperforms relevant state-of-art methods in downstream tasks such as link prediction.
arXiv Detail & Related papers (2022-12-23T13:57:56Z) - Dynamic Graph Message Passing Networks for Visual Recognition [112.49513303433606]
Modelling long-range dependencies is critical for scene understanding tasks in computer vision.
A fully-connected graph is beneficial for such modelling, but its computational overhead is prohibitive.
We propose a dynamic graph message passing network, that significantly reduces the computational complexity.
arXiv Detail & Related papers (2022-09-20T14:41:37Z) - Dynamic Spatial Sparsification for Efficient Vision Transformers and
Convolutional Neural Networks [88.77951448313486]
We present a new approach for model acceleration by exploiting spatial sparsity in visual data.
We propose a dynamic token sparsification framework to prune redundant tokens.
We extend our method to hierarchical models including CNNs and hierarchical vision Transformers.
arXiv Detail & Related papers (2022-07-04T17:00:51Z) - Efficient-Dyn: Dynamic Graph Representation Learning via Event-based
Temporal Sparse Attention Network [2.0047096160313456]
Dynamic graph neural networks have received more and more attention from researchers.
We propose a novel dynamic graph neural network, Efficient-Dyn.
It adaptively encodes temporal information into a sequence of patches with an equal amount of temporal-topological structure.
arXiv Detail & Related papers (2022-01-04T23:52:24Z) - Spatio-Temporal Inception Graph Convolutional Networks for
Skeleton-Based Action Recognition [126.51241919472356]
We design a simple and highly modularized graph convolutional network architecture for skeleton-based action recognition.
Our network is constructed by repeating a building block that aggregates multi-granularity information from both the spatial and temporal paths.
arXiv Detail & Related papers (2020-11-26T14:43:04Z) - Continuous-in-Depth Neural Networks [107.47887213490134]
We first show that ResNets fail to be meaningful dynamical in this richer sense.
We then demonstrate that neural network models can learn to represent continuous dynamical systems.
We introduce ContinuousNet as a continuous-in-depth generalization of ResNet architectures.
arXiv Detail & Related papers (2020-08-05T22:54:09Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.