What Can We Learn from State Space Models for Machine Learning on Graphs?

• URL: http://arxiv.org/abs/2406.05815v1
• Date: Sun, 9 Jun 2024 15:03:36 GMT
• Title: What Can We Learn from State Space Models for Machine Learning on Graphs?
• Authors: Yinan Huang, Siqi Miao, Pan Li,
• Abstract summary: We propose Graph State Space Convolution (GSSC) as a principled extension of State Space Models (SSMs) to graph-structured data. By leveraging global permutation-equivariant set aggregation and factorizable graph kernels, GSSC preserves all three advantages of SSMs. Our findings highlight the potential of GSSC as a powerful and scalable model for graph machine learning.
• Score: 11.38076877943004
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine learning on graphs has recently found extensive applications across domains. However, the commonly used Message Passing Neural Networks (MPNNs) suffer from limited expressive power and struggle to capture long-range dependencies. Graph transformers offer a strong alternative due to their global attention mechanism, but they come with great computational overheads, especially for large graphs. In recent years, State Space Models (SSMs) have emerged as a compelling approach to replace full attention in transformers to model sequential data. It blends the strengths of RNNs and CNNs, offering a) efficient computation, b) the ability to capture long-range dependencies, and c) good generalization across sequences of various lengths. However, extending SSMs to graph-structured data presents unique challenges due to the lack of canonical node ordering in graphs. In this work, we propose Graph State Space Convolution (GSSC) as a principled extension of SSMs to graph-structured data. By leveraging global permutation-equivariant set aggregation and factorizable graph kernels that rely on relative node distances as the convolution kernels, GSSC preserves all three advantages of SSMs. We demonstrate the provably stronger expressiveness of GSSC than MPNNs in counting graph substructures and show its effectiveness across 10 real-world, widely used benchmark datasets, where GSSC achieves best results on 7 out of 10 datasets with all significant improvements compared to the state-of-the-art baselines and second-best results on the other 3 datasets. Our findings highlight the potential of GSSC as a powerful and scalable model for graph machine learning. Our code is available at https://github.com/Graph-COM/GSSC.

Related papers

• A Scalable and Effective Alternative to Graph Transformers [19.018320937729264]
  Graph Transformers (GTs) were introduced, utilizing self-attention mechanism to model pairwise node relationships. GTs suffer from complexity w.r.t. the number of nodes in the graph, hindering their applicability to large graphs. We present Graph-Enhanced Contextual Operator (GECO), a scalable and effective alternative to GTs.
  arXiv  Detail & Related papers  (2024-06-17T19:57:34Z)
• Spectral Greedy Coresets for Graph Neural Networks [61.24300262316091]
  The ubiquity of large-scale graphs in node-classification tasks hinders the real-world applications of Graph Neural Networks (GNNs) This paper studies graph coresets for GNNs and avoids the interdependence issue by selecting ego-graphs based on their spectral embeddings. Our spectral greedy graph coreset (SGGC) scales to graphs with millions of nodes, obviates the need for model pre-training, and applies to low-homophily graphs.
  arXiv  Detail & Related papers  (2024-05-27T17:52:12Z)
• Graph Mamba: Towards Learning on Graphs with State Space Models [2.039632659682125]
  We present a new class of Graph Neural Networks (GNNs) based on selective State Space Models (SSMs) GMNs attain an outstanding performance in long-range, small-scale, large-scale, and benchmark benchmark experiments.
  arXiv  Detail & Related papers  (2024-02-13T18:58:17Z)
• Graph-Mamba: Towards Long-Range Graph Sequence Modeling with Selective State Spaces [4.928791850200171]
  We introduce Graph-Mamba, the first attempt to enhance long-range context modeling in graph networks. We formulate graph-centric node prioritization and permutation strategies to enhance context-aware reasoning. Experiments on ten benchmark datasets demonstrate that Graph-Mamba outperforms state-of-the-art methods in long-range graph prediction tasks.
  arXiv  Detail & Related papers  (2024-02-01T17:21:53Z)
• GLISP: A Scalable GNN Learning System by Exploiting Inherent Structural Properties of Graphs [5.410321469222541]
  We propose GLISP, a sampling based GNN learning system for industrial scale graphs. GLISP consists of three core components: graph partitioner, graph sampling service and graph inference engine. Experiments show that GLISP achieves up to $6.53times$ and $70.77times$ speedups over existing GNN systems for training and inference tasks.
  arXiv  Detail & Related papers  (2024-01-06T02:59:24Z)
• A Comprehensive Study on Large-Scale Graph Training: Benchmarking and Rethinking [124.21408098724551]
  Large-scale graph training is a notoriously challenging problem for graph neural networks (GNNs) We present a new ensembling training manner, named EnGCN, to address the existing issues. Our proposed method has achieved new state-of-the-art (SOTA) performance on large-scale datasets.
  arXiv  Detail & Related papers  (2022-10-14T03:43:05Z)
• Representing Long-Range Context for Graph Neural Networks with Global Attention [37.212747564546156]
  We propose the use of Transformer-based self-attention to learn long-range pairwise relationships. Our method, which we call GraphTrans, applies a permutation-invariant Transformer module after a standard GNN module. Our results suggest that purely-learning-based approaches without graph structure may be suitable for learning high-level, long-range relationships on graphs.
  arXiv  Detail & Related papers  (2022-01-21T18:16:21Z)
• Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction [123.20238648121445]
  We propose a new self-supervised learning framework, Graph Information Aided Node feature exTraction (GIANT) GIANT makes use of the eXtreme Multi-label Classification (XMC) formalism, which is crucial for fine-tuning the language model based on graph information. We demonstrate the superior performance of GIANT over the standard GNN pipeline on Open Graph Benchmark datasets.
  arXiv  Detail & Related papers  (2021-10-29T19:55:12Z)
• GNNAutoScale: Scalable and Expressive Graph Neural Networks via Historical Embeddings [51.82434518719011]
  GNNAutoScale (GAS) is a framework for scaling arbitrary message-passing GNNs to large graphs. Gas prunes entire sub-trees of the computation graph by utilizing historical embeddings from prior training iterations. Gas reaches state-of-the-art performance on large-scale graphs.
  arXiv  Detail & Related papers  (2021-06-10T09:26:56Z)
• Graph Contrastive Learning with Augmentations [109.23158429991298]
  We propose a graph contrastive learning (GraphCL) framework for learning unsupervised representations of graph data. We show that our framework can produce graph representations of similar or better generalizability, transferrability, and robustness compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-10-22T20:13:43Z)
• Robust Optimization as Data Augmentation for Large-scale Graphs [117.2376815614148]
  We propose FLAG (Free Large-scale Adversarial Augmentation on Graphs), which iteratively augments node features with gradient-based adversarial perturbations during training. FLAG is a general-purpose approach for graph data, which universally works in node classification, link prediction, and graph classification tasks.
  arXiv  Detail & Related papers  (2020-10-19T21:51:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.