Graph Attention Multi-Layer Perceptron

• URL: http://arxiv.org/abs/2108.10097v1
• Date: Mon, 23 Aug 2021 11:56:20 GMT
• Title: Graph Attention Multi-Layer Perceptron
• Authors: Wentao Zhang, Ziqi Yin, Zeang Sheng, Wen Ouyang, Xiaosen Li, Yangyu Tao, Zhi Yang, Bin Cui
• Abstract summary: Graph neural networks (GNNs) have recently achieved state-of-the-art performance in many graph-based applications. We introduce a scalable and flexible Graph Attention Multilayer Perceptron (GAMLP) With three principled receptive field attention, each node in GAMLP is flexible and adaptive in leveraging the propagated features over the different sizes of reception field.
• Score: 12.129233487384965
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Graph neural networks (GNNs) have recently achieved state-of-the-art performance in many graph-based applications. Despite the high expressive power, they typically need to perform an expensive recursive neighborhood expansion in multiple training epochs and face a scalability issue. Moreover, most of them are inflexible since they are restricted to fixed-hop neighborhoods and insensitive to actual receptive field demands for different nodes. We circumvent these limitations by introducing a scalable and flexible Graph Attention Multilayer Perceptron (GAMLP). With the separation of the non-linear transformation and feature propagation, GAMLP significantly improves the scalability and efficiency by performing the propagation procedure in a pre-compute manner. With three principled receptive field attention, each node in GAMLP is flexible and adaptive in leveraging the propagated features over the different sizes of reception field. We conduct extensive evaluations on the three large open graph benchmarks (e.g., ogbn-papers100M, ogbn-products and ogbn-mag), demonstrating that GAMLP not only achieves the state-of-art performance, but also additionally provide high scalability and efficiency.

Related papers

• SGFormer: Single-Layer Graph Transformers with Approximation-Free Linear Complexity [74.51827323742506]
  We evaluate the necessity of adopting multi-layer attentions in Transformers on graphs. We show that one-layer propagation can be reduced to one-layer propagation, with the same capability for representation learning. It suggests a new technical path for building powerful and efficient Transformers on graphs.
  arXiv  Detail & Related papers  (2024-09-13T17:37:34Z)
• A Pure Transformer Pretraining Framework on Text-attributed Graphs [50.833130854272774]
  We introduce a feature-centric pretraining perspective by treating graph structure as a prior. Our framework, Graph Sequence Pretraining with Transformer (GSPT), samples node contexts through random walks. GSPT can be easily adapted to both node classification and link prediction, demonstrating promising empirical success on various datasets.
  arXiv  Detail & Related papers  (2024-06-19T22:30:08Z)
• A Scalable and Effective Alternative to Graph Transformers [19.018320937729264]
  Graph Transformers (GTs) were introduced, utilizing self-attention mechanism to model pairwise node relationships. GTs suffer from complexity w.r.t. the number of nodes in the graph, hindering their applicability to large graphs. We present Graph-Enhanced Contextual Operator (GECO), a scalable and effective alternative to GTs.
  arXiv  Detail & Related papers  (2024-06-17T19:57:34Z)
• GRANOLA: Adaptive Normalization for Graph Neural Networks [28.993479890213617]
  We propose a graph-adaptive normalization layer, GRANOLA, for Graph Neural Network (GNN) layers. Unlike existing normalization layers, GRANOLA normalizes node features by adapting to the specific characteristics of the graph. Our empirical evaluation of various graph benchmarks underscores the superior performance of GRANOLA over existing normalization techniques.
  arXiv  Detail & Related papers  (2024-04-20T10:44:13Z)
• Rethinking Node-wise Propagation for Large-scale Graph Learning [42.29535580297932]
  textbfAdaptive textbfTopology-aware textbfPropagation (ATP) ATP reduces potential high-bias propagation and extracts structural patterns of each node in a scalable manner to improve running efficiency and predictive performance. ATP has proven to be efficient in improving the performance of prevalent scalable GNNs for semi-supervised node classification.
  arXiv  Detail & Related papers  (2024-02-09T01:19:47Z)
• Graph Transformers for Large Graphs [57.19338459218758]
  This work advances representation learning on single large-scale graphs with a focus on identifying model characteristics and critical design constraints. A key innovation of this work lies in the creation of a fast neighborhood sampling technique coupled with a local attention mechanism. We report a 3x speedup and 16.8% performance gain on ogbn-products and snap-patents, while we also scale LargeGT on ogbn-100M with a 5.9% performance improvement.
  arXiv  Detail & Related papers  (2023-12-18T11:19:23Z)
• SGFormer: Simplifying and Empowering Transformers for Large-Graph Representations [75.71298846760303]
  We show that a one-layer attention can bring up surprisingly competitive performance across node property prediction benchmarks. We frame the proposed scheme as Simplified Graph Transformers (SGFormer), which is empowered by a simple attention model. We believe the proposed methodology alone enlightens a new technical path of independent interest for building Transformers on large graphs.
  arXiv  Detail & Related papers  (2023-06-19T08:03:25Z)
• AGNN: Alternating Graph-Regularized Neural Networks to Alleviate Over-Smoothing [29.618952407794776]
  We propose an Alternating Graph-regularized Neural Network (AGNN) composed of Graph Convolutional Layer (GCL) and Graph Embedding Layer (GEL) GEL is derived from the graph-regularized optimization containing Laplacian embedding term, which can alleviate the over-smoothing problem. AGNN is evaluated via a large number of experiments including performance comparison with some multi-layer or multi-order graph neural networks.
  arXiv  Detail & Related papers  (2023-04-14T09:20:03Z)
• GNNAutoScale: Scalable and Expressive Graph Neural Networks via Historical Embeddings [51.82434518719011]
  GNNAutoScale (GAS) is a framework for scaling arbitrary message-passing GNNs to large graphs. Gas prunes entire sub-trees of the computation graph by utilizing historical embeddings from prior training iterations. Gas reaches state-of-the-art performance on large-scale graphs.
  arXiv  Detail & Related papers  (2021-06-10T09:26:56Z)
• GMLP: Building Scalable and Flexible Graph Neural Networks with Feature-Message Passing [16.683813354137254]
  Graph Multi-layer Perceptron (GMLP) separates the neural update from the message passing. We conduct extensive evaluations on 11 benchmark datasets.
  arXiv  Detail & Related papers  (2021-04-20T10:19:21Z)
• Robust Optimization as Data Augmentation for Large-scale Graphs [117.2376815614148]
  We propose FLAG (Free Large-scale Adversarial Augmentation on Graphs), which iteratively augments node features with gradient-based adversarial perturbations during training. FLAG is a general-purpose approach for graph data, which universally works in node classification, link prediction, and graph classification tasks.
  arXiv  Detail & Related papers  (2020-10-19T21:51:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.