Rethinking Graph Transformer Architecture Design for Node Classification

• URL: http://arxiv.org/abs/2410.11189v1
• Date: Tue, 15 Oct 2024 02:08:16 GMT
• Title: Rethinking Graph Transformer Architecture Design for Node Classification
• Authors: Jiajun Zhou, Xuanze Chen, Chenxuan Xie, Yu Shanqing, Qi Xuan, Xiaoniu Yang,
• Abstract summary: Graph Transformer (GT) is a special type of Graph Neural Networks (GNNs) that utilize multi-head attention to facilitate high-order message passing. In this work, we conduct observational experiments to explore the adaptability of the GT architecture in node classification tasks. Our proposed GT architecture can effectively adapt to node classification tasks without being affected by global noise and computational efficiency limitations.
• Score: 4.497245600377944
• License:
• Abstract: Graph Transformer (GT), as a special type of Graph Neural Networks (GNNs), utilizes multi-head attention to facilitate high-order message passing. However, this also imposes several limitations in node classification applications: 1) nodes are susceptible to global noise; 2) self-attention computation cannot scale well to large graphs. In this work, we conduct extensive observational experiments to explore the adaptability of the GT architecture in node classification tasks and draw several conclusions: the current multi-head self-attention module in GT can be completely replaceable, while the feed-forward neural network module proves to be valuable. Based on this, we decouple the propagation (P) and transformation (T) of GNNs and explore a powerful GT architecture, named GNNFormer, which is based on the P/T combination message passing and adapted for node classification in both homophilous and heterophilous scenarios. Extensive experiments on 12 benchmark datasets demonstrate that our proposed GT architecture can effectively adapt to node classification tasks without being affected by global noise and computational efficiency limitations.

Related papers

• Degree-based stratification of nodes in Graph Neural Networks [66.17149106033126]
  We modify the Graph Neural Network (GNN) architecture so that the weight matrices are learned, separately, for the nodes in each group. This simple-to-implement modification seems to improve performance across datasets and GNN methods.
  arXiv  Detail & Related papers  (2023-12-16T14:09:23Z)
• MuseGNN: Interpretable and Convergent Graph Neural Network Layers at Scale [15.93424606182961]
  We propose a sampling-based energy function and scalable GNN layers that iteratively reduce it, guided by convergence guarantees in certain settings. We also instantiate a full GNN architecture based on these designs, and the model achieves competitive accuracy and scalability when applied to the largest publicly-available node classification benchmark exceeding 1TB in size.
  arXiv  Detail & Related papers  (2023-10-19T04:30:14Z)
• NDGGNET-A Node Independent Gate based Graph Neural Networks [6.155450481110693]
  For nodes with sparse connectivity, it is difficult to obtain enough information through a single GNN layer. In this thesis, we define a novel framework that allows the normal GNN model to accommodate more layers. Experimental results show that our proposed model can effectively increase the model depth and perform well on several datasets.
  arXiv  Detail & Related papers  (2022-05-11T08:51:04Z)
• Exploiting Neighbor Effect: Conv-Agnostic GNNs Framework for Graphs with Heterophily [58.76759997223951]
  We propose a new metric based on von Neumann entropy to re-examine the heterophily problem of GNNs. We also propose a Conv-Agnostic GNN framework (CAGNNs) to enhance the performance of most GNNs on heterophily datasets.
  arXiv  Detail & Related papers  (2022-03-19T14:26:43Z)
• A Variational Edge Partition Model for Supervised Graph Representation Learning [51.30365677476971]
  This paper introduces a graph generative process to model how the observed edges are generated by aggregating the node interactions over a set of overlapping node communities. We partition each edge into the summation of multiple community-specific weighted edges and use them to define community-specific GNNs. A variational inference framework is proposed to jointly learn a GNN based inference network that partitions the edges into different communities, these community-specific GNNs, and a GNN based predictor that combines community-specific GNNs for the end classification task.
  arXiv  Detail & Related papers  (2022-02-07T14:37:50Z)
• Network In Graph Neural Network [9.951298152023691]
  We present a model-agnostic methodology that allows arbitrary GNN models to increase their model capacity by making the model deeper. Instead of adding or widening GNN layers, NGNN deepens a GNN model by inserting non-linear feedforward neural network layer(s) within each GNN layer.
  arXiv  Detail & Related papers  (2021-11-23T03:58:56Z)
• VQ-GNN: A Universal Framework to Scale up Graph Neural Networks using Vector Quantization [70.8567058758375]
  VQ-GNN is a universal framework to scale up any convolution-based GNNs using Vector Quantization (VQ) without compromising the performance. Our framework avoids the "neighbor explosion" problem of GNNs using quantized representations combined with a low-rank version of the graph convolution matrix.
  arXiv  Detail & Related papers  (2021-10-27T11:48:50Z)
• Dual GNNs: Graph Neural Network Learning with Limited Supervision [33.770877823910176]
  We propose a novel Dual GNN learning framework to address this challenge task. By integrating the two modules in a dual GNN learning framework, we perform joint learning in an end-to-end fashion.
  arXiv  Detail & Related papers  (2021-06-29T23:52:25Z)
• A Unified View on Graph Neural Networks as Graph Signal Denoising [49.980783124401555]
  Graph Neural Networks (GNNs) have risen to prominence in learning representations for graph structured data. In this work, we establish mathematically that the aggregation processes in a group of representative GNN models can be regarded as solving a graph denoising problem. We instantiate a novel GNN model, ADA-UGNN, derived from UGNN, to handle graphs with adaptive smoothness across nodes.
  arXiv  Detail & Related papers  (2020-10-05T04:57:18Z)
• Hierarchical Message-Passing Graph Neural Networks [12.207978823927386]
  We propose a novel Hierarchical Message-passing Graph Neural Networks framework. Key idea is generating a hierarchical structure that re-organises all nodes in a flat graph into multi-level super graphs. We present the first model to implement this framework, termed Hierarchical Community-aware Graph Neural Network (HC-GNN)
  arXiv  Detail & Related papers  (2020-09-08T13:11:07Z)
• Binarized Graph Neural Network [65.20589262811677]
  We develop a binarized graph neural network to learn the binary representations of the nodes with binary network parameters. Our proposed method can be seamlessly integrated into the existing GNN-based embedding approaches. Experiments indicate that the proposed binarized graph neural network, namely BGN, is orders of magnitude more efficient in terms of both time and space.
  arXiv  Detail & Related papers  (2020-04-19T09:43:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.