NTFormer: A Composite Node Tokenized Graph Transformer for Node Classification

• URL: http://arxiv.org/abs/2406.19249v1
• Date: Thu, 27 Jun 2024 15:16:00 GMT
• Title: NTFormer: A Composite Node Tokenized Graph Transformer for Node Classification
• Authors: Jinsong Chen, Siyu Jiang, Kun He,
• Abstract summary: We propose a new graph Transformer called NTFormer to address node classification issues. New token generator called Node2Par generates various token sequences using different token elements for each node. Experiments conducted on various benchmark datasets demonstrate the superiority of NTFormer over representative graph Transformers and graph neural networks for node classification.
• Score: 11.451341325579188
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, the emerging graph Transformers have made significant advancements for node classification on graphs. In most graph Transformers, a crucial step involves transforming the input graph into token sequences as the model input, enabling Transformer to effectively learn the node representations. However, we observe that existing methods only express partial graph information of nodes through single-type token generation. Consequently, they require tailored strategies to encode additional graph-specific features into the Transformer to ensure the quality of node representation learning, limiting the model flexibility to handle diverse graphs. To this end, we propose a new graph Transformer called NTFormer to address this issue. NTFormer introduces a novel token generator called Node2Par, which constructs various token sequences using different token elements for each node. This flexibility allows Node2Par to generate valuable token sequences from different perspectives, ensuring comprehensive expression of rich graph features. Benefiting from the merits of Node2Par, NTFormer only leverages a Transformer-based backbone without graph-specific modifications to learn node representations, eliminating the need for graph-specific modifications. Extensive experiments conducted on various benchmark datasets containing homophily and heterophily graphs with different scales demonstrate the superiority of NTFormer over representative graph Transformers and graph neural networks for node classification.

Related papers

• Leveraging Contrastive Learning for Enhanced Node Representations in Tokenized Graph Transformers [14.123432611346674]
  We propose a novel graph Transformer called GCFormer to harness graph information for learning optimal node representations. GCFormer develops a hybrid token generator to create two types of token sequences, positive and negative, to capture diverse graph information. A tailored Transformer-based backbone is adopted to learn meaningful node representations from these generated token sequences.
  arXiv  Detail & Related papers  (2024-06-27T15:29:47Z)
• Degree-based stratification of nodes in Graph Neural Networks [66.17149106033126]
  We modify the Graph Neural Network (GNN) architecture so that the weight matrices are learned, separately, for the nodes in each group. This simple-to-implement modification seems to improve performance across datasets and GNN methods.
  arXiv  Detail & Related papers  (2023-12-16T14:09:23Z)
• Deep Prompt Tuning for Graph Transformers [55.2480439325792]
  Fine-tuning is resource-intensive and requires storing multiple copies of large models. We propose a novel approach called deep graph prompt tuning as an alternative to fine-tuning. By freezing the pre-trained parameters and only updating the added tokens, our approach reduces the number of free parameters and eliminates the need for multiple model copies.
  arXiv  Detail & Related papers  (2023-09-18T20:12:17Z)
• Discrete Graph Auto-Encoder [52.50288418639075]
  We introduce a new framework named Discrete Graph Auto-Encoder (DGAE) We first use a permutation-equivariant auto-encoder to convert graphs into sets of discrete latent node representations. In the second step, we sort the sets of discrete latent representations and learn their distribution with a specifically designed auto-regressive model.
  arXiv  Detail & Related papers  (2023-06-13T12:40:39Z)
• AGFormer: Efficient Graph Representation with Anchor-Graph Transformer [95.1825252182316]
  We propose a novel graph Transformer architecture, termed Anchor Graph Transformer (AGFormer) AGFormer first obtains some representative anchors and then converts node-to-node message passing into anchor-to-anchor and anchor-to-node message passing process. Extensive experiments on several benchmark datasets demonstrate the effectiveness and benefits of proposed AGFormer.
  arXiv  Detail & Related papers  (2023-05-12T14:35:42Z)
• Pure Transformers are Powerful Graph Learners [51.36884247453605]
  We show that standard Transformers without graph-specific modifications can lead to promising results in graph learning both in theory and practice. We prove that this approach is theoretically at least as expressive as an invariant graph network (2-IGN) composed of equivariant linear layers. Our method coined Tokenized Graph Transformer (TokenGT) achieves significantly better results compared to GNN baselines and competitive results.
  arXiv  Detail & Related papers  (2022-07-06T08:13:06Z)
• Deformable Graph Transformer [31.254872949603982]
  We propose Deformable Graph Transformer (DGT) that performs sparse attention with dynamically sampled key and value pairs. Experiments demonstrate that our novel graph Transformer consistently outperforms existing Transformer-based models.
  arXiv  Detail & Related papers  (2022-06-29T00:23:25Z)
• NAGphormer: Neighborhood Aggregation Graph Transformer for Node Classification in Large Graphs [10.149586598073421]
  We propose a Neighborhood Aggregation Graph Transformer (NAGphormer) that is scalable to large graphs with millions of nodes. NAGphormer constructs tokens for each node by a neighborhood aggregation module called Hop2Token. We conduct extensive experiments on various popular benchmarks, including six small datasets and three large datasets.
  arXiv  Detail & Related papers  (2022-06-10T07:23:51Z)
• Modeling Graph Structure via Relative Position for Text Generation from Knowledge Graphs [54.176285420428776]
  We present Graformer, a novel Transformer-based encoder-decoder architecture for graph-to-text generation. With our novel graph self-attention, the encoding of a node relies on all nodes in the input graph - not only direct neighbors - facilitating the detection of global patterns. Graformer learns to weight these node-node relations differently for different attention heads, thus virtually learning differently connected views of the input graph.
  arXiv  Detail & Related papers  (2020-06-16T15:20:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.