Rethinking Tokenized Graph Transformers for Node Classification

• URL: http://arxiv.org/abs/2502.08101v1
• Date: Wed, 12 Feb 2025 03:56:35 GMT
• Title: Rethinking Tokenized Graph Transformers for Node Classification
• Authors: Jinsong Chen, Chenyang Li, GaiChao Li, John E. Hopcroft, Kun He,
• Abstract summary: node tokenized graph Transformers (GTs) have shown promising performance in node classification. In this paper, we propose a new method termed SwapGT to generate diverse token sequences.
• Score: 16.5581306015661
• License:
• Abstract: Node tokenized graph Transformers (GTs) have shown promising performance in node classification. The generation of token sequences is the key module in existing tokenized GTs which transforms the input graph into token sequences, facilitating the node representation learning via Transformer. In this paper, we observe that the generations of token sequences in existing GTs only focus on the first-order neighbors on the constructed similarity graphs, which leads to the limited usage of nodes to generate diverse token sequences, further restricting the potential of tokenized GTs for node classification. To this end, we propose a new method termed SwapGT. SwapGT first introduces a novel token swapping operation based on the characteristics of token sequences that fully leverages the semantic relevance of nodes to generate more informative token sequences. Then, SwapGT leverages a Transformer-based backbone to learn node representations from the generated token sequences. Moreover, SwapGT develops a center alignment loss to constrain the representation learning from multiple token sequences, further enhancing the model performance. Extensive empirical results on various datasets showcase the superiority of SwapGT for node classification.

Related papers

• Leveraging Contrastive Learning for Enhanced Node Representations in Tokenized Graph Transformers [14.123432611346674]
  We propose a novel graph Transformer called GCFormer to harness graph information for learning optimal node representations. GCFormer develops a hybrid token generator to create two types of token sequences, positive and negative, to capture diverse graph information. A tailored Transformer-based backbone is adopted to learn meaningful node representations from these generated token sequences.
  arXiv  Detail & Related papers  (2024-06-27T15:29:47Z)
• NTFormer: A Composite Node Tokenized Graph Transformer for Node Classification [11.451341325579188]
  We propose a new graph Transformer called NTFormer to address node classification issues. New token generator called Node2Par generates various token sequences using different token elements for each node. Experiments conducted on various benchmark datasets demonstrate the superiority of NTFormer over representative graph Transformers and graph neural networks for node classification.
  arXiv  Detail & Related papers  (2024-06-27T15:16:00Z)
• A Pure Transformer Pretraining Framework on Text-attributed Graphs [50.833130854272774]
  We introduce a feature-centric pretraining perspective by treating graph structure as a prior. Our framework, Graph Sequence Pretraining with Transformer (GSPT), samples node contexts through random walks. GSPT can be easily adapted to both node classification and link prediction, demonstrating promising empirical success on various datasets.
  arXiv  Detail & Related papers  (2024-06-19T22:30:08Z)
• GraphViz2Vec: A Structure-aware Feature Generation Model to Improve Classification in GNNs [2.0823678201707034]
  GNNs are widely used to solve various tasks including node classification and link prediction. In this paper, we presented a novel feature extraction methodology GraphViz2Vec that can capture structural information of a node's local neighbourhood. These initial embeddings helps existing models achieve state-of-the-art results in various classification tasks.
  arXiv  Detail & Related papers  (2024-01-30T17:11:04Z)
• Degree-based stratification of nodes in Graph Neural Networks [66.17149106033126]
  We modify the Graph Neural Network (GNN) architecture so that the weight matrices are learned, separately, for the nodes in each group. This simple-to-implement modification seems to improve performance across datasets and GNN methods.
  arXiv  Detail & Related papers  (2023-12-16T14:09:23Z)
• Discrete Graph Auto-Encoder [52.50288418639075]
  We introduce a new framework named Discrete Graph Auto-Encoder (DGAE) We first use a permutation-equivariant auto-encoder to convert graphs into sets of discrete latent node representations. In the second step, we sort the sets of discrete latent representations and learn their distribution with a specifically designed auto-regressive model.
  arXiv  Detail & Related papers  (2023-06-13T12:40:39Z)
• AGFormer: Efficient Graph Representation with Anchor-Graph Transformer [95.1825252182316]
  We propose a novel graph Transformer architecture, termed Anchor Graph Transformer (AGFormer) AGFormer first obtains some representative anchors and then converts node-to-node message passing into anchor-to-anchor and anchor-to-node message passing process. Extensive experiments on several benchmark datasets demonstrate the effectiveness and benefits of proposed AGFormer.
  arXiv  Detail & Related papers  (2023-05-12T14:35:42Z)
• Deformable Graph Transformer [31.254872949603982]
  We propose Deformable Graph Transformer (DGT) that performs sparse attention with dynamically sampled key and value pairs. Experiments demonstrate that our novel graph Transformer consistently outperforms existing Transformer-based models.
  arXiv  Detail & Related papers  (2022-06-29T00:23:25Z)
• Gransformer: Transformer-based Graph Generation [14.161975556325796]
  Gransformer is an algorithm based on Transformer for generating graphs. We modify the Transformer encoder to exploit the structural information of the given graph. We also introduce a graph-based familiarity measure between node pairs.
  arXiv  Detail & Related papers  (2022-03-25T14:05:12Z)
• A Variational Edge Partition Model for Supervised Graph Representation Learning [51.30365677476971]
  This paper introduces a graph generative process to model how the observed edges are generated by aggregating the node interactions over a set of overlapping node communities. We partition each edge into the summation of multiple community-specific weighted edges and use them to define community-specific GNNs. A variational inference framework is proposed to jointly learn a GNN based inference network that partitions the edges into different communities, these community-specific GNNs, and a GNN based predictor that combines community-specific GNNs for the end classification task.
  arXiv  Detail & Related papers  (2022-02-07T14:37:50Z)
• Graph Inference Learning for Semi-supervised Classification [50.55765399527556]
  We propose a Graph Inference Learning framework to boost the performance of semi-supervised node classification. For learning the inference process, we introduce meta-optimization on structure relations from training nodes to validation nodes. Comprehensive evaluations on four benchmark datasets demonstrate the superiority of our proposed GIL when compared against state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-01-17T02:52:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.