Attention Beyond Neighborhoods: Reviving Transformer for Graph Clustering

• URL: http://arxiv.org/abs/2509.15024v1
• Date: Thu, 18 Sep 2025 14:51:13 GMT
• Title: Attention Beyond Neighborhoods: Reviving Transformer for Graph Clustering
• Authors: Xuanting Xie, Bingheng Li, Erlin Pan, Rui Hou, Wenyu Chen, Zhao Kang,
• Abstract summary: Attentive Graph Clustering Network (AGCN) is a novel architecture that reinterprets the notion that graph is attention.<n>AGCN embeds the attention mechanism into the graph structure, enabling effective global information extraction.<n>Our framework incorporates theoretical analysis to contrast AGCN behavior with Graph Neural Networks (GNNs) and Transformer.
• Score: 21.941792185132996
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Attention mechanisms have become a cornerstone in modern neural networks, driving breakthroughs across diverse domains. However, their application to graph structured data, where capturing topological connections is essential, remains underexplored and underperforming compared to Graph Neural Networks (GNNs), particularly in the graph clustering task. GNN tends to overemphasize neighborhood aggregation, leading to a homogenization of node representations. Conversely, Transformer tends to over globalize, highlighting distant nodes at the expense of meaningful local patterns. This dichotomy raises a key question: Is attention inherently redundant for unsupervised graph learning? To address this, we conduct a comprehensive empirical analysis, uncovering the complementary weaknesses of GNN and Transformer in graph clustering. Motivated by these insights, we propose the Attentive Graph Clustering Network (AGCN) a novel architecture that reinterprets the notion that graph is attention. AGCN directly embeds the attention mechanism into the graph structure, enabling effective global information extraction while maintaining sensitivity to local topological cues. Our framework incorporates theoretical analysis to contrast AGCN behavior with GNN and Transformer and introduces two innovations: (1) a KV cache mechanism to improve computational efficiency, and (2) a pairwise margin contrastive loss to boost the discriminative capacity of the attention space. Extensive experimental results demonstrate that AGCN outperforms state-of-the-art methods.

Related papers

• Dual-Kernel Graph Community Contrastive Learning [14.92920991249099]
  Graph Contrastive Learning (GCL) has emerged as a powerful paradigm for training Graph Neural Networks (GNNs)<n>We propose an efficient GCL framework that transforms the input graph into a compact network of interconnected node sets.<n>Our method outperforms state-of-the-art GCL baselines in both effectiveness and scalability.
  arXiv  Detail & Related papers  (2025-11-11T14:20:39Z)
• GLANCE: Graph Logic Attention Network with Cluster Enhancement for Heterophilous Graph Representation Learning [54.60090631330295]
  Graph Neural Networks (GNNs) have demonstrated significant success in learning from graph-structured data but often struggle on heterophilous graphs.<n>We propose GLANCE, a novel framework that integrates logic-guided reasoning, dynamic graph refinement, and adaptive clustering to enhance graph representation learning.
  arXiv  Detail & Related papers  (2025-07-24T15:45:26Z)
• Graph Attention for Heterogeneous Graphs with Positional Encoding [0.0]
  Graph Neural Networks (GNNs) have emerged as the de facto standard for modeling graph data.<n>This work benchmarks various GNN architectures to identify the most effective methods for heterogeneous graphs.<n>Our findings reveal that graph attention networks excel in these tasks.
  arXiv  Detail & Related papers  (2025-04-03T18:00:02Z)
• Deep Contrastive Graph Learning with Clustering-Oriented Guidance [61.103996105756394]
  Graph Convolutional Network (GCN) has exhibited remarkable potential in improving graph-based clustering. Models estimate an initial graph beforehand to apply GCN. Deep Contrastive Graph Learning (DCGL) model is proposed for general data clustering.
  arXiv  Detail & Related papers  (2024-02-25T07:03:37Z)
• DGNN: Decoupled Graph Neural Networks with Structural Consistency between Attribute and Graph Embedding Representations [62.04558318166396]
  Graph neural networks (GNNs) demonstrate a robust capability for representation learning on graphs with complex structures. A novel GNNs framework, dubbed Decoupled Graph Neural Networks (DGNN), is introduced to obtain a more comprehensive embedding representation of nodes. Experimental results conducted on several graph benchmark datasets verify DGNN's superiority in node classification task.
  arXiv  Detail & Related papers  (2024-01-28T06:43:13Z)
• Locality-Aware Graph-Rewiring in GNNs [5.356465360780597]
  Graph Neural Networks (GNNs) are popular models for machine learning on graphs. In this work, we identify three desiderata for graph-rewiring: (i) reduce over-squashing, (ii) respect the locality of the graph, and (iii) preserve the sparsity of the graph. We propose a novel rewiring framework that satisfies all of (i)iii through a locality-aware sequence of rewiring operations.
  arXiv  Detail & Related papers  (2023-10-02T21:59:44Z)
• Transforming Graphs for Enhanced Attribute Clustering: An Innovative Graph Transformer-Based Method [8.989218350080844]
  This study introduces an innovative method known as the Graph Transformer Auto-Encoder for Graph Clustering (GTAGC) By melding the Graph Auto-Encoder with the Graph Transformer, GTAGC is adept at capturing global dependencies between nodes. The architecture of GTAGC encompasses graph embedding, integration of the Graph Transformer within the autoencoder structure, and a clustering component.
  arXiv  Detail & Related papers  (2023-06-20T06:04:03Z)
• DEGREE: Decomposition Based Explanation For Graph Neural Networks [55.38873296761104]
  We propose DEGREE to provide a faithful explanation for GNN predictions. By decomposing the information generation and aggregation mechanism of GNNs, DEGREE allows tracking the contributions of specific components of the input graph to the final prediction. We also design a subgraph level interpretation algorithm to reveal complex interactions between graph nodes that are overlooked by previous methods.
  arXiv  Detail & Related papers  (2023-05-22T10:29:52Z)
• Graph Transformer GANs for Graph-Constrained House Generation [223.739067413952]
  We present a novel graph Transformer generative adversarial network (GTGAN) to learn effective graph node relations. The GTGAN learns effective graph node relations in an end-to-end fashion for the challenging graph-constrained house generation task.
  arXiv  Detail & Related papers  (2023-03-14T20:35:45Z)
• MentorGNN: Deriving Curriculum for Pre-Training GNNs [61.97574489259085]
  We propose an end-to-end model named MentorGNN that aims to supervise the pre-training process of GNNs across graphs. We shed new light on the problem of domain adaption on relational data (i.e., graphs) by deriving a natural and interpretable upper bound on the generalization error of the pre-trained GNNs.
  arXiv  Detail & Related papers  (2022-08-21T15:12:08Z)
• ACE-HGNN: Adaptive Curvature Exploration Hyperbolic Graph Neural Network [72.16255675586089]
  We propose an Adaptive Curvature Exploration Hyperbolic Graph NeuralNetwork named ACE-HGNN to adaptively learn the optimal curvature according to the input graph and downstream tasks. Experiments on multiple real-world graph datasets demonstrate a significant and consistent performance improvement in model quality with competitive performance and good generalization ability.
  arXiv  Detail & Related papers  (2021-10-15T07:18:57Z)
• Hierarchical Message-Passing Graph Neural Networks [12.207978823927386]
  We propose a novel Hierarchical Message-passing Graph Neural Networks framework. Key idea is generating a hierarchical structure that re-organises all nodes in a flat graph into multi-level super graphs. We present the first model to implement this framework, termed Hierarchical Community-aware Graph Neural Network (HC-GNN)
  arXiv  Detail & Related papers  (2020-09-08T13:11:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.