TAG-DLM: Diffusion Language Models for Text-Attributed Graph Learning

• URL: http://arxiv.org/abs/2606.31166v1
• Date: Tue, 30 Jun 2026 05:56:18 GMT
• Title: TAG-DLM: Diffusion Language Models for Text-Attributed Graph Learning
• Abstract summary: Text-attributed graphs (TAGs) require models to jointly reason over text and graph topology.<n>We propose method that unifies textual reasoning and graph message passing within a masked diffusion language model.<n>Experiments show that method outperforms graph neural networks, graph transformers, and LLM-based baselines on all three TAG benchmarks.
• Score: 44.15996692667212
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Text-attributed graphs (TAGs), where each node carries a natural language description, require models to jointly reason over text and graph topology. Existing approaches often handle the two modalities separately: graph neural networks operate on shallow text features, while hybrids of LLMs and graphs use the language model mainly as a text encoder and delegate structure learning to a separate graph module. We propose method that unifies textual reasoning and graph message passing within a masked diffusion language model, a language model with bidirectional attention and generative decoding. For each graph instance, method linearises a sampled local neighbourhood into a token sequence and injects graph structure through a topology attention mask, which realises message passing over the graph. Because the diffusion language model can both interpret and generate text, the method adapts to different tasks simply by changing the prompt, supporting node classification, link prediction, and cross-dataset transfer with no target-specific fine-tuning. Experiments show that method outperforms graph neural networks, graph transformers, and LLM-based baselines on all three TAG benchmarks across two tasks, improving over the strongest baseline by up to 3.9 points.

Related papers

• LLM as GNN: Graph Vocabulary Learning for Text-Attributed Graph Foundation Models [87.68057302738457]
  Text-Attributed Graphs (TAGs) are ubiquitous in real-world scenarios.<n>Despite large efforts to integrate Large Language Models (LLMs) and Graph Neural Networks (GNNs) for TAGs, existing approaches suffer from decoupled architectures.<n>We propose PromptGFM, a versatile GFM for TAGs grounded in graph vocabulary learning.
  arXiv  Detail & Related papers  (2025-03-05T09:45:22Z)
• Pre-Training and Prompting for Few-Shot Node Classification on Text-Attributed Graphs [35.44563283531432]
  Text-attributed graph (TAG) is one kind of important real-world graph-structured data with each node associated with raw texts. For TAGs, traditional few-shot node classification methods directly conduct training on the pre-processed node features. We propose P2TAG, a framework designed for few-shot node classification on TAGs with graph pre-training and prompting.
  arXiv  Detail & Related papers  (2024-07-22T07:24:21Z)
• UniGLM: Training One Unified Language Model for Text-Attributed Graph Embedding [31.464021556351685]
  Unified Graph Language Model (UniGLM) is a graph embedding model that generalizes well to both in-domain and cross-domain TAGs.<n>UniGLM includes an adaptive positive sample selection technique for identifying structurally similar nodes and a lazy contrastive module that is devised to accelerate training.
  arXiv  Detail & Related papers  (2024-06-17T19:45:21Z)
• Parameter-Efficient Tuning Large Language Models for Graph Representation Learning [62.26278815157628]
  We introduce Graph-aware. Efficient Fine-Tuning - GPEFT, a novel approach for efficient graph representation learning. We use a graph neural network (GNN) to encode structural information from neighboring nodes into a graph prompt. We validate our approach through comprehensive experiments conducted on 8 different text-rich graphs, observing an average improvement of 2% in hit@1 and Mean Reciprocal Rank (MRR) in link prediction evaluations.
  arXiv  Detail & Related papers  (2024-04-28T18:36:59Z)
• Pretraining Language Models with Text-Attributed Heterogeneous Graphs [28.579509154284448]
  We present a new pretraining framework for Language Models (LMs) that explicitly considers the topological and heterogeneous information in Text-Attributed Heterogeneous Graphs (TAHGs) We propose a topology-aware pretraining task to predict nodes involved in the context graph by jointly optimizing an LM and an auxiliary heterogeneous graph neural network. We conduct link prediction and node classification tasks on three datasets from various domains.
  arXiv  Detail & Related papers  (2023-10-19T08:41:21Z)
• Learning Multiplex Representations on Text-Attributed Graphs with One Language Model Encoder [55.24276913049635]
  We propose METAG, a new framework for learning Multiplex rEpresentations on Text-Attributed Graphs. In contrast to existing methods, METAG uses one text encoder to model the shared knowledge across relations. We conduct experiments on nine downstream tasks in five graphs from both academic and e-commerce domains.
  arXiv  Detail & Related papers  (2023-10-10T14:59:22Z)
• One for All: Towards Training One Graph Model for All Classification Tasks [61.656962278497225]
  A unified model for various graph tasks remains underexplored, primarily due to the challenges unique to the graph learning domain. We propose textbfOne for All (OFA), the first general framework that can use a single graph model to address the above challenges. OFA performs well across different tasks, making it the first general-purpose across-domains classification model on graphs.
  arXiv  Detail & Related papers  (2023-09-29T21:15:26Z)
• ConGraT: Self-Supervised Contrastive Pretraining for Joint Graph and Text Embeddings [20.25180279903009]
  We propose Contrastive Graph-Text pretraining (ConGraT) for jointly learning separate representations of texts and nodes in a text-attributed graph (TAG) Our method trains a language model (LM) and a graph neural network (GNN) to align their representations in a common latent space using a batch-wise contrastive learning objective inspired by CLIP. Experiments demonstrate that ConGraT outperforms baselines on various downstream tasks, including node and text category classification, link prediction, and language modeling.
  arXiv  Detail & Related papers  (2023-05-23T17:53:30Z)
• GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph [53.70520466556453]
  We propose GraphFormers, where layerwise GNN components are nested alongside the transformer blocks of language models. With the proposed architecture, the text encoding and the graph aggregation are fused into an iterative workflow. In addition, a progressive learning strategy is introduced, where the model is successively trained on manipulated data and original data to reinforce its capability of integrating information on graph.
  arXiv  Detail & Related papers  (2021-05-06T12:20:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.