Multi-scale Graph Autoregressive Modeling: Molecular Property Prediction via Next Token Prediction

• URL: http://arxiv.org/abs/2601.02530v2
• Date: Mon, 12 Jan 2026 13:52:52 GMT
• Title: Multi-scale Graph Autoregressive Modeling: Molecular Property Prediction via Next Token Prediction
• Abstract summary: Connection-Aware Motif Sequencing (CamS) is a graph-to-sequence representation for learning molecular graphs.<n>CamS bridges the gap by serializing molecular graphs into structure-rich causal sequences.<n>We instantiate CamS-LLaMA by pre-training a vanilla LLaMA backbone on CamS sequences.
• Score: 6.028057217858973
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present Connection-Aware Motif Sequencing (CamS), a graph-to-sequence representation that enables decoder-only Transformers to learn molecular graphs via standard next-token prediction (NTP). For molecular property prediction, SMILES-based NTP scales well but lacks explicit topology, whereas graph-native masked modeling captures connectivity but risks disrupting the pivotal chemical details (e.g., activity cliffs). CamS bridges this gap by serializing molecular graphs into structure-rich causal sequences. CamS first mines data-driven connection-aware motifs. It then serializes motifs via scaffold-rooted breadth-first search (BFS) to establish a stable core-to-periphery order. Crucially, CamS enables hierarchical modeling by concatenating sequences from fine to coarse motif scales, allowing the model to condition global scaffolds on dense, uncorrupted local structural evidence. We instantiate CamS-LLaMA by pre-training a vanilla LLaMA backbone on CamS sequences. It achieves state-of-the-art performance on MoleculeNet and the activity-cliff benchmark MoleculeACE, outperforming both SMILES-based language models and strong graph baselines. Interpretability analysis confirms that our multi-scale causal serialization effectively drives attention toward cliff-determining differences.

Related papers

• Multi-View Molecular Representation Learning with Hierarchical Graphs and Contextualized Fingerprints [2.7742696928124917]
  HiFi-Mol is a multi-view framework that pretrains a hierarchical graph encoder and a contextualized fingerprint encoder before downstream integration.<n>During fine-tuning, HiFi-Mol combines projected multi-resolution graph features with fingerprint embeddings for downstream prediction.
  arXiv  Detail & Related papers  (2026-09-14T14:07:19Z)
• SERUM: State Extraction and Refinement for User Modeling [17.94894550460533]
  We present SERUM, a framework that extracts finite-state behavioral models directly from unstructured egocentric video.<n>We evaluate behavioral structure by fitting first-order Markov models over the resulting label sequences.<n>Across 61 egocentric videos in four domains, we find: (1) iterative label refinement converges to a stable state vocabulary, which we term equilibrium, after several passes; (2) normalized Markov models achieve substantially lower perplexity and higher action predictions than frequency baselines; and (3) human annotators rate final-pass labels as accurate and meaningfully improved over first-pass labels.
  arXiv  Detail & Related papers  (2026-07-31T09:02:33Z)
• Scalable Graph Generative Modeling via Substructure Sequences [50.32639806800683]
  We introduce Generative Graph Pattern Machine (G$2$PM), a generative Transformer pre-training framework for graphs.<n>G$2$PM represents graph instances (nodes, edges, or entire graphs) as sequences of substructures.<n>It employs generative pre-training over the sequences to learn generalizable and transferable representations.
  arXiv  Detail & Related papers  (2025-05-22T02:16:34Z)
• Beyond Message Passing: Neural Graph Pattern Machine [50.78679002846741]
  We introduce the Neural Graph Pattern Machine (GPM), a novel framework that bypasses message passing by learning directly from graph substructures.<n>GPM efficiently extracts, encodes, and prioritizes task-relevant graph patterns, offering greater expressivity and improved ability to capture long-range dependencies.
  arXiv  Detail & Related papers  (2025-01-30T20:37:47Z)
• Best of Both Worlds: Advantages of Hybrid Graph Sequence Models [20.564009321626198]
  We present a unifying framework for adopting graph sequence models for learning on graphs. We evaluate the representation power of Transformers and modern recurrent models through the lens of global and local graph tasks. We present GSM++, a fast hybrid model that uses the Hierarchical Affinity Clustering (HAC) algorithm to tokenize the graph into hierarchical sequences.
  arXiv  Detail & Related papers  (2024-11-23T23:24:42Z)
• Pre-trained Molecular Language Models with Random Functional Group Masking [54.900360309677794]
  We propose a SMILES-based underlineem Molecular underlineem Language underlineem Model, which randomly masking SMILES subsequences corresponding to specific molecular atoms. This technique aims to compel the model to better infer molecular structures and properties, thus enhancing its predictive capabilities.
  arXiv  Detail & Related papers  (2024-11-03T01:56:15Z)
• GOFA: A Generative One-For-All Model for Joint Graph Language Modeling [38.267339613261996]
  We propose a novel generative graph language model GOFA to solve the problem.<n>GOFA is pre-trained on newly proposed graph-level next-word prediction, question-answering, and structural tasks.<n>The model is evaluated on various downstream tasks, demonstrating a strong ability to solve structural and contextual problems in zero-shot scenarios.
  arXiv  Detail & Related papers  (2024-07-12T22:23:51Z)
• STMT: A Spatial-Temporal Mesh Transformer for MoCap-Based Action Recognition [50.064502884594376]
  We study the problem of human action recognition using motion capture (MoCap) sequences. We propose a novel Spatial-Temporal Mesh Transformer (STMT) to directly model the mesh sequences. The proposed method achieves state-of-the-art performance compared to skeleton-based and point-cloud-based models.
  arXiv  Detail & Related papers  (2023-03-31T16:19:27Z)
• Hypergraph Transformer for Skeleton-based Action Recognition [21.763844802116857]
  Skeleton-based action recognition aims to recognize human actions given human joint coordinates with skeletal interconnections. Previous works successfully adopted Graph Convolutional networks (GCNs) to model joint co-occurrences. We propose a new self-attention (SA) mechanism on hypergraph, termed Hypergraph Self-Attention (HyperSA), to incorporate intrinsic higher-order relations into the model.
  arXiv  Detail & Related papers  (2022-11-17T15:36:48Z)
• Dynamic Graph Message Passing Networks for Visual Recognition [112.49513303433606]
  Modelling long-range dependencies is critical for scene understanding tasks in computer vision. A fully-connected graph is beneficial for such modelling, but its computational overhead is prohibitive. We propose a dynamic graph message passing network, that significantly reduces the computational complexity.
  arXiv  Detail & Related papers  (2022-09-20T14:41:37Z)
• Motif-based Graph Self-Supervised Learning forMolecular Property Prediction [12.789013658551454]
  Graph Neural Networks (GNNs) have demonstrated remarkable success in various molecular generation and prediction tasks. Most existing self-supervised pre-training frameworks for GNNs only focus on node-level or graph-level tasks. We propose a novel self-supervised motif generation framework for GNNs.
  arXiv  Detail & Related papers  (2021-10-03T11:45:51Z)
• SAC: Accelerating and Structuring Self-Attention via Sparse Adaptive Connection [51.376723069962]
  We present a method for accelerating and structuring self-attentions: Sparse Adaptive Connection. In SAC, we regard the input sequence as a graph and attention operations are performed between linked nodes. We show that SAC is competitive with state-of-the-art models while significantly reducing memory cost.
  arXiv  Detail & Related papers  (2020-03-22T07:58:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.