Scaling Laws of Graph Neural Networks for Atomistic Materials Modeling

• URL: http://arxiv.org/abs/2504.08112v1
• Date: Thu, 10 Apr 2025 20:19:20 GMT
• Title: Scaling Laws of Graph Neural Networks for Atomistic Materials Modeling
• Authors: Chaojian Li, Zhifan Ye, Massimiliano Lupo Pasini, Jong Youl Choi, Cheng Wan, Yingyan Celine Lin, Prasanna Balaprakash,
• Abstract summary: Atomistic materials modeling is a critical task with wide-ranging applications, from drug discovery to materials science.<n>Graph Neural Networks (GNNs) represent the state-of-the-art approach for modeling atomistic material data.<n>GNNs for atomistic materials modeling remain relatively small compared to large language models (LLMs), which leverage billions of parameters and terabyte-scale datasets.
• Score: 11.61369154220932
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Atomistic materials modeling is a critical task with wide-ranging applications, from drug discovery to materials science, where accurate predictions of the target material property can lead to significant advancements in scientific discovery. Graph Neural Networks (GNNs) represent the state-of-the-art approach for modeling atomistic material data thanks to their capacity to capture complex relational structures. While machine learning performance has historically improved with larger models and datasets, GNNs for atomistic materials modeling remain relatively small compared to large language models (LLMs), which leverage billions of parameters and terabyte-scale datasets to achieve remarkable performance in their respective domains. To address this gap, we explore the scaling limits of GNNs for atomistic materials modeling by developing a foundational model with billions of parameters, trained on extensive datasets in terabyte-scale. Our approach incorporates techniques from LLM libraries to efficiently manage large-scale data and models, enabling both effective training and deployment of these large-scale GNN models. This work addresses three fundamental questions in scaling GNNs: the potential for scaling GNN model architectures, the effect of dataset size on model accuracy, and the applicability of LLM-inspired techniques to GNN architectures. Specifically, the outcomes of this study include (1) insights into the scaling laws for GNNs, highlighting the relationship between model size, dataset volume, and accuracy, (2) a foundational GNN model optimized for atomistic materials modeling, and (3) a GNN codebase enhanced with advanced LLM-based training techniques. Our findings lay the groundwork for large-scale GNNs with billions of parameters and terabyte-scale datasets, establishing a scalable pathway for future advancements in atomistic materials modeling.

Related papers

• MatterTune: An Integrated, User-Friendly Platform for Fine-Tuning Atomistic Foundation Models to Accelerate Materials Simulation and Discovery [7.1240120153291535]
  We introduce MatterTune, a framework that provides advanced fine-tuning capabilities and seamless integration of atomistic foundation models into downstream materials informatics and simulation. MatterTune supports a number of state-of-the-art foundation models such as ORB, MatterSim, JMP, and EquformerV2.
  arXiv  Detail & Related papers  (2025-04-14T19:12:43Z)
• DenseGNN: universal and scalable deeper graph neural networks for high-performance property prediction in crystals and molecules [4.648990939177395]
  We introduce DenseGNN, which employs Dense Connectivity Network (DCN), Hierarchical Node-Edge-Graph Residual Networks (HRN), and Local Structure Order Parameters Embedding (LOPE)<n>DenseGNN achieves state-of-the-art performance on datasets such as JARVIS-DFT, Materials Project, and QM9, improving the performance of models like GIN, Schnet, and Hamnet on materials datasets.
  arXiv  Detail & Related papers  (2025-01-05T13:41:28Z)
• Conservation-informed Graph Learning for Spatiotemporal Dynamics Prediction [84.26340606752763]
  In this paper, we introduce the conservation-informed GNN (CiGNN), an end-to-end explainable learning framework.<n>The network is designed to conform to the general symmetry conservation law via symmetry where conservative and non-conservative information passes over a multiscale space by a latent temporal marching strategy.<n>Results demonstrate that CiGNN exhibits remarkable baseline accuracy and generalizability, and is readily applicable to learning for prediction of varioustemporal dynamics.
  arXiv  Detail & Related papers  (2024-12-30T13:55:59Z)
• Neural P$^3$M: A Long-Range Interaction Modeling Enhancer for Geometric GNNs [66.98487644676906]
  We introduce Neural P$3$M, a versatile enhancer of geometric GNNs to expand the scope of their capabilities. It exhibits flexibility across a wide range of molecular systems and demonstrates remarkable accuracy in predicting energies and forces. It also achieves an average improvement of 22% on the OE62 dataset while integrating with various architectures.
  arXiv  Detail & Related papers  (2024-09-26T08:16:59Z)
• Improving Molecular Modeling with Geometric GNNs: an Empirical Study [56.52346265722167]
  This paper focuses on the impact of different canonicalization methods, (2) graph creation strategies, and (3) auxiliary tasks, on performance, scalability and symmetry enforcement. Our findings and insights aim to guide researchers in selecting optimal modeling components for molecular modeling tasks.
  arXiv  Detail & Related papers  (2024-07-11T09:04:12Z)
• On the Scalability of GNNs for Molecular Graphs [7.402389334892391]
  Graph Neural Networks (GNNs) are yet to show the benefits of scale due to the lower efficiency of sparse operations, large data requirements, and lack of clarity about the effectiveness of various architectures. We analyze message-passing networks, graph Transformers, and hybrid architectures on the largest public collection of 2D molecular graphs. For the first time, we observe that GNNs benefit tremendously from the increasing scale of depth, width, number of molecules, number of labels, and the diversity in the pretraining datasets.
  arXiv  Detail & Related papers  (2024-04-17T17:11:31Z)
• Diffusion-Based Neural Network Weights Generation [80.89706112736353]
  D2NWG is a diffusion-based neural network weights generation technique that efficiently produces high-performing weights for transfer learning. Our method extends generative hyper-representation learning to recast the latent diffusion paradigm for neural network weights generation. Our approach is scalable to large architectures such as large language models (LLMs), overcoming the limitations of current parameter generation techniques.
  arXiv  Detail & Related papers  (2024-02-28T08:34:23Z)
• Data-Free Adversarial Knowledge Distillation for Graph Neural Networks [62.71646916191515]
  We propose the first end-to-end framework for data-free adversarial knowledge distillation on graph structured data (DFAD-GNN) To be specific, our DFAD-GNN employs a generative adversarial network, which mainly consists of three components: a pre-trained teacher model and a student model are regarded as two discriminators, and a generator is utilized for deriving training graphs to distill knowledge from the teacher model into the student model. Our DFAD-GNN significantly surpasses state-of-the-art data-free baselines in the graph classification task.
  arXiv  Detail & Related papers  (2022-05-08T08:19:40Z)
• Scalable deeper graph neural networks for high-performance materials property prediction [1.9129213267332026]
  We propose a novel graph attention neural network model DeeperGATGNN with differenti groupable normalization and skip-connections. Our work shows that to deal with the high complexity of mapping the crystal materials structures to their properties, large-scale very deep graph neural networks are needed to achieve robust performances.
  arXiv  Detail & Related papers  (2021-09-25T05:58:04Z)
• Rank-R FNN: A Tensor-Based Learning Model for High-Order Data Classification [69.26747803963907]
  Rank-R Feedforward Neural Network (FNN) is a tensor-based nonlinear learning model that imposes Canonical/Polyadic decomposition on its parameters. First, it handles inputs as multilinear arrays, bypassing the need for vectorization, and can thus fully exploit the structural information along every data dimension. We establish the universal approximation and learnability properties of Rank-R FNN, and we validate its performance on real-world hyperspectral datasets.
  arXiv  Detail & Related papers  (2021-04-11T16:37:32Z)
• Global Attention based Graph Convolutional Neural Networks for Improved Materials Property Prediction [8.371766047183739]
  We develop a novel model, GATGNN, for predicting inorganic material properties based on graph neural networks. We show that our method is able to both outperform the previous models' predictions and provide insight into the crystallization of the material.
  arXiv  Detail & Related papers  (2020-03-11T07:43:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.