Crystal Twins: Self-supervised Learning for Crystalline Material Property Prediction

• URL: http://arxiv.org/abs/2205.01893v1
• Date: Wed, 4 May 2022 05:08:46 GMT
• Title: Crystal Twins: Self-supervised Learning for Crystalline Material Property Prediction
• Authors: Rishikesh Magar, Yuyang Wang, and Amir Barati Farimani
• Abstract summary: We introduce Crystal Twins (CT): an SSL method for crystalline materials property prediction. We pre-train a Graph Neural Network (GNN) by applying the redundancy reduction principle to the graph latent embeddings of augmented instances. By sharing the pre-trained weights when fine-tuning the GNN for regression tasks, we significantly improve the performance for 7 challenging material property prediction benchmarks.
• Score: 8.048439531116367
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine learning (ML) models have been widely successful in the prediction of material properties. However, large labeled datasets required for training accurate ML models are elusive and computationally expensive to generate. Recent advances in Self-Supervised Learning (SSL) frameworks capable of training ML models on unlabeled data have mitigated this problem and demonstrated superior performance in computer vision and natural language processing tasks. Drawing inspiration from the developments in SSL, we introduce Crystal Twins (CT): an SSL method for crystalline materials property prediction. Using a large unlabeled dataset, we pre-train a Graph Neural Network (GNN) by applying the redundancy reduction principle to the graph latent embeddings of augmented instances obtained from the same crystalline system. By sharing the pre-trained weights when fine-tuning the GNN for regression tasks, we significantly improve the performance for 7 challenging material property prediction benchmarks

Related papers

• What Do Learning Dynamics Reveal About Generalization in LLM Reasoning? [83.83230167222852]
  We find that a model's generalization behavior can be effectively characterized by a training metric we call pre-memorization train accuracy. By connecting a model's learning behavior to its generalization, pre-memorization train accuracy can guide targeted improvements to training strategies.
  arXiv  Detail & Related papers  (2024-11-12T09:52:40Z)
• Self-supervised learning for crystal property prediction via denoising [43.148818844265236]
  We propose a novel self-supervised learning (SSL) strategy for material property prediction. Our approach, crystal denoising self-supervised learning (CDSSL), pretrains predictive models with a pretext task based on recovering valid material structures. We demonstrate that CDSSL models out-perform models trained without SSL, across material types, properties, and dataset sizes.
  arXiv  Detail & Related papers  (2024-08-30T12:53:40Z)
• Diffusion-Based Neural Network Weights Generation [80.89706112736353]
  D2NWG is a diffusion-based neural network weights generation technique that efficiently produces high-performing weights for transfer learning. Our method extends generative hyper-representation learning to recast the latent diffusion paradigm for neural network weights generation. Our approach is scalable to large architectures such as large language models (LLMs), overcoming the limitations of current parameter generation techniques.
  arXiv  Detail & Related papers  (2024-02-28T08:34:23Z)
• Curated LLM: Synergy of LLMs and Data Curation for tabular augmentation in low-data regimes [57.62036621319563]
  We introduce CLLM, which leverages the prior knowledge of Large Language Models (LLMs) for data augmentation in the low-data regime. We demonstrate the superior performance of CLLM in the low-data regime compared to conventional generators.
  arXiv  Detail & Related papers  (2023-12-19T12:34:46Z)
• Defect Classification in Additive Manufacturing Using CNN-Based Vision Processing [76.72662577101988]
  This paper examines two scenarios: first, using convolutional neural networks (CNNs) to accurately classify defects in an image dataset from AM and second, applying active learning techniques to the developed classification model. This allows the construction of a human-in-the-loop mechanism to reduce the size of the data required to train and generate training data.
  arXiv  Detail & Related papers  (2023-07-14T14:36:58Z)
• Substitutional Alloying Using Crystal Graph Neural Networks [0.0]
  Graph Neural Networks (GNNs) allow for direct learning representations on graphs, such as the ones formed by crystals. We use CGNNs to predict crystal properties with DFT level accuracy, through graphs with encoding of the atomic (node/vertex), bond (edge), and global state attributes. We perform DFT validation to assess the accuracy in the prediction of formation energies and structural features.
  arXiv  Detail & Related papers  (2023-06-19T08:18:17Z)
• CrysGNN : Distilling pre-trained knowledge to enhance property prediction for crystalline materials [25.622724168215097]
  This paper presents CrysGNN, a new pre-trained GNN framework for crystalline materials. It captures both node and graph level structural information of crystal graphs using unlabelled material data. We conduct extensive experiments to show that with distilled knowledge from the pre-trained model, all the SOTA algorithms are able to outperform their own vanilla version with good margins.
  arXiv  Detail & Related papers  (2023-01-14T08:12:01Z)
• Graph Contrastive Learning for Materials [6.667711415870472]
  We introduce CrystalCLR, a framework for constrastive learning of representations with crystal graph neural networks. With the addition of a novel loss function, our framework is able to learn representations competitive with engineered fingerprinting methods. We also demonstrate that via model finetuning, contrastive pretraining can improve the performance of graph neural networks for prediction of material properties.
  arXiv  Detail & Related papers  (2022-11-24T04:15:47Z)
• An Adversarial Active Sampling-based Data Augmentation Framework for Manufacturable Chip Design [55.62660894625669]
  Lithography modeling is a crucial problem in chip design to ensure a chip design mask is manufacturable. Recent developments in machine learning have provided alternative solutions in replacing the time-consuming lithography simulations with deep neural networks. We propose a litho-aware data augmentation framework to resolve the dilemma of limited data and improve the machine learning model performance.
  arXiv  Detail & Related papers  (2022-10-27T20:53:39Z)
• Scalable deeper graph neural networks for high-performance materials property prediction [1.9129213267332026]
  We propose a novel graph attention neural network model DeeperGATGNN with differenti groupable normalization and skip-connections. Our work shows that to deal with the high complexity of mapping the crystal materials structures to their properties, large-scale very deep graph neural networks are needed to achieve robust performances.
  arXiv  Detail & Related papers  (2021-09-25T05:58:04Z)
• On the Transferability of Pre-trained Language Models: A Study from Artificial Datasets [74.11825654535895]
  Pre-training language models (LMs) on large-scale unlabeled text data makes the model much easier to achieve exceptional downstream performance. We study what specific traits in the pre-training data, other than the semantics, make a pre-trained LM superior to their counterparts trained from scratch on downstream tasks.
  arXiv  Detail & Related papers  (2021-09-08T10:39:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.