Supervised Pretraining for Material Property Prediction

• URL: http://arxiv.org/abs/2504.20112v1
• Date: Sun, 27 Apr 2025 19:00:41 GMT
• Title: Supervised Pretraining for Material Property Prediction
• Authors: Chowdhury Mohammad Abid Rahman, Aldo H. Romero, Prashnna K. Gyawali,
• Abstract summary: Self-supervised learning (SSL) offers a promising alternative by pretraining on large, unlabeled datasets to develop foundation models.<n>In this work, we propose supervised pretraining, where available class information serves as surrogate labels to guide learning.<n>To further enhance representation learning, we propose a graph-based augmentation technique that injects noise to improve robustness without structurally deforming material graphs.
• Score: 0.36868085124383626
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Accurate prediction of material properties facilitates the discovery of novel materials with tailored functionalities. Deep learning models have recently shown superior accuracy and flexibility in capturing structure-property relationships. However, these models often rely on supervised learning, which requires large, well-annotated datasets an expensive and time-consuming process. Self-supervised learning (SSL) offers a promising alternative by pretraining on large, unlabeled datasets to develop foundation models that can be fine-tuned for material property prediction. In this work, we propose supervised pretraining, where available class information serves as surrogate labels to guide learning, even when downstream tasks involve unrelated material properties. We evaluate this strategy on two state-of-the-art SSL models and introduce a novel framework for supervised pretraining. To further enhance representation learning, we propose a graph-based augmentation technique that injects noise to improve robustness without structurally deforming material graphs. The resulting foundation models are fine-tuned for six challenging material property predictions, achieving significant performance gains over baselines, ranging from 2% to 6.67% improvement in mean absolute error (MAE) and establishing a new benchmark in material property prediction. This study represents the first exploration of supervised pertaining with surrogate labels in material property prediction, advancing methodology and application in the field.

Related papers

• Accurate predictive model of band gap with selected important features based on explainable machine learning [0.0]
  This study employs explainable ML (XML) techniques, including permutation feature importance and the SHapley Additive exPlanation.<n> Guided by XML-derived individual feature importance, a simple framework is proposed to construct reduced-feature predictive models.<n>Model evaluations indicate that an XML-guided compact model, consisting of the top five features, achieves comparable accuracy to the pristine model on in-domain datasets.
  arXiv  Detail & Related papers  (2025-03-06T14:40:21Z)
• Self-supervised learning for crystal property prediction via denoising [43.148818844265236]
  We propose a novel self-supervised learning (SSL) strategy for material property prediction. Our approach, crystal denoising self-supervised learning (CDSSL), pretrains predictive models with a pretext task based on recovering valid material structures. We demonstrate that CDSSL models out-perform models trained without SSL, across material types, properties, and dataset sizes.
  arXiv  Detail & Related papers  (2024-08-30T12:53:40Z)
• Out-of-distribution materials property prediction using adversarial learning based fine-tuning [0.0]
  We propose an adversarial learning based targeting finetuning approach to make the model adapted to a particular dataset. Our experiments demonstrate the success of our CAL algorithm with its high effectiveness in ML with limited samples.
  arXiv  Detail & Related papers  (2024-08-17T21:22:21Z)
• Establishing Deep InfoMax as an effective self-supervised learning methodology in materials informatics [0.0]
  Deep InfoMax is a self-supervised machine learning framework for materials informatics. Deep InfoMax maximises the mutual information between a point set (or graph) representation of a crystal and a vector representation suitable for downstream learning. We investigate the benefits of Deep InfoMax pretraining implemented on the Site-Net architecture to improve the performance of downstream property prediction models.
  arXiv  Detail & Related papers  (2024-06-30T11:33:49Z)
• F-FOMAML: GNN-Enhanced Meta-Learning for Peak Period Demand Forecasting with Proxy Data [65.6499834212641]
  We formulate the demand prediction as a meta-learning problem and develop the Feature-based First-Order Model-Agnostic Meta-Learning (F-FOMAML) algorithm. By considering domain similarities through task-specific metadata, our model improved generalization, where the excess risk decreases as the number of training tasks increases. Compared to existing state-of-the-art models, our method demonstrates a notable improvement in demand prediction accuracy, reducing the Mean Absolute Error by 26.24% on an internal vending machine dataset and by 1.04% on the publicly accessible JD.com dataset.
  arXiv  Detail & Related papers  (2024-06-23T21:28:50Z)
• Impact of Noisy Supervision in Foundation Model Learning [91.56591923244943]
  This paper is the first work to comprehensively understand and analyze the nature of noise in pre-training datasets.<n>We propose a tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise and improve generalization.
  arXiv  Detail & Related papers  (2024-03-11T16:22:41Z)
• Structure-based out-of-distribution (OOD) materials property prediction: a benchmark study [1.3711992220025948]
  We present a benchmark study of structure-based graph neural networks (GNNs) for extrapolative OOD materials property prediction. Our experiments show that current state-of-the-art GNN algorithms significantly underperform for the OOD property prediction tasks. We identify the sources of CGCNN, ALIGNN, and DeeperGATGNN's significantly more robust OOD performance than those of the current best models.
  arXiv  Detail & Related papers  (2024-01-16T01:03:39Z)
• Universal Domain Adaptation from Foundation Models: A Baseline Study [58.51162198585434]
  We make empirical studies of state-of-the-art UniDA methods using foundation models. We introduce textitCLIP distillation, a parameter-free method specifically designed to distill target knowledge from CLIP models. Although simple, our method outperforms previous approaches in most benchmark tasks.
  arXiv  Detail & Related papers  (2023-05-18T16:28:29Z)
• Large Language Models with Controllable Working Memory [64.71038763708161]
  Large language models (LLMs) have led to a series of breakthroughs in natural language processing (NLP) What further sets these models apart is the massive amounts of world knowledge they internalize during pretraining. How the model's world knowledge interacts with the factual information presented in the context remains under explored.
  arXiv  Detail & Related papers  (2022-11-09T18:58:29Z)
• Self-Distillation for Further Pre-training of Transformers [83.84227016847096]
  We propose self-distillation as a regularization for a further pre-training stage. We empirically validate the efficacy of self-distillation on a variety of benchmark datasets for image and text classification tasks.
  arXiv  Detail & Related papers  (2022-09-30T02:25:12Z)
• Towards Open-World Feature Extrapolation: An Inductive Graph Learning Approach [80.8446673089281]
  We propose a new learning paradigm with graph representation and learning. Our framework contains two modules: 1) a backbone network (e.g., feedforward neural nets) as a lower model takes features as input and outputs predicted labels; 2) a graph neural network as an upper model learns to extrapolate embeddings for new features via message passing over a feature-data graph built from observed data.
  arXiv  Detail & Related papers  (2021-10-09T09:02:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.