MatMMFuse: Multi-Modal Fusion model for Material Property Prediction

• URL: http://arxiv.org/abs/2505.04634v1
• Date: Wed, 30 Apr 2025 09:26:28 GMT
• Title: MatMMFuse: Multi-Modal Fusion model for Material Property Prediction
• Authors: Abhiroop Bhattacharya, Sylvain G. Cloutier,
• Abstract summary: We propose a fusion based model which uses structure aware embedding from the Crystal Graph Convolution Network (CGCNN) and text embeddings from the SciBERT model.<n>We show that our proposed model shows an improvement compared to the vanilla CGCNN model and 68% compared to the SciBERT model for predicting the formation energy per atom.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The recent progress of using graph based encoding of crystal structures for high throughput material property prediction has been quite successful. However, using a single modality model prevents us from exploiting the advantages of an enhanced features space by combining different representations. Specifically, pre-trained Large language models(LLMs) can encode a large amount of knowledge which is beneficial for training of models. Moreover, the graph encoder is able to learn the local features while the text encoder is able to learn global information such as space group and crystal symmetry. In this work, we propose Material Multi-Modal Fusion(MatMMFuse), a fusion based model which uses a multi-head attention mechanism for the combination of structure aware embedding from the Crystal Graph Convolution Network (CGCNN) and text embeddings from the SciBERT model. We train our model in an end-to-end framework using data from the Materials Project Dataset. We show that our proposed model shows an improvement compared to the vanilla CGCNN and SciBERT model for all four key properties: formation energy, band gap, energy above hull and fermi energy. Specifically, we observe an improvement of 40% compared to the vanilla CGCNN model and 68% compared to the SciBERT model for predicting the formation energy per atom. Importantly, we demonstrate the zero shot performance of the trained model on small curated datasets of Perovskites, Chalcogenides and the Jarvis Dataset. The results show that the proposed model exhibits better zero shot performance than the individual plain vanilla CGCNN and SciBERT model. This enables researchers to deploy the model for specialized industrial applications where collection of training data is prohibitively expensive.

Related papers

• Scaling Laws of Graph Neural Networks for Atomistic Materials Modeling [11.61369154220932]
  Atomistic materials modeling is a critical task with wide-ranging applications, from drug discovery to materials science.<n>Graph Neural Networks (GNNs) represent the state-of-the-art approach for modeling atomistic material data.<n>GNNs for atomistic materials modeling remain relatively small compared to large language models (LLMs), which leverage billions of parameters and terabyte-scale datasets.
  arXiv  Detail & Related papers  (2025-04-10T20:19:20Z)
• A Collaborative Ensemble Framework for CTR Prediction [73.59868761656317]
  We propose a novel framework, Collaborative Ensemble Training Network (CETNet), to leverage multiple distinct models. Unlike naive model scaling, our approach emphasizes diversity and collaboration through collaborative learning. We validate our framework on three public datasets and a large-scale industrial dataset from Meta.
  arXiv  Detail & Related papers  (2024-11-20T20:38:56Z)
• Enabling Small Models for Zero-Shot Selection and Reuse through Model Label Learning [50.68074833512999]
  We introduce a novel paradigm, Model Label Learning (MLL), which bridges the gap between models and their functionalities.<n>Experiments on seven real-world datasets validate the effectiveness and efficiency of MLL.
  arXiv  Detail & Related papers  (2024-08-21T09:08:26Z)
• Generative Expansion of Small Datasets: An Expansive Graph Approach [13.053285552524052]
  We introduce an Expansive Synthesis model generating large-scale, information-rich datasets from minimal samples. An autoencoder with self-attention layers and optimal transport refines distributional consistency. Results show comparable performance, demonstrating the model's potential to augment training data effectively.
  arXiv  Detail & Related papers  (2024-06-25T02:59:02Z)
• StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data [129.92449761766025]
  We propose a novel data collection methodology that synchronously synthesizes images and dialogues for visual instruction tuning. This approach harnesses the power of generative models, marrying the abilities of ChatGPT and text-to-image generative models. Our research includes comprehensive experiments conducted on various datasets.
  arXiv  Detail & Related papers  (2023-08-20T12:43:52Z)
• CrysGNN : Distilling pre-trained knowledge to enhance property prediction for crystalline materials [25.622724168215097]
  This paper presents CrysGNN, a new pre-trained GNN framework for crystalline materials. It captures both node and graph level structural information of crystal graphs using unlabelled material data. We conduct extensive experiments to show that with distilled knowledge from the pre-trained model, all the SOTA algorithms are able to outperform their own vanilla version with good margins.
  arXiv  Detail & Related papers  (2023-01-14T08:12:01Z)
• Heterogenous Ensemble of Models for Molecular Property Prediction [55.91865861896012]
  We propose a method for considering different modalities on molecules. We ensemble these models with a HuberRegressor. This yields a winning solution to the 2textsuperscriptnd edition of the OGB Large-Scale Challenge (2022)
  arXiv  Detail & Related papers  (2022-11-20T17:25:26Z)
• Comparing Test Sets with Item Response Theory [53.755064720563]
  We evaluate 29 datasets using predictions from 18 pretrained Transformer models on individual test examples. We find that Quoref, HellaSwag, and MC-TACO are best suited for distinguishing among state-of-the-art models. We also observe span selection task format, which is used for QA datasets like QAMR or SQuAD2.0, is effective in differentiating between strong and weak models.
  arXiv  Detail & Related papers  (2021-06-01T22:33:53Z)
• Ensemble Distillation for Robust Model Fusion in Federated Learning [72.61259487233214]
  Federated Learning (FL) is a machine learning setting where many devices collaboratively train a machine learning model. In most of the current training schemes the central model is refined by averaging the parameters of the server model and the updated parameters from the client side. We propose ensemble distillation for model fusion, i.e. training the central classifier through unlabeled data on the outputs of the models from the clients.
  arXiv  Detail & Related papers  (2020-06-12T14:49:47Z)
• Global Attention based Graph Convolutional Neural Networks for Improved Materials Property Prediction [8.371766047183739]
  We develop a novel model, GATGNN, for predicting inorganic material properties based on graph neural networks. We show that our method is able to both outperform the previous models' predictions and provide insight into the crystallization of the material.
  arXiv  Detail & Related papers  (2020-03-11T07:43:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.