Materials Informatics Transformer: A Language Model for Interpretable
Materials Properties Prediction
- URL: http://arxiv.org/abs/2308.16259v2
- Date: Fri, 1 Sep 2023 12:40:29 GMT
- Title: Materials Informatics Transformer: A Language Model for Interpretable
Materials Properties Prediction
- Authors: Hongshuo Huang, Rishikesh Magar, Changwen Xu and Amir Barati Farimani
- Abstract summary: We introduce our model Materials Informatics Transformer (MatInFormer) for material property prediction.
Specifically, we introduce a novel approach that involves learning the grammar of crystallography through the tokenization of pertinent space group information.
- Score: 6.349503549199403
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Recently, the remarkable capabilities of large language models (LLMs) have
been illustrated across a variety of research domains such as natural language
processing, computer vision, and molecular modeling. We extend this paradigm by
utilizing LLMs for material property prediction by introducing our model
Materials Informatics Transformer (MatInFormer). Specifically, we introduce a
novel approach that involves learning the grammar of crystallography through
the tokenization of pertinent space group information. We further illustrate
the adaptability of MatInFormer by incorporating task-specific data pertaining
to Metal-Organic Frameworks (MOFs). Through attention visualization, we uncover
the key features that the model prioritizes during property prediction. The
effectiveness of our proposed model is empirically validated across 14 distinct
datasets, hereby underscoring its potential for high throughput screening
through accurate material property prediction.
Related papers
- MatText: Do Language Models Need More than Text & Scale for Materials Modeling? [5.561723952524538]
MatText is a suite of benchmarking tools and datasets designed to systematically evaluate the performance of language models in modeling materials.
MatText provides essential tools for training and benchmarking the performance of language models in the context of materials science.
arXiv Detail & Related papers (2024-06-25T05:45:07Z) - Multi-modal Auto-regressive Modeling via Visual Words [96.25078866446053]
We propose the concept of visual words, which maps the visual features to probability distributions over Large Multi-modal Models' vocabulary.
We further explore the distribution of visual features in the semantic space within LMM and the possibility of using text embeddings to represent visual information.
arXiv Detail & Related papers (2024-03-12T14:58:52Z) - Learning to Extract Structured Entities Using Language Models [52.281701191329]
Recent advances in machine learning have significantly impacted the field of information extraction.
We reformulate the task to be entity-centric, enabling the use of diverse metrics that can provide more insights.
We introduce a new model that harnesses the power of Language Models (LMs) for enhanced effectiveness and efficiency.
arXiv Detail & Related papers (2024-02-06T22:15:09Z) - Fine-Tuned Language Models Generate Stable Inorganic Materials as Text [57.01994216693825]
Fine-tuning large language models on text-encoded atomistic data is simple to implement yet reliable.
We show that our strongest model can generate materials predicted to be metastable at about twice the rate of CDVAE.
Because of text prompting's inherent flexibility, our models can simultaneously be used for unconditional generation of stable material.
arXiv Detail & Related papers (2024-02-06T20:35:28Z) - SINC: Self-Supervised In-Context Learning for Vision-Language Tasks [64.44336003123102]
We propose a framework to enable in-context learning in large language models.
A meta-model can learn on self-supervised prompts consisting of tailored demonstrations.
Experiments show that SINC outperforms gradient-based methods in various vision-language tasks.
arXiv Detail & Related papers (2023-07-15T08:33:08Z) - Investigating Masking-based Data Generation in Language Models [0.0]
A feature of BERT and models with similar architecture is the objective of masked language modeling.
Data augmentation is a data-driven technique widely used in machine learning.
Recent studies have utilized masked language model to generate artificially augmented data for NLP downstream tasks.
arXiv Detail & Related papers (2023-06-16T16:48:27Z) - Entity Aware Modelling: A Survey [22.32009539611539]
Recent machine learning advances have led to new state-of-the-art response prediction models.
Models built at a population level often lead to sub-optimal performance in many personalized prediction settings.
In personalized prediction, the goal is to incorporate inherent characteristics of different entities to improve prediction performance.
arXiv Detail & Related papers (2023-02-16T16:33:33Z) - Metadata Might Make Language Models Better [1.7100280218774935]
Using 19th-century newspapers as a case study, we compare different strategies for inserting temporal, political and geographical information into a Masked Language Model.
We find that showing relevant metadata to a language model has a beneficial impact and may even produce more robust and fairer models.
arXiv Detail & Related papers (2022-11-18T08:29:00Z) - Improving VAE based molecular representations for compound property
prediction [0.0]
We propose a simple method to improve chemical property prediction performance of machine learning models.
We show the relation between the performance of property prediction models and the distance between property prediction dataset and the larger unlabeled dataset.
arXiv Detail & Related papers (2022-01-13T12:57:11Z) - Model-agnostic multi-objective approach for the evolutionary discovery
of mathematical models [55.41644538483948]
In modern data science, it is more interesting to understand the properties of the model, which parts could be replaced to obtain better results.
We use multi-objective evolutionary optimization for composite data-driven model learning to obtain the algorithm's desired properties.
arXiv Detail & Related papers (2021-07-07T11:17:09Z) - Comparing Test Sets with Item Response Theory [53.755064720563]
We evaluate 29 datasets using predictions from 18 pretrained Transformer models on individual test examples.
We find that Quoref, HellaSwag, and MC-TACO are best suited for distinguishing among state-of-the-art models.
We also observe span selection task format, which is used for QA datasets like QAMR or SQuAD2.0, is effective in differentiating between strong and weak models.
arXiv Detail & Related papers (2021-06-01T22:33:53Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.