Crystal Transformer: Self-learning neural language model for Generative
and Tinkering Design of Materials
- URL: http://arxiv.org/abs/2204.11953v1
- Date: Mon, 25 Apr 2022 20:20:26 GMT
- Title: Crystal Transformer: Self-learning neural language model for Generative
and Tinkering Design of Materials
- Authors: Lai Wei, Qinyang Li, Yuqi Song, Stanislav Stefanov, Edirisuriya M. D.
Siriwardane, Fanglin Chen, Jianjun Hu
- Abstract summary: BLMM Crystal Transformer is a neural network based probabilistic generative model for generative and tinkering design of inorganic materials.
It can generate chemically valid materials compositions with as high as 89.7% charge neutrality and 84.8% balanced electronegativity.
A user-friendly web app has been developed for computational materials doping and can be accessed freely at urlwww.materialsatlas.org/blmtinker.
- Score: 4.813020904720316
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Self-supervised neural language models have recently achieved unprecedented
success, from natural language processing to learning the languages of
biological sequences and organic molecules. These models have demonstrated
superior performance in the generation, structure classification, and
functional predictions for proteins and molecules with learned representations.
However, most of the masking-based pre-trained language models are not designed
for generative design, and their black-box nature makes it difficult to
interpret their design logic. Here we propose BLMM Crystal Transformer, a
neural network based probabilistic generative model for generative and
tinkering design of inorganic materials. Our model is built on the blank
filling language model for text generation and has demonstrated unique
advantages in learning the "materials grammars" together with high-quality
generation, interpretability, and data efficiency. It can generate chemically
valid materials compositions with as high as 89.7\% charge neutrality and
84.8\% balanced electronegativity, which are more than 4 and 8 times higher
compared to a pseudo random sampling baseline. The probabilistic generation
process of BLMM allows it to recommend tinkering operations based on learned
materials chemistry and makes it useful for materials doping. Combined with the
TCSP crysal structure prediction algorithm, We have applied our model to
discover a set of new materials as validated using DFT calculations. Our work
thus brings the unsupervised transformer language models based generative
artificial intelligence to inorganic materials. A user-friendly web app has
been developed for computational materials doping and can be accessed freely at
\url{www.materialsatlas.org/blmtinker}.
Related papers
- Pre-trained Molecular Language Models with Random Functional Group Masking [54.900360309677794]
We propose a SMILES-based underlineem Molecular underlineem Language underlineem Model, which randomly masking SMILES subsequences corresponding to specific molecular atoms.
This technique aims to compel the model to better infer molecular structures and properties, thus enhancing its predictive capabilities.
arXiv Detail & Related papers (2024-11-03T01:56:15Z) - Fine-Tuned Language Models Generate Stable Inorganic Materials as Text [57.01994216693825]
Fine-tuning large language models on text-encoded atomistic data is simple to implement yet reliable.
We show that our strongest model can generate materials predicted to be metastable at about twice the rate of CDVAE.
Because of text prompting's inherent flexibility, our models can simultaneously be used for unconditional generation of stable material.
arXiv Detail & Related papers (2024-02-06T20:35:28Z) - In-Context Language Learning: Architectures and Algorithms [73.93205821154605]
We study ICL through the lens of a new family of model problems we term in context language learning (ICLL)
We evaluate a diverse set of neural sequence models on regular ICLL tasks.
arXiv Detail & Related papers (2024-01-23T18:59:21Z) - Data Distillation for Neural Network Potentials toward Foundational
Dataset [6.373914211316965]
generative models can swiftly propose promising materials for targeted applications.
However, the predicted properties of materials through the generative models often do not match with calculated properties through ab initio calculations.
This study utilized extended ensemble molecular dynamics (MD) to secure a broad range of liquid- and solid-phase configurations in one of the metallic systems, nickel.
We found that the NNP trained from the distilled data could predict different energy-minimized closed-pack crystal structures even though those structures were not explicitly part of the initial data.
arXiv Detail & Related papers (2023-11-09T14:41:45Z) - CodeGen2: Lessons for Training LLMs on Programming and Natural Languages [116.74407069443895]
We unify encoder and decoder-based models into a single prefix-LM.
For learning methods, we explore the claim of a "free lunch" hypothesis.
For data distributions, the effect of a mixture distribution and multi-epoch training of programming and natural languages on model performance is explored.
arXiv Detail & Related papers (2023-05-03T17:55:25Z) - An Adversarial Active Sampling-based Data Augmentation Framework for
Manufacturable Chip Design [55.62660894625669]
Lithography modeling is a crucial problem in chip design to ensure a chip design mask is manufacturable.
Recent developments in machine learning have provided alternative solutions in replacing the time-consuming lithography simulations with deep neural networks.
We propose a litho-aware data augmentation framework to resolve the dilemma of limited data and improve the machine learning model performance.
arXiv Detail & Related papers (2022-10-27T20:53:39Z) - Probabilistic Generative Transformer Language models for Generative
Design of Molecules [10.412989388092084]
Generative Molecular Transformer (GMTransformer) is a probabilistic neural network model for generative design of molecules.
Our model is built on the blank filling language model originally developed for text processing.
Our models achieve high novelty and Scaf compared to other baselines.
arXiv Detail & Related papers (2022-09-20T01:51:57Z) - Materials Transformers Language Models for Generative Materials Design:
a benchmark study [4.047301375093173]
We train seven modern transformer language models (GPT, GPT-2, GPT-Neo, GPT-J, BLMM, BART, and RoBERTa) using the expanded formulas from material deposited in the ICSD, OQMD, and Materials Projects databases.
Six different datasets with/out non-charge-neutral or balanced electronegativity samples are used to benchmark the performances.
Experiments showed that the causal language models based materials transformers can generate chemically valid materials compositions with as high as 97.54% to be charge neutral and 91.40% to be electronegativity balanced.
arXiv Detail & Related papers (2022-06-27T18:50:05Z) - Dependency-based Mixture Language Models [53.152011258252315]
We introduce the Dependency-based Mixture Language Models.
In detail, we first train neural language models with a novel dependency modeling objective.
We then formulate the next-token probability by mixing the previous dependency modeling probability distributions with self-attention.
arXiv Detail & Related papers (2022-03-19T06:28:30Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.