Transferring a molecular foundation model for polymer property
predictions
- URL: http://arxiv.org/abs/2310.16958v1
- Date: Wed, 25 Oct 2023 19:55:00 GMT
- Title: Transferring a molecular foundation model for polymer property
predictions
- Authors: Pei Zhang, Logan Kearney, Debsindhu Bhowmik, Zachary Fox, Amit K.
Naskar, John Gounley
- Abstract summary: Self-supervised pretraining of transformer models requires large-scale datasets.
We show that using transformers pretrained on small molecules and fine-tuned on polymer properties achieve comparable accuracy to those trained on augmented polymer datasets.
- Score: 3.067983186439152
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Transformer-based large language models have remarkable potential to
accelerate design optimization for applications such as drug development and
materials discovery. Self-supervised pretraining of transformer models requires
large-scale datasets, which are often sparsely populated in topical areas such
as polymer science. State-of-the-art approaches for polymers conduct data
augmentation to generate additional samples but unavoidably incurs extra
computational costs. In contrast, large-scale open-source datasets are
available for small molecules and provide a potential solution to data scarcity
through transfer learning. In this work, we show that using transformers
pretrained on small molecules and fine-tuned on polymer properties achieve
comparable accuracy to those trained on augmented polymer datasets for a series
of benchmark prediction tasks.
Related papers
- Transformers for molecular property prediction: Lessons learned from the past five years [0.0]
We analyze the currently available models and explore key questions that arise when training and fine-tuning a transformer model for MPP.
We address the challenges in comparing different models, emphasizing the need for standardized data splitting and robust statistical analysis.
arXiv Detail & Related papers (2024-04-05T09:05:37Z) - Implicit Geometry and Interaction Embeddings Improve Few-Shot Molecular
Property Prediction [53.06671763877109]
We develop molecular embeddings that encode complex molecular characteristics to improve the performance of few-shot molecular property prediction.
Our approach leverages large amounts of synthetic data, namely the results of molecular docking calculations.
On multiple molecular property prediction benchmarks, training from the embedding space substantially improves Multi-Task, MAML, and Prototypical Network few-shot learning performance.
arXiv Detail & Related papers (2023-02-04T01:32:40Z) - TransPolymer: a Transformer-based language model for polymer property
predictions [9.04563945965023]
TransPolymer is a Transformer-based language model for polymer property prediction.
Our proposed polymer tokenizer with chemical awareness enables learning representations from polymer sequences.
arXiv Detail & Related papers (2022-09-03T01:29:59Z) - Retrieval-based Controllable Molecule Generation [63.44583084888342]
We propose a new retrieval-based framework for controllable molecule generation.
We use a small set of molecules to steer the pre-trained generative model towards synthesizing molecules that satisfy the given design criteria.
Our approach is agnostic to the choice of generative models and requires no task-specific fine-tuning.
arXiv Detail & Related papers (2022-08-23T17:01:16Z) - Improving Molecular Representation Learning with Metric
Learning-enhanced Optimal Transport [49.237577649802034]
We develop a novel optimal transport-based algorithm termed MROT to enhance their generalization capability for molecular regression problems.
MROT significantly outperforms state-of-the-art models, showing promising potential in accelerating the discovery of new substances.
arXiv Detail & Related papers (2022-02-13T04:56:18Z) - Improving VAE based molecular representations for compound property
prediction [0.0]
We propose a simple method to improve chemical property prediction performance of machine learning models.
We show the relation between the performance of property prediction models and the distance between property prediction dataset and the larger unlabeled dataset.
arXiv Detail & Related papers (2022-01-13T12:57:11Z) - Geometric Transformer for End-to-End Molecule Properties Prediction [92.28929858529679]
We introduce a Transformer-based architecture for molecule property prediction, which is able to capture the geometry of the molecule.
We modify the classical positional encoder by an initial encoding of the molecule geometry, as well as a learned gated self-attention mechanism.
arXiv Detail & Related papers (2021-10-26T14:14:40Z) - BIGDML: Towards Exact Machine Learning Force Fields for Materials [55.944221055171276]
Machine-learning force fields (MLFF) should be accurate, computationally and data efficient, and applicable to molecules, materials, and interfaces thereof.
Here, we introduce the Bravais-Inspired Gradient-Domain Machine Learning approach and demonstrate its ability to construct reliable force fields using a training set with just 10-200 atoms.
arXiv Detail & Related papers (2021-06-08T10:14:57Z) - Accelerating the screening of amorphous polymer electrolytes by learning
to reduce random and systematic errors in molecular dynamics simulations [0.8061216572215162]
In this work, we aim to screen amorphous polymer electrolytes which are promising candidates for the next generation lithium-ion battery technology.
We demonstrate that a multi-task graph neural network can learn from a large amount of noisy, biased data and a small number of unbiased data.
We screen a space of 6247 polymer electrolytes, orders of magnitude larger than previous computational studies.
arXiv Detail & Related papers (2021-01-13T20:46:24Z) - Polymer Informatics: Current Status and Critical Next Steps [1.3238373064156097]
Surrogate models are trained on available polymer data for instant property prediction.
Data-driven strategies to tackle unique challenges resulting from the extraordinary chemical and physical diversity of polymers at small and large scales are being explored.
Methods to solve inverse problems, wherein polymer recommendations are made using advanced AI algorithms that meet application targets, are being investigated.
arXiv Detail & Related papers (2020-11-01T14:17:22Z) - Self-Supervised Graph Transformer on Large-Scale Molecular Data [73.3448373618865]
We propose a novel framework, GROVER, for molecular representation learning.
GROVER can learn rich structural and semantic information of molecules from enormous unlabelled molecular data.
We pre-train GROVER with 100 million parameters on 10 million unlabelled molecules -- the biggest GNN and the largest training dataset in molecular representation learning.
arXiv Detail & Related papers (2020-06-18T08:37:04Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.