Diffusion Language Models Are Versatile Protein Learners
- URL: http://arxiv.org/abs/2402.18567v1
- Date: Wed, 28 Feb 2024 18:57:56 GMT
- Title: Diffusion Language Models Are Versatile Protein Learners
- Authors: Xinyou Wang, Zaixiang Zheng, Fei Ye, Dongyu Xue, Shujian Huang,
Quanquan Gu
- Abstract summary: diffusion protein language model (DPLM) is a versatile protein language model that demonstrates strong generative and predictive capabilities for protein sequences.
We first pre-train scalable DPLMs from evolutionary-scale protein sequences within a generative self-supervised discrete diffusion probabilistic framework.
After pre-training, DPLM exhibits the ability to generate structurally plausible, novel, and diverse protein sequences for unconditional generation.
- Score: 80.51049288791717
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: This paper introduces diffusion protein language model (DPLM), a versatile
protein language model that demonstrates strong generative and predictive
capabilities for protein sequences. We first pre-train scalable DPLMs from
evolutionary-scale protein sequences within a generative self-supervised
discrete diffusion probabilistic framework, which generalizes language modeling
for proteins in a principled way. After pre-training, DPLM exhibits the ability
to generate structurally plausible, novel, and diverse protein sequences for
unconditional generation. We further demonstrate the proposed diffusion
generative pre-training makes DPLM possess a better understanding of proteins,
making it a superior representation learner, which can be fine-tuned for
various predictive tasks, comparing favorably to ESM2 (Lin et al., 2022).
Moreover, DPLM can be tailored for various needs, which showcases its prowess
of conditional generation in several ways: (1) conditioning on partial peptide
sequences, e.g., generating scaffolds for functional motifs with high success
rate; (2) incorporating other modalities as conditioner, e.g.,
structure-conditioned generation for inverse folding; and (3) steering sequence
generation towards desired properties, e.g., satisfying specified secondary
structures, through a plug-and-play classifier guidance.
Related papers
- Sequence-Augmented SE(3)-Flow Matching For Conditional Protein Backbone Generation [55.93511121486321]
We introduce FoldFlow-2, a novel sequence-conditioned flow matching model for protein structure generation.
We train FoldFlow-2 at scale on a new dataset that is an order of magnitude larger than PDB datasets of prior works.
We empirically observe that FoldFlow-2 outperforms previous state-of-the-art protein structure-based generative models.
arXiv Detail & Related papers (2024-05-30T17:53:50Z) - Diffusion on language model embeddings for protein sequence generation [0.5442686600296733]
We introduce DiMA, a model that leverages continuous diffusion to generate amino acid sequences.
We quantitatively illustrate the impact of the design choices that lead to its superior performance.
Our approach consistently produces novel, diverse protein sequences that accurately reflect the inherent structural and functional diversity of the protein space.
arXiv Detail & Related papers (2024-03-06T14:15:20Z) - xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering
the Language of Protein [76.18058946124111]
We propose a unified protein language model, xTrimoPGLM, to address protein understanding and generation tasks simultaneously.
xTrimoPGLM significantly outperforms other advanced baselines in 18 protein understanding benchmarks across four categories.
It can also generate de novo protein sequences following the principles of natural ones, and can perform programmable generation after supervised fine-tuning.
arXiv Detail & Related papers (2024-01-11T15:03:17Z) - Protein Design with Guided Discrete Diffusion [67.06148688398677]
A popular approach to protein design is to combine a generative model with a discriminative model for conditional sampling.
We propose diffusioN Optimized Sampling (NOS), a guidance method for discrete diffusion models.
NOS makes it possible to perform design directly in sequence space, circumventing significant limitations of structure-based methods.
arXiv Detail & Related papers (2023-05-31T16:31:24Z) - EigenFold: Generative Protein Structure Prediction with Diffusion Models [10.24107243529341]
EigenFold is a diffusion generative modeling framework for sampling a distribution of structures from a given protein sequence.
On recent CAMEO targets, EigenFold achieves a median TMScore of 0.84, while providing a more comprehensive picture of model uncertainty.
arXiv Detail & Related papers (2023-04-05T02:46:13Z) - Structure-informed Language Models Are Protein Designers [69.70134899296912]
We present LM-Design, a generic approach to reprogramming sequence-based protein language models (pLMs)
We conduct a structural surgery on pLMs, where a lightweight structural adapter is implanted into pLMs and endows it with structural awareness.
Experiments show that our approach outperforms the state-of-the-art methods by a large margin.
arXiv Detail & Related papers (2023-02-03T10:49:52Z) - Protein Sequence and Structure Co-Design with Equivariant Translation [19.816174223173494]
Existing approaches generate both protein sequence and structure using either autoregressive models or diffusion models.
We propose a new approach capable of protein sequence and structure co-design, which iteratively translates both protein sequence and structure into the desired state.
Our model consists of a trigonometry-aware encoder that reasons geometrical constraints and interactions from context features.
All protein amino acids are updated in one shot in each translation step, which significantly accelerates the inference process.
arXiv Detail & Related papers (2022-10-17T06:00:12Z) - Protein Structure and Sequence Generation with Equivariant Denoising
Diffusion Probabilistic Models [3.5450828190071646]
An important task in bioengineering is designing proteins with specific 3D structures and chemical properties which enable targeted functions.
We introduce a generative model of both protein structure and sequence that can operate at significantly larger scales than previous molecular generative modeling approaches.
arXiv Detail & Related papers (2022-05-26T16:10:09Z) - EBM-Fold: Fully-Differentiable Protein Folding Powered by Energy-based
Models [53.17320541056843]
We propose a fully-differentiable approach for protein structure optimization, guided by a data-driven generative network.
Our EBM-Fold approach can efficiently produce high-quality decoys, compared against traditional Rosetta-based structure optimization routines.
arXiv Detail & Related papers (2021-05-11T03:40:29Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.