FoldToken: Learning Protein Language via Vector Quantization and Beyond

• URL: http://arxiv.org/abs/2403.09673v2
• Date: Tue, 19 Mar 2024 05:29:23 GMT
• Title: FoldToken: Learning Protein Language via Vector Quantization and Beyond
• Authors: Zhangyang Gao, Cheng Tan, Jue Wang, Yufei Huang, Lirong Wu, Stan Z. Li,
• Abstract summary: We introduce textbfFoldTokenizer to represent protein sequence-structure as discrete symbols. We refer to the learned symbols as textbfFoldToken, and the sequence of FoldTokens serves as a new protein language.
• Score: 56.19308144551836
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Is there a foreign language describing protein sequences and structures simultaneously? Protein structures, represented by continuous 3D points, have long posed a challenge due to the contrasting modeling paradigms of discrete sequences. We introduce \textbf{FoldTokenizer} to represent protein sequence-structure as discrete symbols. This innovative approach involves projecting residue types and structures into a discrete space, guided by a reconstruction loss for information preservation. We refer to the learned discrete symbols as \textbf{FoldToken}, and the sequence of FoldTokens serves as a new protein language, transforming the protein sequence-structure into a unified modality. We apply the created protein language on general backbone inpainting and antibody design tasks, building the first GPT-style model (\textbf{FoldGPT}) for sequence-structure co-generation with promising results. Key to our success is the substantial enhancement of the vector quantization module, Soft Conditional Vector Quantization (\textbf{SoftCVQ}).

Related papers

• FoldToken2: Learning compact, invariant and generative protein structure language [48.1647245005672]
  We propose FoldToken2 to transfer equivariant structures into discrete tokens, while maintaining the recoverability of the original structures. We evaluate FoldToken2 on the protein structure reconstruction task and show that it outperforms previous FoldToken1 by 20% in TMScore and 81% in RMSD. We believe that FoldToken2 will inspire further improvement in protein structure representation learning, structure alignment, and structure generation tasks.
  arXiv  Detail & Related papers  (2024-06-11T09:24:51Z)
• Learning the Language of Protein Structure [8.364087723533537]
  We introduce an approach using a vector-quantized autoencoder that effectively tokenizes protein structures into discrete representations. To demonstrate the efficacy of our learned representations, we show that a simple GPT model trained on our codebooks can generate novel, diverse, and designable protein structures.
  arXiv  Detail & Related papers  (2024-05-24T16:03:47Z)
• Diffusion Language Models Are Versatile Protein Learners [75.98083311705182]
  This paper introduces diffusion protein language model (DPLM), a versatile protein language model that demonstrates strong generative and predictive capabilities for protein sequences. We first pre-train scalable DPLMs from evolutionary-scale protein sequences within a generative self-supervised discrete diffusion probabilistic framework. After pre-training, DPLM exhibits the ability to generate structurally plausible, novel, and diverse protein sequences for unconditional generation.
  arXiv  Detail & Related papers  (2024-02-28T18:57:56Z)
• Endowing Protein Language Models with Structural Knowledge [5.587293092389789]
  We introduce a novel framework that enhances protein language models by integrating protein structural data. The refined model, termed Protein Structure Transformer (PST), is further pretrained on a small protein structure database. PST consistently outperforms the state-of-the-art foundation model for protein sequences, ESM-2, setting a new benchmark in protein function prediction.
  arXiv  Detail & Related papers  (2024-01-26T12:47:54Z)
• xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein [76.18058946124111]
  We propose a unified protein language model, xTrimoPGLM, to address protein understanding and generation tasks simultaneously. xTrimoPGLM significantly outperforms other advanced baselines in 18 protein understanding benchmarks across four categories. It can also generate de novo protein sequences following the principles of natural ones, and can perform programmable generation after supervised fine-tuning.
  arXiv  Detail & Related papers  (2024-01-11T15:03:17Z)
• Protein Sequence and Structure Co-Design with Equivariant Translation [19.816174223173494]
  Existing approaches generate both protein sequence and structure using either autoregressive models or diffusion models. We propose a new approach capable of protein sequence and structure co-design, which iteratively translates both protein sequence and structure into the desired state. Our model consists of a trigonometry-aware encoder that reasons geometrical constraints and interactions from context features. All protein amino acids are updated in one shot in each translation step, which significantly accelerates the inference process.
  arXiv  Detail & Related papers  (2022-10-17T06:00:12Z)
• Benchmarking deep generative models for diverse antibody sequence design [18.515971640245997]
  Deep generative models that learn from sequences alone or from sequences and structures jointly have shown impressive performance on this task. We consider three recently proposed deep generative frameworks for protein design: (AR) the sequence-based autoregressive generative model, (GVP) the precise structure-based graph neural network, and Fold2Seq that leverages a fuzzy and scale-free representation of a three-dimensional fold. We benchmark these models on the task of computational design of antibody sequences, which demand designing sequences with high diversity for functional implication.
  arXiv  Detail & Related papers  (2021-11-12T16:23:32Z)
• Fold2Seq: A Joint Sequence(1D)-Fold(3D) Embedding-based Generative Model for Protein Design [70.27706384570723]
  We propose Fold2Seq, a novel framework for designing protein sequences conditioned on a specific target fold. We show improved or comparable performance of Fold2Seq in terms of speed, coverage, and reliability for sequence design. The unique advantages of fold-based Fold2Seq, in comparison to a structure-based deep model and RosettaDesign, become more evident on three additional real-world challenges.
  arXiv  Detail & Related papers  (2021-06-24T14:34:24Z)
• BERTology Meets Biology: Interpreting Attention in Protein Language Models [124.8966298974842]
  We demonstrate methods for analyzing protein Transformer models through the lens of attention. We show that attention captures the folding structure of proteins, connecting amino acids that are far apart in the underlying sequence, but spatially close in the three-dimensional structure. We also present a three-dimensional visualization of the interaction between attention and protein structure.
  arXiv  Detail & Related papers  (2020-06-26T21:50:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.