Structure-Informed Protein Language Model

• URL: http://arxiv.org/abs/2402.05856v1
• Date: Wed, 7 Feb 2024 09:32:35 GMT
• Title: Structure-Informed Protein Language Model
• Authors: Zuobai Zhang, Jiarui Lu, Vijil Chenthamarakshan, Aur\'elie Lozano, Payel Das, Jian Tang
• Abstract summary: We introduce the integration of remote homology detection to distill structural information into protein language models. We evaluate the impact of this structure-informed training on downstream protein function prediction tasks.
• Score: 38.019425619750265
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Protein language models are a powerful tool for learning protein representations through pre-training on vast protein sequence datasets. However, traditional protein language models lack explicit structural supervision, despite its relevance to protein function. To address this issue, we introduce the integration of remote homology detection to distill structural information into protein language models without requiring explicit protein structures as input. We evaluate the impact of this structure-informed training on downstream protein function prediction tasks. Experimental results reveal consistent improvements in function annotation accuracy for EC number and GO term prediction. Performance on mutant datasets, however, varies based on the relationship between targeted properties and protein structures. This underscores the importance of considering this relationship when applying structure-aware training to protein function prediction tasks. Code and model weights are available at https://github.com/DeepGraphLearning/esm-s.

Related papers

• CPE-Pro: A Structure-Sensitive Deep Learning Method for Protein Representation and Origin Evaluation [7.161099050722313]
  We develop a structure-sensitive supervised deep learning model, Crystal vs Predicted Evaluator for Protein Structure (CPE-Pro) CPE-Pro learns the structural information of proteins and captures inter-structural differences to achieve accurate traceability on four data classes. We utilize Foldseek to encode protein structures into "structure-sequences" and trained a protein Structural Sequence Language Model, SSLM.
  arXiv  Detail & Related papers  (2024-10-21T02:21:56Z)
• NaNa and MiGu: Semantic Data Augmentation Techniques to Enhance Protein Classification in Graph Neural Networks [60.48306899271866]
  We propose novel semantic data augmentation methods to incorporate backbone chemical and side-chain biophysical information into protein classification tasks. Specifically, we leverage molecular biophysical, secondary structure, chemical bonds, andionic features of proteins to facilitate classification tasks.
  arXiv  Detail & Related papers  (2024-03-21T13:27:57Z)
• ProtLLM: An Interleaved Protein-Language LLM with Protein-as-Word Pre-Training [82.37346937497136]
  We propose a versatile cross-modal large language model (LLM) for both protein-centric and protein-language tasks. ProtLLM features a unique dynamic protein mounting mechanism, enabling it to handle complex inputs. By developing a specialized protein vocabulary, we equip the model with the capability to predict not just natural language but also proteins from a vast pool of candidates.
  arXiv  Detail & Related papers  (2024-02-28T01:29:55Z)
• Endowing Protein Language Models with Structural Knowledge [5.587293092389789]
  We introduce a novel framework that enhances protein language models by integrating protein structural data. The refined model, termed Protein Structure Transformer (PST), is further pretrained on a small protein structure database. PST consistently outperforms the state-of-the-art foundation model for protein sequences, ESM-2, setting a new benchmark in protein function prediction.
  arXiv  Detail & Related papers  (2024-01-26T12:47:54Z)
• Multi-level Protein Representation Learning for Blind Mutational Effect Prediction [5.207307163958806]
  This paper introduces a novel pre-training framework that cascades sequential and geometric analyzers for protein structures. It guides mutational directions toward desired traits by simulating natural selection on wild-type proteins. We assess the proposed approach using a public database and two new databases for a variety of variant effect prediction tasks.
  arXiv  Detail & Related papers  (2023-06-08T03:00:50Z)
• CCPL: Cross-modal Contrastive Protein Learning [47.095862120116976]
  We introduce a novel unsupervised protein structure representation pretraining method, cross-modal contrastive protein learning (CCPL) CCPL leverages a robust protein language model and uses unsupervised contrastive alignment to enhance structure learning. We evaluated our model across various benchmarks, demonstrating the framework's superiority.
  arXiv  Detail & Related papers  (2023-03-19T08:19:10Z)
• Structure-aware Protein Self-supervised Learning [50.04673179816619]
  We propose a novel structure-aware protein self-supervised learning method to capture structural information of proteins. In particular, a well-designed graph neural network (GNN) model is pretrained to preserve the protein structural information. We identify the relation between the sequential information in the protein language model and the structural information in the specially designed GNN model via a novel pseudo bi-level optimization scheme.
  arXiv  Detail & Related papers  (2022-04-06T02:18:41Z)
• Protein Representation Learning by Geometric Structure Pretraining [27.723095456631906]
  Existing approaches usually pretrain protein language models on a large number of unlabeled amino acid sequences. We first present a simple yet effective encoder to learn protein geometry features. Experimental results on both function prediction and fold classification tasks show that our proposed pretraining methods outperform or are on par with the state-of-the-art sequence-based methods using much less data.
  arXiv  Detail & Related papers  (2022-03-11T17:52:13Z)
• BERTology Meets Biology: Interpreting Attention in Protein Language Models [124.8966298974842]
  We demonstrate methods for analyzing protein Transformer models through the lens of attention. We show that attention captures the folding structure of proteins, connecting amino acids that are far apart in the underlying sequence, but spatially close in the three-dimensional structure. We also present a three-dimensional visualization of the interaction between attention and protein structure.
  arXiv  Detail & Related papers  (2020-06-26T21:50:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.