Beyond ESM2: Graph-Enhanced Protein Sequence Modeling with Efficient Clustering

• URL: http://arxiv.org/abs/2404.15805v1
• Date: Wed, 24 Apr 2024 11:09:43 GMT
• Title: Beyond ESM2: Graph-Enhanced Protein Sequence Modeling with Efficient Clustering
• Authors: Shujian Jiao, Bingxuan Li, Lei Wang, Xiaojin Zhang, Wei Chen, Jiajie Peng, Zhongyu Wei,
• Abstract summary: Proteins are essential to life's processes, underpinning evolution and diversity. Advances in sequencing technology have revealed millions of proteins, underscoring the need for sophisticated pre-trained protein models for biological analysis and AI development. Facebook's ESM2, the most advanced protein language model to date, leverages a masked prediction task for unsupervised learning, crafting amino acid representations with notable biochemical accuracy. Yet, it lacks in delivering functional protein insights, signaling an opportunity for enhancing representation quality. This study addresses this gap by incorporating protein family classification into ESM2's training, while a contextual prediction task fine-tunes local
• Score: 24.415612744612773
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Proteins are essential to life's processes, underpinning evolution and diversity. Advances in sequencing technology have revealed millions of proteins, underscoring the need for sophisticated pre-trained protein models for biological analysis and AI development. Facebook's ESM2, the most advanced protein language model to date, leverages a masked prediction task for unsupervised learning, crafting amino acid representations with notable biochemical accuracy. Yet, it lacks in delivering functional protein insights, signaling an opportunity for enhancing representation quality.Our study addresses this gap by incorporating protein family classification into ESM2's training.This approach, augmented with Community Propagation-Based Clustering Algorithm, improves global protein representations, while a contextual prediction task fine-tunes local amino acid accuracy. Significantly, our model achieved state-of-the-art results in several downstream experiments, demonstrating the power of combining global and local methodologies to substantially boost protein representation quality.

Related papers

• SFM-Protein: Integrative Co-evolutionary Pre-training for Advanced Protein Sequence Representation [97.99658944212675]
  We introduce a novel pre-training strategy for protein foundation models. It emphasizes the interactions among amino acid residues to enhance the extraction of both short-range and long-range co-evolutionary features. Trained on a large-scale protein sequence dataset, our model demonstrates superior generalization ability.
  arXiv  Detail & Related papers  (2024-10-31T15:22:03Z)
• EMOCPD: Efficient Attention-based Models for Computational Protein Design Using Amino Acid Microenvironment [8.661662081290265]
  We develop Efficient attention-based Models for Computational Protein Design using amino acid microenvironment (EMOCPD) It aims to predict the category of each amino acid in a protein by analyzing the three-dimensional atomic environment surrounding the amino acids, and optimize the protein based on the predicted high-probability potential amino acid categories. It achieves over 80% accuracy on the training set and 68.33% and 62.32% accuracy on two independent test sets, respectively.
  arXiv  Detail & Related papers  (2024-10-28T14:31:18Z)
• Protein-Mamba: Biological Mamba Models for Protein Function Prediction [18.642511763423048]
  Protein-Mamba is a novel two-stage model that leverages both self-supervised learning and fine-tuning to improve protein function prediction. Our experiments demonstrate that Protein-Mamba achieves competitive performance, compared with a couple of state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-09-22T22:51:56Z)
• Advanced atom-level representations for protein flexibility prediction utilizing graph neural networks [0.0]
  We propose graph neural networks (GNNs) to learn protein representations at the atomic level and predict B-factors from protein 3D structures. The Meta-GNN model achieves a correlation coefficient of 0.71 on a large and diverse test set of over 4k proteins.
  arXiv  Detail & Related papers  (2024-08-22T16:15:13Z)
• GOProteinGNN: Leveraging Protein Knowledge Graphs for Protein Representation Learning [27.192150057715835]
  GOProteinGNN is a novel architecture that enhances protein language models by integrating protein knowledge graph information. Our approach allows for the integration of information at both the individual amino acid level and the entire protein level, enabling a comprehensive and effective learning process.
  arXiv  Detail & Related papers  (2024-07-31T17:54:22Z)
• Boosting Protein Language Models with Negative Sample Mining [20.721167029530168]
  We introduce a pioneering methodology for boosting large language models in the domain of protein representation learning. Our primary contribution lies in the refinement process for correlating the over-reliance on co-evolution knowledge. By capitalizing on this novel approach, our technique steers the training of transformer-based models within the attention score space.
  arXiv  Detail & Related papers  (2024-05-28T07:24:20Z)
• Enhancing Protein Predictive Models via Proteins Data Augmentation: A Benchmark and New Directions [58.819567030843025]
  This paper extends data augmentation techniques previously used for images and texts to proteins and then benchmarks these techniques on a variety of protein-related tasks. We propose two novel semantic-level protein augmentation methods, namely Integrated Gradients Substitution and Back Translation Substitution. Finally, we integrate extended and proposed augmentations into an augmentation pool and propose a simple but effective framework, namely Automated Protein Augmentation (APA)
  arXiv  Detail & Related papers  (2024-03-01T07:58:29Z)
• xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein [76.18058946124111]
  We propose a unified protein language model, xTrimoPGLM, to address protein understanding and generation tasks simultaneously. xTrimoPGLM significantly outperforms other advanced baselines in 18 protein understanding benchmarks across four categories. It can also generate de novo protein sequences following the principles of natural ones, and can perform programmable generation after supervised fine-tuning.
  arXiv  Detail & Related papers  (2024-01-11T15:03:17Z)
• Efficiently Predicting Protein Stability Changes Upon Single-point Mutation with Large Language Models [51.57843608615827]
  The ability to precisely predict protein thermostability is pivotal for various subfields and applications in biochemistry. We introduce an ESM-assisted efficient approach that integrates protein sequence and structural features to predict the thermostability changes in protein upon single-point mutations.
  arXiv  Detail & Related papers  (2023-12-07T03:25:49Z)
• Integration of Pre-trained Protein Language Models into Geometric Deep Learning Networks [68.90692290665648]
  We integrate knowledge learned by protein language models into several state-of-the-art geometric networks. Our findings show an overall improvement of 20% over baselines. Strong evidence indicates that the incorporation of protein language models' knowledge enhances geometric networks' capacity by a significant margin.
  arXiv  Detail & Related papers  (2022-12-07T04:04:04Z)
• Learning Geometrically Disentangled Representations of Protein Folding Simulations [72.03095377508856]
  This work focuses on learning a generative neural network on a structural ensemble of a drug-target protein. Model tasks involve characterizing the distinct structural fluctuations of the protein bound to various drug molecules. Results show that our geometric learning-based method enjoys both accuracy and efficiency for generating complex structural variations.
  arXiv  Detail & Related papers  (2022-05-20T19:38:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.