Beyond ESM2: Graph-Enhanced Protein Sequence Modeling with Efficient Clustering

• URL: http://arxiv.org/abs/2404.15805v1
• Date: Wed, 24 Apr 2024 11:09:43 GMT
• Title: Beyond ESM2: Graph-Enhanced Protein Sequence Modeling with Efficient Clustering
• Authors: Shujian Jiao, Bingxuan Li, Lei Wang, Xiaojin Zhang, Wei Chen, Jiajie Peng, Zhongyu Wei,
• Abstract summary: Proteins are essential to life's processes, underpinning evolution and diversity. Advances in sequencing technology have revealed millions of proteins, underscoring the need for sophisticated pre-trained protein models for biological analysis and AI development. Facebook's ESM2, the most advanced protein language model to date, leverages a masked prediction task for unsupervised learning, crafting amino acid representations with notable biochemical accuracy. Yet, it lacks in delivering functional protein insights, signaling an opportunity for enhancing representation quality. This study addresses this gap by incorporating protein family classification into ESM2's training, while a contextual prediction task fine-tunes local
• Score: 24.415612744612773
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Proteins are essential to life's processes, underpinning evolution and diversity. Advances in sequencing technology have revealed millions of proteins, underscoring the need for sophisticated pre-trained protein models for biological analysis and AI development. Facebook's ESM2, the most advanced protein language model to date, leverages a masked prediction task for unsupervised learning, crafting amino acid representations with notable biochemical accuracy. Yet, it lacks in delivering functional protein insights, signaling an opportunity for enhancing representation quality.Our study addresses this gap by incorporating protein family classification into ESM2's training.This approach, augmented with Community Propagation-Based Clustering Algorithm, improves global protein representations, while a contextual prediction task fine-tunes local amino acid accuracy. Significantly, our model achieved state-of-the-art results in several downstream experiments, demonstrating the power of combining global and local methodologies to substantially boost protein representation quality.

Related papers

• Boosting Protein Language Models with Negative Sample Mining [20.721167029530168]
  We introduce a pioneering methodology for boosting large language models in the domain of protein representation learning. Our primary contribution lies in the refinement process for correlating the over-reliance on co-evolution knowledge. By capitalizing on this novel approach, our technique steers the training of transformer-based models within the attention score space.
  arXiv  Detail & Related papers  (2024-05-28T07:24:20Z)
• Diffusion on language model embeddings for protein sequence generation [0.5442686600296733]
  We introduce DiMA, a model that leverages continuous diffusion to generate amino acid sequences. We quantitatively illustrate the impact of the design choices that lead to its superior performance. Our approach consistently produces novel, diverse protein sequences that accurately reflect the inherent structural and functional diversity of the protein space.
  arXiv  Detail & Related papers  (2024-03-06T14:15:20Z)
• Enhancing Protein Predictive Models via Proteins Data Augmentation: A Benchmark and New Directions [58.819567030843025]
  This paper extends data augmentation techniques previously used for images and texts to proteins and then benchmarks these techniques on a variety of protein-related tasks. We propose two novel semantic-level protein augmentation methods, namely Integrated Gradients Substitution and Back Translation Substitution. Finally, we integrate extended and proposed augmentations into an augmentation pool and propose a simple but effective framework, namely Automated Protein Augmentation (APA)
  arXiv  Detail & Related papers  (2024-03-01T07:58:29Z)
• Efficiently Predicting Mutational Effect on Homologous Proteins by Evolution Encoding [7.067145619709089]
  EvolMPNN is an efficient model to learn evolution-aware protein embeddings. Our model shows up to 6.4% better than state-of-the-art methods and attains 36X inference speedup.
  arXiv  Detail & Related papers  (2024-02-20T23:06:21Z)
• xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein [76.18058946124111]
  We propose a unified protein language model, xTrimoPGLM, to address protein understanding and generation tasks simultaneously. xTrimoPGLM significantly outperforms other advanced baselines in 18 protein understanding benchmarks across four categories. It can also generate de novo protein sequences following the principles of natural ones, and can perform programmable generation after supervised fine-tuning.
  arXiv  Detail & Related papers  (2024-01-11T15:03:17Z)
• Efficiently Predicting Protein Stability Changes Upon Single-point Mutation with Large Language Models [51.57843608615827]
  The ability to precisely predict protein thermostability is pivotal for various subfields and applications in biochemistry. We introduce an ESM-assisted efficient approach that integrates protein sequence and structural features to predict the thermostability changes in protein upon single-point mutations.
  arXiv  Detail & Related papers  (2023-12-07T03:25:49Z)
• Plug & Play Directed Evolution of Proteins with Gradient-based Discrete MCMC [1.0499611180329804]
  A long-standing goal of machine-learning-based protein engineering is to accelerate the discovery of novel mutations. We introduce a sampling framework for evolving proteins in silico that supports mixing and matching a variety of unsupervised models. By composing these models, we aim to improve our ability to evaluate unseen mutations and constrain search to regions of sequence space likely to contain functional proteins.
  arXiv  Detail & Related papers  (2022-12-20T00:26:23Z)
• Integration of Pre-trained Protein Language Models into Geometric Deep Learning Networks [68.90692290665648]
  We integrate knowledge learned by protein language models into several state-of-the-art geometric networks. Our findings show an overall improvement of 20% over baselines. Strong evidence indicates that the incorporation of protein language models' knowledge enhances geometric networks' capacity by a significant margin.
  arXiv  Detail & Related papers  (2022-12-07T04:04:04Z)
• Learning Geometrically Disentangled Representations of Protein Folding Simulations [72.03095377508856]
  This work focuses on learning a generative neural network on a structural ensemble of a drug-target protein. Model tasks involve characterizing the distinct structural fluctuations of the protein bound to various drug molecules. Results show that our geometric learning-based method enjoys both accuracy and efficiency for generating complex structural variations.
  arXiv  Detail & Related papers  (2022-05-20T19:38:00Z)
• Structure-aware Protein Self-supervised Learning [50.04673179816619]
  We propose a novel structure-aware protein self-supervised learning method to capture structural information of proteins. In particular, a well-designed graph neural network (GNN) model is pretrained to preserve the protein structural information. We identify the relation between the sequential information in the protein language model and the structural information in the specially designed GNN model via a novel pseudo bi-level optimization scheme.
  arXiv  Detail & Related papers  (2022-04-06T02:18:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.