Transformers in Protein: A Survey

• URL: http://arxiv.org/abs/2505.20098v2
• Date: Tue, 27 May 2025 10:44:08 GMT
• Title: Transformers in Protein: A Survey
• Authors: Xiaowen Ling, Zhiqiang Li, Yanbin Wang, Zhuhong You,
• Abstract summary: Transformer models have demonstrated unprecedented potential in addressing diverse challenges across protein research.<n>This review systematically covers critical domains, including protein structure prediction, function prediction, protein-protein interaction analysis, functional annotation, and drug discovery/target identification.<n>For each research domain, we outline its objectives and background, critically evaluate prior methods and their limitations, and highlight transformative contributions enabled by Transformer models.
• Score: 3.4460628622243448
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As protein informatics advances rapidly, the demand for enhanced predictive accuracy, structural analysis, and functional understanding has intensified. Transformer models, as powerful deep learning architectures, have demonstrated unprecedented potential in addressing diverse challenges across protein research. However, a comprehensive review of Transformer applications in this field remains lacking. This paper bridges this gap by surveying over 100 studies, offering an in-depth analysis of practical implementations and research progress of Transformers in protein-related tasks. Our review systematically covers critical domains, including protein structure prediction, function prediction, protein-protein interaction analysis, functional annotation, and drug discovery/target identification. To contextualize these advancements across various protein domains, we adopt a domain-oriented classification system. We first introduce foundational concepts: the Transformer architecture and attention mechanisms, categorize Transformer variants tailored for protein science, and summarize essential protein knowledge. For each research domain, we outline its objectives and background, critically evaluate prior methods and their limitations, and highlight transformative contributions enabled by Transformer models. We also curate and summarize pivotal datasets and open-source code resources to facilitate reproducibility and benchmarking. Finally, we discuss persistent challenges in applying Transformers to protein informatics and propose future research directions. This review aims to provide a consolidated foundation for the synergistic integration of Transformer and protein informatics, fostering further innovation and expanded applications in the field.

Related papers

• A Comprehensive Review of Transformer-based language models for Protein Sequence Analysis and Design [0.9600277231719874]
  The impact of Transformer-based language models has been unprecedented in Natural Language Processing (NLP)<n>The success of such models has also led to their adoption in other fields including bioinformatics.<n>In this review, we have discussed and analysed a significant number of works pertaining to such applications.
  arXiv  Detail & Related papers  (2025-07-18T04:20:33Z)
• Advanced Deep Learning Methods for Protein Structure Prediction and Design [28.575821996185024]
  We comprehensively explore advanced deep learning methods applied to protein structure prediction and design.<n>The text analyses key components including structure generation, evaluation metrics, multiple sequence alignment processing, and network architecture.<n> Strategies for enhancing prediction accuracy and integrating deep learning techniques with experimental validation are thoroughly explored.
  arXiv  Detail & Related papers  (2025-03-14T21:28:29Z)
• Protein Large Language Models: A Comprehensive Survey [71.65899614084853]
  Protein-specific large language models (Protein LLMs) are revolutionizing protein science by enabling more efficient protein structure prediction, function annotation, and design.<n>This work provides the first comprehensive overview of Protein LLMs, covering their architectures, training datasets, evaluation metrics, and diverse applications.
  arXiv  Detail & Related papers  (2025-02-21T19:22:10Z)
• Strategic priorities for transformative progress in advancing biology with proteomics and artificial intelligence [54.14779179869007]
  We highlight key areas where AI is driving innovation, from data analysis to new biological insights.<n>These include developing an AI-friendly ecosystem for data generation, sharing, and analysis.
  arXiv  Detail & Related papers  (2025-02-21T13:20:33Z)
• Computational Protein Science in the Era of Large Language Models (LLMs) [54.35488233989787]
  Computational protein science is dedicated to revealing knowledge and developing applications within the protein sequence-structure-function paradigm.<n>Recently, Language Models (pLMs) have emerged as a milestone in AI due to their unprecedented language processing & generalization capability.
  arXiv  Detail & Related papers  (2025-01-17T16:21:18Z)
• OneProt: Towards Multi-Modal Protein Foundation Models [5.440531199006399]
  We introduce OneProt, a multi-modal AI for proteins that integrates structural, sequence, text, and binding site data.<n>Using the ImageBind framework, OneProt aligns the latent spaces of protein modality encoders in a lightweight fine-tuning scheme.<n>This work expands the horizons of multi-modal protein models, paving the way for transformative applications in drug discovery, biocatalytic reaction planning, and protein engineering.
  arXiv  Detail & Related papers  (2024-11-07T16:54:54Z)
• ProteinBench: A Holistic Evaluation of Protein Foundation Models [53.59325047872512]
  We introduce ProteinBench, a holistic evaluation framework for protein foundation models. Our approach consists of three key components: (i) A taxonomic classification of tasks that broadly encompass the main challenges in the protein domain, based on the relationships between different protein modalities; (ii) A multi-metric evaluation approach that assesses performance across four key dimensions: quality, novelty, diversity, and robustness; and (iii) In-depth analyses from various user objectives, providing a holistic view of model performance.
  arXiv  Detail & Related papers  (2024-09-10T06:52:33Z)
• Endowing Protein Language Models with Structural Knowledge [5.587293092389789]
  We introduce a novel framework that enhances protein language models by integrating protein structural data. The refined model, termed Protein Structure Transformer (PST), is further pretrained on a small protein structure database. PST consistently outperforms the state-of-the-art foundation model for protein sequences, ESM-2, setting a new benchmark in protein function prediction.
  arXiv  Detail & Related papers  (2024-01-26T12:47:54Z)
• Efficiently Predicting Protein Stability Changes Upon Single-point Mutation with Large Language Models [51.57843608615827]
  The ability to precisely predict protein thermostability is pivotal for various subfields and applications in biochemistry. We introduce an ESM-assisted efficient approach that integrates protein sequence and structural features to predict the thermostability changes in protein upon single-point mutations.
  arXiv  Detail & Related papers  (2023-12-07T03:25:49Z)
• PersGNN: Applying Topological Data Analysis and Geometric Deep Learning to Structure-Based Protein Function Prediction [0.07340017786387766]
  In this work, we isolate protein structure to make functional annotations for proteins in the Protein Data Bank. We present PersGNN - an end-to-end trainable deep learning model that combines graph representation learning with topological data analysis.
  arXiv  Detail & Related papers  (2020-10-30T02:24:35Z)
• BERTology Meets Biology: Interpreting Attention in Protein Language Models [124.8966298974842]
  We demonstrate methods for analyzing protein Transformer models through the lens of attention. We show that attention captures the folding structure of proteins, connecting amino acids that are far apart in the underlying sequence, but spatially close in the three-dimensional structure. We also present a three-dimensional visualization of the interaction between attention and protein structure.
  arXiv  Detail & Related papers  (2020-06-26T21:50:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.