A Comprehensive Review of Transformer-based language models for Protein Sequence Analysis and Design

• URL: http://arxiv.org/abs/2507.13646v1
• Date: Fri, 18 Jul 2025 04:20:33 GMT
• Title: A Comprehensive Review of Transformer-based language models for Protein Sequence Analysis and Design
• Authors: Nimisha Ghosh, Daniele Santoni, Debaleena Nawn, Eleonora Ottaviani, Giovanni Felici,
• Abstract summary: The impact of Transformer-based language models has been unprecedented in Natural Language Processing (NLP)<n>The success of such models has also led to their adoption in other fields including bioinformatics.<n>In this review, we have discussed and analysed a significant number of works pertaining to such applications.
• Score: 0.9600277231719874
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The impact of Transformer-based language models has been unprecedented in Natural Language Processing (NLP). The success of such models has also led to their adoption in other fields including bioinformatics. Taking this into account, this paper discusses recent advances in Transformer-based models for protein sequence analysis and design. In this review, we have discussed and analysed a significant number of works pertaining to such applications. These applications encompass gene ontology, functional and structural protein identification, generation of de novo proteins and binding of proteins. We attempt to shed light on the strength and weaknesses of the discussed works to provide a comprehensive insight to readers. Finally, we highlight shortcomings in existing research and explore potential avenues for future developments. We believe that this review will help researchers working in this field to have an overall idea of the state of the art in this field, and to orient their future studies.

Related papers

• Transformers in Protein: A Survey [3.4460628622243448]
  Transformer models have demonstrated unprecedented potential in addressing diverse challenges across protein research.<n>This review systematically covers critical domains, including protein structure prediction, function prediction, protein-protein interaction analysis, functional annotation, and drug discovery/target identification.<n>For each research domain, we outline its objectives and background, critically evaluate prior methods and their limitations, and highlight transformative contributions enabled by Transformer models.
  arXiv  Detail & Related papers  (2025-05-26T15:08:18Z)
• Advanced Deep Learning Methods for Protein Structure Prediction and Design [28.575821996185024]
  We comprehensively explore advanced deep learning methods applied to protein structure prediction and design.<n>The text analyses key components including structure generation, evaluation metrics, multiple sequence alignment processing, and network architecture.<n> Strategies for enhancing prediction accuracy and integrating deep learning techniques with experimental validation are thoroughly explored.
  arXiv  Detail & Related papers  (2025-03-14T21:28:29Z)
• Protein Large Language Models: A Comprehensive Survey [71.65899614084853]
  Protein-specific large language models (Protein LLMs) are revolutionizing protein science by enabling more efficient protein structure prediction, function annotation, and design.<n>This work provides the first comprehensive overview of Protein LLMs, covering their architectures, training datasets, evaluation metrics, and diverse applications.
  arXiv  Detail & Related papers  (2025-02-21T19:22:10Z)
• GENERator: A Long-Context Generative Genomic Foundation Model [66.46537421135996]
  We present GENERator, a generative genomic foundation model featuring a context length of 98k base pairs (bp) and 1.2B parameters.<n>Trained on an expansive dataset comprising 386B bp of DNA, the GENERator demonstrates state-of-the-art performance across both established and newly proposed benchmarks.<n>It also shows significant promise in sequence optimization, particularly through the prompt-responsive generation of enhancer sequences with specific activity profiles.
  arXiv  Detail & Related papers  (2025-02-11T05:39:49Z)
• Computational Protein Science in the Era of Large Language Models (LLMs) [54.35488233989787]
  Computational protein science is dedicated to revealing knowledge and developing applications within the protein sequence-structure-function paradigm.<n>Recently, Language Models (pLMs) have emerged as a milestone in AI due to their unprecedented language processing & generalization capability.
  arXiv  Detail & Related papers  (2025-01-17T16:21:18Z)
• Open-Source Protein Language Models for Function Prediction and Protein Design [0.0]
  Protein language models (PLMs) have shown promise in improving the understanding of protein sequences, contributing to advances in areas such as function prediction and protein engineering.<n>We integrate a PLM into DeepChem, an open-source framework for computational biology and chemistry, to provide a more accessible platform for protein-related tasks.<n>We evaluate the performance of the integrated model on various protein prediction tasks, showing that it achieves reasonable results across benchmarks.
  arXiv  Detail & Related papers  (2024-12-18T05:41:15Z)
• ProteinBench: A Holistic Evaluation of Protein Foundation Models [53.59325047872512]
  We introduce ProteinBench, a holistic evaluation framework for protein foundation models. Our approach consists of three key components: (i) A taxonomic classification of tasks that broadly encompass the main challenges in the protein domain, based on the relationships between different protein modalities; (ii) A multi-metric evaluation approach that assesses performance across four key dimensions: quality, novelty, diversity, and robustness; and (iii) In-depth analyses from various user objectives, providing a holistic view of model performance.
  arXiv  Detail & Related papers  (2024-09-10T06:52:33Z)
• Boosting Protein Language Models with Negative Sample Mining [20.721167029530168]
  We introduce a pioneering methodology for boosting large language models in the domain of protein representation learning. Our primary contribution lies in the refinement process for correlating the over-reliance on co-evolution knowledge. By capitalizing on this novel approach, our technique steers the training of transformer-based models within the attention score space.
  arXiv  Detail & Related papers  (2024-05-28T07:24:20Z)
• An Evaluation of Large Language Models in Bioinformatics Research [52.100233156012756]
  We study the performance of large language models (LLMs) on a wide spectrum of crucial bioinformatics tasks. These tasks include the identification of potential coding regions, extraction of named entities for genes and proteins, detection of antimicrobial and anti-cancer peptides, molecular optimization, and resolution of educational bioinformatics problems. Our findings indicate that, given appropriate prompts, LLMs like GPT variants can successfully handle most of these tasks.
  arXiv  Detail & Related papers  (2024-02-21T11:27:31Z)
• xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein [74.64101864289572]
  We propose a unified protein language model, xTrimoPGLM, to address protein understanding and generation tasks simultaneously.<n>xTrimoPGLM significantly outperforms other advanced baselines in 18 protein understanding benchmarks across four categories.<n>It can also generate de novo protein sequences following the principles of natural ones, and can perform programmable generation after supervised fine-tuning.
  arXiv  Detail & Related papers  (2024-01-11T15:03:17Z)
• Exploring the Protein Sequence Space with Global Generative Models [0.0]
  Language models have demonstrated exceptional capabilities in processing, translating, and generating human languages. Protein generative models have been utilized to embed proteins, generate novel ones, and predict tertiary structures. In this book chapter, we provide an overview of the use of protein generative models, reviewing 1) language models for the design of novel artificial proteins, 2) works that use non-Transformer architectures, and 3) applications in directed evolution approaches.
  arXiv  Detail & Related papers  (2023-05-03T07:45:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.