ProtLLM: An Interleaved Protein-Language LLM with Protein-as-Word Pre-Training

• URL: http://arxiv.org/abs/2403.07920v1
• Date: Wed, 28 Feb 2024 01:29:55 GMT
• Title: ProtLLM: An Interleaved Protein-Language LLM with Protein-as-Word Pre-Training
• Authors: Le Zhuo, Zewen Chi, Minghao Xu, Heyan Huang, Heqi Zheng, Conghui He, Xian-Ling Mao, Wentao Zhang,
• Abstract summary: We propose a versatile cross-modal large language model (LLM) for both protein-centric and protein-language tasks. ProtLLM features a unique dynamic protein mounting mechanism, enabling it to handle complex inputs. By developing a specialized protein vocabulary, we equip the model with the capability to predict not just natural language but also proteins from a vast pool of candidates.
• Score: 82.37346937497136
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose ProtLLM, a versatile cross-modal large language model (LLM) for both protein-centric and protein-language tasks. ProtLLM features a unique dynamic protein mounting mechanism, enabling it to handle complex inputs where the natural language text is interspersed with an arbitrary number of proteins. Besides, we propose the protein-as-word language modeling approach to train ProtLLM. By developing a specialized protein vocabulary, we equip the model with the capability to predict not just natural language but also proteins from a vast pool of candidates. Additionally, we construct a large-scale interleaved protein-text dataset, named InterPT, for pre-training. This dataset comprehensively encompasses both (1) structured data sources like protein annotations and (2) unstructured data sources like biological research papers, thereby endowing ProtLLM with crucial knowledge for understanding proteins. We evaluate ProtLLM on classic supervised protein-centric tasks and explore its novel protein-language applications. Experimental results demonstrate that ProtLLM not only achieves superior performance against protein-specialized baselines on protein-centric tasks but also induces zero-shot and in-context learning capabilities on protein-language tasks.

Related papers

• Computational Protein Science in the Era of Large Language Models (LLMs) [54.35488233989787]
  Computational protein science is dedicated to revealing knowledge and developing applications within the protein sequence-structure-function paradigm. Recently, Language Models (pLMs) have emerged as a milestone in AI due to their unprecedented language processing & generalization capability.
  arXiv  Detail & Related papers  (2025-01-17T16:21:18Z)
• EvoLlama: Enhancing LLMs' Understanding of Proteins via Multimodal Structure and Sequence Representations [28.298740080002077]
  Current Large Language Models (LLMs) for understanding proteins primarily treats amino acid sequences as a text modality. EvoLlama is a framework that connects a structure-based encoder, a sequence-based protein encoder and an LLM for protein understanding. Our experiments show that EvoLlama's protein understanding capabilities have been significantly enhanced.
  arXiv  Detail & Related papers  (2024-12-16T10:01:33Z)
• ProtDAT: A Unified Framework for Protein Sequence Design from Any Protein Text Description [7.198238666986253]
  We propose a de novo fine-grained framework capable of designing proteins from any descriptive text input. Prot DAT builds upon the inherent characteristics of protein data to unify sequences and text as a cohesive whole rather than separate entities. Experimental results demonstrate that Prot DAT achieves the state-of-the-art performance in protein sequence generation, excelling in rationality, functionality, structural similarity, and validity.
  arXiv  Detail & Related papers  (2024-12-05T11:05:46Z)
• Long-context Protein Language Model [76.95505296417866]
  Self-supervised training of language models (LMs) has seen great success for protein sequences in learning meaningful representations and for generative drug design. Most protein LMs are based on the Transformer architecture trained on individual proteins with short context lengths. We propose LC-PLM based on an alternative protein LM architecture, BiMamba-S, built off selective structured state-space models. We also introduce its graph-contextual variant, LC-PLM-G, which contextualizes protein-protein interaction graphs for a second stage of training.
  arXiv  Detail & Related papers  (2024-10-29T16:43:28Z)
• ProtT3: Protein-to-Text Generation for Text-based Protein Understanding [88.43323947543996]
  Language Models (LMs) excel in understanding textual descriptions of proteins. Protein Language Models (PLMs) can understand and convert protein data into high-quality representations, but struggle to process texts. We introduce ProtT3, a framework for Protein-to-Text Generation for Text-based Protein Understanding.
  arXiv  Detail & Related papers  (2024-05-21T08:06:13Z)
• ProLLM: Protein Chain-of-Thoughts Enhanced LLM for Protein-Protein Interaction Prediction [54.132290875513405]
  The prediction of protein-protein interactions (PPIs) is crucial for understanding biological functions and diseases. Previous machine learning approaches to PPI prediction mainly focus on direct physical interactions. We propose a novel framework ProLLM that employs an LLM tailored for PPI for the first time.
  arXiv  Detail & Related papers  (2024-03-30T05:32:42Z)
• A Text-guided Protein Design Framework [106.79061950107922]
  We propose ProteinDT, a multi-modal framework that leverages textual descriptions for protein design. ProteinDT consists of three subsequent steps: ProteinCLAP which aligns the representation of two modalities, a facilitator that generates the protein representation from the text modality, and a decoder that creates the protein sequences from the representation. We quantitatively verify the effectiveness of ProteinDT on three challenging tasks: (1) over 90% accuracy for text-guided protein generation; (2) best hit ratio on 12 zero-shot text-guided protein editing tasks; (3) superior performance on four out of six protein property prediction benchmarks.
  arXiv  Detail & Related papers  (2023-02-09T12:59:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.