ProtCLIP: Function-Informed Protein Multi-Modal Learning

• URL: http://arxiv.org/abs/2412.20014v1
• Date: Sat, 28 Dec 2024 04:23:47 GMT
• Title: ProtCLIP: Function-Informed Protein Multi-Modal Learning
• Authors: Hanjing Zhou, Mingze Yin, Wei Wu, Mingyang Li, Kun Fu, Jintai Chen, Jian Wu, Zheng Wang,
• Abstract summary: We develop ProtCLIP, a multi-modality foundation model that represents function-aware protein embeddings. Our ProtCLIP consistently achieves SOTA performance, with remarkable improvements of 75% on average in five cross-modal transformation benchmarks. The experimental results verify the extraordinary potential of ProtCLIP serving as the protein multi-modality foundation model.
• Score: 18.61302416993122
• License:
• Abstract: Multi-modality pre-training paradigm that aligns protein sequences and biological descriptions has learned general protein representations and achieved promising performance in various downstream applications. However, these works were still unable to replicate the extraordinary success of language-supervised visual foundation models due to the ineffective usage of aligned protein-text paired data and the lack of an effective function-informed pre-training paradigm. To address these issues, this paper curates a large-scale protein-text paired dataset called ProtAnno with a property-driven sampling strategy, and introduces a novel function-informed protein pre-training paradigm. Specifically, the sampling strategy determines selecting probability based on the sample confidence and property coverage, balancing the data quality and data quantity in face of large-scale noisy data. Furthermore, motivated by significance of the protein specific functional mechanism, the proposed paradigm explicitly model protein static and dynamic functional segments by two segment-wise pre-training objectives, injecting fine-grained information in a function-informed manner. Leveraging all these innovations, we develop ProtCLIP, a multi-modality foundation model that comprehensively represents function-aware protein embeddings. On 22 different protein benchmarks within 5 types, including protein functionality classification, mutation effect prediction, cross-modal transformation, semantic similarity inference and protein-protein interaction prediction, our ProtCLIP consistently achieves SOTA performance, with remarkable improvements of 75% on average in five cross-modal transformation benchmarks, 59.9% in GO-CC and 39.7% in GO-BP protein function prediction. The experimental results verify the extraordinary potential of ProtCLIP serving as the protein multi-modality foundation model.

Related papers

• SFM-Protein: Integrative Co-evolutionary Pre-training for Advanced Protein Sequence Representation [97.99658944212675]
  We introduce a novel pre-training strategy for protein foundation models. It emphasizes the interactions among amino acid residues to enhance the extraction of both short-range and long-range co-evolutionary features. Trained on a large-scale protein sequence dataset, our model demonstrates superior generalization ability.
  arXiv  Detail & Related papers  (2024-10-31T15:22:03Z)
• Long-context Protein Language Model [76.95505296417866]
  Self-supervised training of language models (LMs) has seen great success for protein sequences in learning meaningful representations and for generative drug design. Most protein LMs are based on the Transformer architecture trained on individual proteins with short context lengths. We propose LC-PLM based on an alternative protein LM architecture, BiMamba-S, built off selective structured state-space models. We also introduce its graph-contextual variant, LC-PLM-G, which contextualizes protein-protein interaction graphs for a second stage of training.
  arXiv  Detail & Related papers  (2024-10-29T16:43:28Z)
• Protein-Mamba: Biological Mamba Models for Protein Function Prediction [18.642511763423048]
  Protein-Mamba is a novel two-stage model that leverages both self-supervised learning and fine-tuning to improve protein function prediction. Our experiments demonstrate that Protein-Mamba achieves competitive performance, compared with a couple of state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-09-22T22:51:56Z)
• PSC-CPI: Multi-Scale Protein Sequence-Structure Contrasting for Efficient and Generalizable Compound-Protein Interaction Prediction [63.50967073653953]
  Compound-Protein Interaction prediction aims to predict the pattern and strength of compound-protein interactions for rational drug discovery. Existing deep learning-based methods utilize only the single modality of protein sequences or structures. We propose a novel multi-scale Protein Sequence-structure Contrasting framework for CPI prediction.
  arXiv  Detail & Related papers  (2024-02-13T03:51:10Z)
• Efficiently Predicting Protein Stability Changes Upon Single-point Mutation with Large Language Models [51.57843608615827]
  The ability to precisely predict protein thermostability is pivotal for various subfields and applications in biochemistry. We introduce an ESM-assisted efficient approach that integrates protein sequence and structural features to predict the thermostability changes in protein upon single-point mutations.
  arXiv  Detail & Related papers  (2023-12-07T03:25:49Z)
• Prot2Text: Multimodal Protein's Function Generation with GNNs and Transformers [18.498779242323582]
  We propose a novel approach, Prot2Text, which predicts a protein's function in a free text style. By combining Graph Neural Networks(GNNs) and Large Language Models(LLMs), in an encoder-decoder framework, our model effectively integrates diverse data types.
  arXiv  Detail & Related papers  (2023-07-25T09:35:43Z)
• Multi-level Protein Representation Learning for Blind Mutational Effect Prediction [5.207307163958806]
  This paper introduces a novel pre-training framework that cascades sequential and geometric analyzers for protein structures. It guides mutational directions toward desired traits by simulating natural selection on wild-type proteins. We assess the proposed approach using a public database and two new databases for a variety of variant effect prediction tasks.
  arXiv  Detail & Related papers  (2023-06-08T03:00:50Z)
• Structure-informed Language Models Are Protein Designers [69.70134899296912]
  We present LM-Design, a generic approach to reprogramming sequence-based protein language models (pLMs) We conduct a structural surgery on pLMs, where a lightweight structural adapter is implanted into pLMs and endows it with structural awareness. Experiments show that our approach outperforms the state-of-the-art methods by a large margin.
  arXiv  Detail & Related papers  (2023-02-03T10:49:52Z)
• Ankh: Optimized Protein Language Model Unlocks General-Purpose Modelling [0.0]
  We present Ankh, the first general-purpose protein language model trained on Google's TPU-v4. Ankh succeeds in learning protein evolutionary conservation-mutation trends and introducing functional diversity while retaining key structural-functional characteristics.
  arXiv  Detail & Related papers  (2023-01-16T19:04:45Z)
• Multi-Scale Representation Learning on Proteins [78.31410227443102]
  This paper introduces a multi-scale graph construction of a protein -- HoloProt. The surface captures coarser details of the protein, while sequence as primary component and structure captures finer details. Our graph encoder then learns a multi-scale representation by allowing each level to integrate the encoding from level(s) below with the graph at that level.
  arXiv  Detail & Related papers  (2022-04-04T08:29:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.