Pre-Training Protein Bi-level Representation Through Span Mask Strategy On 3D Protein Chains

• URL: http://arxiv.org/abs/2402.01481v4
• Date: Sun, 2 Jun 2024 23:17:38 GMT
• Title: Pre-Training Protein Bi-level Representation Through Span Mask Strategy On 3D Protein Chains
• Authors: Jiale Zhao, Wanru Zhuang, Jia Song, Yaqi Li, Shuqi Lu,
• Abstract summary: We introduce a span mask pre-training strategy on 3D protein chains to learn meaningful representations of both residues and atoms. This leads to a simple yet effective approach to learning protein representation suitable for diverse downstream tasks.
• Score: 1.2893576217358405
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In recent years, there has been a surge in the development of 3D structure-based pre-trained protein models, representing a significant advancement over pre-trained protein language models in various downstream tasks. However, most existing structure-based pre-trained models primarily focus on the residue level, i.e., alpha carbon atoms, while ignoring other atoms like side chain atoms. We argue that modeling proteins at both residue and atom levels is important since the side chain atoms can also be crucial for numerous downstream tasks, for example, molecular docking. Nevertheless, we find that naively combining residue and atom information during pre-training typically fails. We identify a key reason is the information leakage caused by the inclusion of atom structure in the input, which renders residue-level pre-training tasks trivial and results in insufficiently expressive residue representations. To address this issue, we introduce a span mask pre-training strategy on 3D protein chains to learn meaningful representations of both residues and atoms. This leads to a simple yet effective approach to learning protein representation suitable for diverse downstream tasks. Extensive experimental results on binding site prediction and function prediction tasks demonstrate our proposed pre-training approach significantly outperforms other methods. Our code will be made public.

Related papers

• S$^2$Drug: Bridging Protein Sequence and 3D Structure in Contrastive Representation Learning for Virtual Screening [72.89086338778098]
  We propose a two-stage framework for protein-ligand contrastive representation learning.<n>In the first stage, we perform protein sequence pretraining on ChemBL using an ESM2-based backbone.<n>In the second stage, we fine-tune on PDBBind by fusing sequence and structure information through a residue-level gating module.<n>This auxiliary task guides the model to accurately localize binding residues within the protein sequence and capture their 3D spatial arrangement.
  arXiv  Detail & Related papers  (2025-11-10T11:57:47Z)
• Ankh3: Multi-Task Pretraining with Sequence Denoising and Completion Enhances Protein Representations [0.3124884279860061]
  Protein language models (PLMs) have emerged as powerful tools to detect complex patterns of protein sequences.<n>Our research investigated a multi-task pre-training strategy for PLMs.<n>This multi-task pre-training demonstrated that PLMs can learn richer and more generalizable representations solely from protein sequences.
  arXiv  Detail & Related papers  (2025-05-26T14:41:10Z)
• SFM-Protein: Integrative Co-evolutionary Pre-training for Advanced Protein Sequence Representation [97.99658944212675]
  We introduce a novel pre-training strategy for protein foundation models. It emphasizes the interactions among amino acid residues to enhance the extraction of both short-range and long-range co-evolutionary features. Trained on a large-scale protein sequence dataset, our model demonstrates superior generalization ability.
  arXiv  Detail & Related papers  (2024-10-31T15:22:03Z)
• Advanced atom-level representations for protein flexibility prediction utilizing graph neural networks [0.0]
  We propose graph neural networks (GNNs) to learn protein representations at the atomic level and predict B-factors from protein 3D structures. The Meta-GNN model achieves a correlation coefficient of 0.71 on a large and diverse test set of over 4k proteins.
  arXiv  Detail & Related papers  (2024-08-22T16:15:13Z)
• Geometric Self-Supervised Pretraining on 3D Protein Structures using Subgraphs [26.727436310732692]
  We propose a novel self-supervised method to pretrain 3D graph neural networks on 3D protein structures. We experimentally show that our proposed pertaining strategy leads to significant improvements up to 6%.
  arXiv  Detail & Related papers  (2024-06-20T09:34:31Z)
• xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein [76.18058946124111]
  We propose a unified protein language model, xTrimoPGLM, to address protein understanding and generation tasks simultaneously. xTrimoPGLM significantly outperforms other advanced baselines in 18 protein understanding benchmarks across four categories. It can also generate de novo protein sequences following the principles of natural ones, and can perform programmable generation after supervised fine-tuning.
  arXiv  Detail & Related papers  (2024-01-11T15:03:17Z)
• EquiPocket: an E(3)-Equivariant Geometric Graph Neural Network for Ligand Binding Site Prediction [49.674494450107005]
  Predicting the binding sites of target proteins plays a fundamental role in drug discovery. Most existing deep-learning methods consider a protein as a 3D image by spatially clustering its atoms into voxels. This work proposes EquiPocket, an E(3)-equivariant Graph Neural Network (GNN) for binding site prediction.
  arXiv  Detail & Related papers  (2023-02-23T17:18:26Z)
• Data-Efficient Protein 3D Geometric Pretraining via Refinement of Diffused Protein Structure Decoy [42.49977473599661]
  Learning meaningful protein representation is important for a variety of biological downstream tasks such as structure-based drug design. In this paper, we propose a unified framework for protein pretraining and a 3D geometric-based, data-efficient, and protein-specific pretext task: RefineDiff.
  arXiv  Detail & Related papers  (2023-02-05T14:13:32Z)
• Reprogramming Pretrained Language Models for Protein Sequence Representation Learning [68.75392232599654]
  We propose Representation Learning via Dictionary Learning (R2DL), an end-to-end representation learning framework. R2DL reprograms a pretrained English language model to learn the embeddings of protein sequences. Our model can attain better accuracy and significantly improve the data efficiency by up to $105$ times over the baselines set by pretrained and standard supervised methods.
  arXiv  Detail & Related papers  (2023-01-05T15:55:18Z)
• Structure-aware Protein Self-supervised Learning [50.04673179816619]
  We propose a novel structure-aware protein self-supervised learning method to capture structural information of proteins. In particular, a well-designed graph neural network (GNN) model is pretrained to preserve the protein structural information. We identify the relation between the sequential information in the protein language model and the structural information in the specially designed GNN model via a novel pseudo bi-level optimization scheme.
  arXiv  Detail & Related papers  (2022-04-06T02:18:41Z)
• Multi-Scale Representation Learning on Proteins [78.31410227443102]
  This paper introduces a multi-scale graph construction of a protein -- HoloProt. The surface captures coarser details of the protein, while sequence as primary component and structure captures finer details. Our graph encoder then learns a multi-scale representation by allowing each level to integrate the encoding from level(s) below with the graph at that level.
  arXiv  Detail & Related papers  (2022-04-04T08:29:17Z)
• Protein Representation Learning by Geometric Structure Pretraining [27.723095456631906]
  Existing approaches usually pretrain protein language models on a large number of unlabeled amino acid sequences. We first present a simple yet effective encoder to learn protein geometry features. Experimental results on both function prediction and fold classification tasks show that our proposed pretraining methods outperform or are on par with the state-of-the-art sequence-based methods using much less data.
  arXiv  Detail & Related papers  (2022-03-11T17:52:13Z)
• Profile Prediction: An Alignment-Based Pre-Training Task for Protein Sequence Models [11.483725773928382]
  Recent deep-learning approaches to protein prediction have shown that pre-training on unlabeled data can yield useful representations for downstream tasks. We introduce a new pre-training task: directly predicting protein profiles derived from multiple sequence alignments. Our results suggest that protein sequence models may benefit from leveraging biologically-inspired inductive biases.
  arXiv  Detail & Related papers  (2020-12-01T01:01:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.