{\lambda}-Scaled-Attention: A Novel Fast Attention Mechanism for Efficient Modeling of Protein Sequences

• URL: http://arxiv.org/abs/2201.02912v1
• Date: Sun, 9 Jan 2022 03:04:23 GMT
• Title: {\lambda}-Scaled-Attention: A Novel Fast Attention Mechanism for Efficient Modeling of Protein Sequences
• Authors: Ashish Ranjan, Md Shah Fahad, Akshay Deepak
• Abstract summary: Attention-based deep networks have been successfully applied on textual data in the field of NLP. Their application on protein sequences poses additional challenges due to the weak semantics of the protein words, unlike the plain text words. We introduce a novel lambda-scaled attention technique for fast and efficient modeling of the protein sequences.
• Score: 1.9744199942774268
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Attention-based deep networks have been successfully applied on textual data in the field of NLP. However, their application on protein sequences poses additional challenges due to the weak semantics of the protein words, unlike the plain text words. These unexplored challenges faced by the standard attention technique include (i) vanishing attention score problem and (ii) high variations in the attention distribution. In this regard, we introduce a novel {\lambda}-scaled attention technique for fast and efficient modeling of the protein sequences that addresses both the above problems. This is used to develop the {\lambda}-scaled attention network and is evaluated for the task of protein function prediction implemented at the protein sub-sequence level. Experiments on the datasets for biological process (BP) and molecular function (MF) showed significant improvements in the F1 score values for the proposed {\lambda}-scaled attention technique over its counterpart approach based on the standard attention technique (+2.01% for BP and +4.67% for MF) and state-of-the-art ProtVecGen-Plus approach (+2.61% for BP and +4.20% for MF). Further, fast convergence (converging in half the number of epochs) and efficient learning (in terms of very low difference between the training and validation losses) were also observed during the training process.

Related papers

• S$^2$Drug: Bridging Protein Sequence and 3D Structure in Contrastive Representation Learning for Virtual Screening [72.89086338778098]
  We propose a two-stage framework for protein-ligand contrastive representation learning.<n>In the first stage, we perform protein sequence pretraining on ChemBL using an ESM2-based backbone.<n>In the second stage, we fine-tune on PDBBind by fusing sequence and structure information through a residue-level gating module.<n>This auxiliary task guides the model to accurately localize binding residues within the protein sequence and capture their 3D spatial arrangement.
  arXiv  Detail & Related papers  (2025-11-10T11:57:47Z)
• A Novel Framework for Multi-Modal Protein Representation Learning [13.33566214386641]
  We propose Diffused and Aligned Multi-modal Protein Embedding (DAMPE), a unified framework that addresses two core mechanisms.<n>First, we propose Optimal Transport (OT)-based representation alignment that establishes correspondence between intrinsic embedding spaces of different modalities.<n>Second, we develop a Conditional Graph Generation (CGG)-based information fusion method, where a condition encoder fuses the aligned intrinsic embeddings to provide informative cues for graph reconstruction.
  arXiv  Detail & Related papers  (2025-10-27T12:33:01Z)
• ProteinAE: Protein Diffusion Autoencoders for Structure Encoding [64.77182442408254]
  We introduce ProteinAE, a novel and streamlined protein diffusion autoencoder.<n>ProteinAE directly maps protein backbone coordinates from E(3) into a continuous, compact latent space.<n>We demonstrate that ProteinAE achieves state-of-the-art reconstruction quality, outperforming existing autoencoders.
  arXiv  Detail & Related papers  (2025-10-12T14:30:32Z)
• AMix-1: A Pathway to Test-Time Scalable Protein Foundation Model [92.51919604882984]
  We introduce AMix-1, a powerful protein foundation model built on Flow Bayesian Networks.<n>AMix-1 is empowered by a systematic training methodology, encompassing pretraining scaling laws, emergent capability analysis, in-context learning mechanism, and test-time scaling algorithm.<n>Building on this foundation, we devise a multiple sequence alignment (MSA)-based in-context learning strategy to unify protein design into a general framework.
  arXiv  Detail & Related papers  (2025-07-11T17:02:25Z)
• PLAME: Leveraging Pretrained Language Models to Generate Enhanced Protein Multiple Sequence Alignments [53.55710514466851]
  Protein structure prediction is essential for drug discovery and understanding biological functions.<n>Most folding models rely heavily on multiple sequence alignments (MSAs) to boost prediction performance.<n>We propose PLAME, a novel MSA design model that leverages evolutionary embeddings from pretrained protein language models.
  arXiv  Detail & Related papers  (2025-06-17T04:11:30Z)
• ProtCLIP: Function-Informed Protein Multi-Modal Learning [18.61302416993122]
  We develop ProtCLIP, a multi-modality foundation model that represents function-aware protein embeddings. Our ProtCLIP consistently achieves SOTA performance, with remarkable improvements of 75% on average in five cross-modal transformation benchmarks. The experimental results verify the extraordinary potential of ProtCLIP serving as the protein multi-modality foundation model.
  arXiv  Detail & Related papers  (2024-12-28T04:23:47Z)
• SeqProFT: Applying LoRA Finetuning for Sequence-only Protein Property Predictions [8.112057136324431]
  This study employs the LoRA method to perform end-to-end fine-tuning of the ESM-2 model. A multi-head attention mechanism is integrated into the downstream network to combine sequence features with contact map information.
  arXiv  Detail & Related papers  (2024-11-18T12:40:39Z)
• Position Specific Scoring Is All You Need? Revisiting Protein Sequence Classification Tasks [41.7345616221048]
  We propose a weighted PSS kernel matrix (or W-PSSKM) that combines a PSS representation of protein sequences with the notion of the string kernel. This results in a novel kernel function that outperforms many other approaches for protein sequence classification.
  arXiv  Detail & Related papers  (2024-10-16T15:16:50Z)
• NovoBench: Benchmarking Deep Learning-based De Novo Peptide Sequencing Methods in Proteomics [58.03989832372747]
  We present the first unified benchmark NovoBench for emphde novo peptide sequencing. It comprises diverse mass spectrum data, integrated models, and comprehensive evaluation metrics. Recent methods, including DeepNovo, PointNovo, Casanovo, InstaNovo, AdaNovo and $pi$-HelixNovo are integrated into our framework.
  arXiv  Detail & Related papers  (2024-06-16T08:23:21Z)
• Beyond ESM2: Graph-Enhanced Protein Sequence Modeling with Efficient Clustering [24.415612744612773]
  Proteins are essential to life's processes, underpinning evolution and diversity. Advances in sequencing technology have revealed millions of proteins, underscoring the need for sophisticated pre-trained protein models for biological analysis and AI development. Facebook's ESM2, the most advanced protein language model to date, leverages a masked prediction task for unsupervised learning, crafting amino acid representations with notable biochemical accuracy. Yet, it lacks in delivering functional protein insights, signaling an opportunity for enhancing representation quality. This study addresses this gap by incorporating protein family classification into ESM2's training, while a contextual prediction task fine-tunes local
  arXiv  Detail & Related papers  (2024-04-24T11:09:43Z)
• DisorderUnetLM: Validating ProteinUnet for efficient protein intrinsic disorder prediction [0.0]
  The prediction of intrinsic disorder regions has significant implications for understanding protein functions and dynamics. Recently, a new generation of predictors based on protein language models (pLMs) is emerging. The article introduces the new DisorderUnetLM disorder predictor, which builds upon the idea of ProteinUnet.
  arXiv  Detail & Related papers  (2024-04-11T20:14:14Z)
• ContraNovo: A Contrastive Learning Approach to Enhance De Novo Peptide Sequencing [70.12220342151113]
  ContraNovo is a pioneering algorithm that leverages contrastive learning to extract the relationship between spectra and peptides. ContraNovo consistently outshines contemporary state-of-the-art solutions.
  arXiv  Detail & Related papers  (2023-12-18T12:49:46Z)
• Efficiently Predicting Protein Stability Changes Upon Single-point Mutation with Large Language Models [51.57843608615827]
  The ability to precisely predict protein thermostability is pivotal for various subfields and applications in biochemistry. We introduce an ESM-assisted efficient approach that integrates protein sequence and structural features to predict the thermostability changes in protein upon single-point mutations.
  arXiv  Detail & Related papers  (2023-12-07T03:25:49Z)
• Exploring Post-Training Quantization of Protein Language Models [9.562786130999188]
  We study post-training quantization (PTQ) for protein language models (ProteinLMs) Our study is the first attempt to quantize all weights and activations of ProteinLMs. We propose a new PTQ method for ProteinLMs, utilizing piecewise linear quantization for asymmetric activation values.
  arXiv  Detail & Related papers  (2023-10-30T15:18:06Z)
• Structure-aware Protein Self-supervised Learning [50.04673179816619]
  We propose a novel structure-aware protein self-supervised learning method to capture structural information of proteins. In particular, a well-designed graph neural network (GNN) model is pretrained to preserve the protein structural information. We identify the relation between the sequential information in the protein language model and the structural information in the specially designed GNN model via a novel pseudo bi-level optimization scheme.
  arXiv  Detail & Related papers  (2022-04-06T02:18:41Z)
• Pre-training Co-evolutionary Protein Representation via A Pairwise Masked Language Model [93.9943278892735]
  Key problem in protein sequence representation learning is to capture the co-evolutionary information reflected by the inter-residue co-variation in the sequences. We propose a novel method to capture this information directly by pre-training via a dedicated language model, i.e., Pairwise Masked Language Model (PMLM) Our result shows that the proposed method can effectively capture the interresidue correlations and improves the performance of contact prediction by up to 9% compared to the baseline.
  arXiv  Detail & Related papers  (2021-10-29T04:01:32Z)
• EBM-Fold: Fully-Differentiable Protein Folding Powered by Energy-based Models [53.17320541056843]
  We propose a fully-differentiable approach for protein structure optimization, guided by a data-driven generative network. Our EBM-Fold approach can efficiently produce high-quality decoys, compared against traditional Rosetta-based structure optimization routines.
  arXiv  Detail & Related papers  (2021-05-11T03:40:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.