Exploring Post-Training Quantization of Protein Language Models

• URL: http://arxiv.org/abs/2310.19624v1
• Date: Mon, 30 Oct 2023 15:18:06 GMT
• Title: Exploring Post-Training Quantization of Protein Language Models
• Authors: Shuang Peng, Fei Yang, Ning Sun, Sheng Chen, Yanfeng Jiang, Aimin Pan
• Abstract summary: We study post-training quantization (PTQ) for protein language models (ProteinLMs) Our study is the first attempt to quantize all weights and activations of ProteinLMs. We propose a new PTQ method for ProteinLMs, utilizing piecewise linear quantization for asymmetric activation values.
• Score: 9.562786130999188
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advancements in unsupervised protein language models (ProteinLMs), like ESM-1b and ESM-2, have shown promise in different protein prediction tasks. However, these models face challenges due to their high computational demands, significant memory needs, and latency, restricting their usage on devices with limited resources. To tackle this, we explore post-training quantization (PTQ) for ProteinLMs, focusing on ESMFold, a simplified version of AlphaFold based on ESM-2 ProteinLM. Our study is the first attempt to quantize all weights and activations of ProteinLMs. We observed that the typical uniform quantization method performs poorly on ESMFold, causing a significant drop in TM-Score when using 8-bit quantization. We conducted extensive quantization experiments, uncovering unique challenges associated with ESMFold, particularly highly asymmetric activation ranges before Layer Normalization, making representation difficult using low-bit fixed-point formats. To address these challenges, we propose a new PTQ method for ProteinLMs, utilizing piecewise linear quantization for asymmetric activation values to ensure accurate approximation. We demonstrated the effectiveness of our method in protein structure prediction tasks, demonstrating that ESMFold can be accurately quantized to low-bit widths without compromising accuracy. Additionally, we applied our method to the contact prediction task, showcasing its versatility. In summary, our study introduces an innovative PTQ method for ProteinLMs, addressing specific quantization challenges and potentially leading to the development of more efficient ProteinLMs with significant implications for various protein-related applications.

Related papers

• Self Distillation Fine-Tuning of Protein Language Models Improves Versatility in Protein Design [61.2846583160056]
  Supervised fine-tuning (SFT) is a standard approach for adapting large language models to specialized domains.<n>This is in part because high-quality annotated data are far more difficult to obtain for proteins than for natural language.<n>We present a simple and general recipe for fast SFT of PLMs, designed to improve the fidelity, reliability, and novelty of generated protein sequences.
  arXiv  Detail & Related papers  (2025-12-10T05:34:47Z)
• Protein as a Second Language for LLMs [50.34983283157322]
  "Protein-as-Second-Language" framework reformulates amino-acid sequences as sentences in a novel symbolic language.<n>We curate a bilingual corpus of 79,926 protein-QA instances spanning attribute prediction, descriptive understanding, and extended reasoning.<n>Our method delivers consistent gains across diverse open-source LLMs and GPT-4, achieving up to 17.2% ROUGE-L improvement.
  arXiv  Detail & Related papers  (2025-10-13T09:21:45Z)
• Sparse Autoencoders for Low-$N$ Protein Function Prediction and Design [0.0]
  Predicting protein function from amino acid sequence remains a central challenge in data-scarce regimes.<n>Protein language models (pLMs) have advanced the field by providing evolutionary-informed embeddings and sparse autoencoders (SAEs)<n>We show that SAEs, with as few as 24 sequences, consistently outperform or compete with their ESM2 baselines in fitness prediction.
  arXiv  Detail & Related papers  (2025-08-25T23:56:39Z)
• RinQ: Predicting central sites in proteins on current quantum computers [0.0]
  RinQ is a hybrid quantum-classical framework for identifying functionally critical residues in proteins.<n>This work highlights the promise of near-term quantum and quantum-inspired methods for advancing protein network analysis.
  arXiv  Detail & Related papers  (2025-08-02T21:53:09Z)
• Steering Protein Language Models [22.308373820985793]
  Activation Steering is a technique originally developed for controlling text generation in Large Language Models.<n>We propose a simple yet effective method that employs activation editing to steer PLM outputs.<n>We show that our methods can be seamlessly integrated into both auto-encoding and autoregressive PLMs without requiring additional training.
  arXiv  Detail & Related papers  (2025-07-01T16:03:55Z)
• PLAME: Leveraging Pretrained Language Models to Generate Enhanced Protein Multiple Sequence Alignments [53.55710514466851]
  Protein structure prediction is essential for drug discovery and understanding biological functions.<n>Most folding models rely heavily on multiple sequence alignments (MSAs) to boost prediction performance.<n>We propose PLAME, a novel MSA design model that leverages evolutionary embeddings from pretrained protein language models.
  arXiv  Detail & Related papers  (2025-06-17T04:11:30Z)
• Ankh3: Multi-Task Pretraining with Sequence Denoising and Completion Enhances Protein Representations [0.3124884279860061]
  Protein language models (PLMs) have emerged as powerful tools to detect complex patterns of protein sequences.<n>Our research investigated a multi-task pre-training strategy for PLMs.<n>This multi-task pre-training demonstrated that PLMs can learn richer and more generalizable representations solely from protein sequences.
  arXiv  Detail & Related papers  (2025-05-26T14:41:10Z)
• Beyond Simple Concatenation: Fairly Assessing PLM Architectures for Multi-Chain Protein-Protein Interactions Prediction [0.2509487459755192]
  Protein-protein interactions (PPIs) are fundamental to numerous cellular processes.<n>PLMs have demonstrated remarkable success in predicting protein structure and function.<n>Their application to sequence-based PPI binding affinity prediction remains relatively underexplored.
  arXiv  Detail & Related papers  (2025-05-26T14:23:08Z)
• Protein Large Language Models: A Comprehensive Survey [71.65899614084853]
  Protein-specific large language models (Protein LLMs) are revolutionizing protein science by enabling more efficient protein structure prediction, function annotation, and design. This work provides the first comprehensive overview of Protein LLMs, covering their architectures, training datasets, evaluation metrics, and diverse applications.
  arXiv  Detail & Related papers  (2025-02-21T19:22:10Z)
• Computational Protein Science in the Era of Large Language Models (LLMs) [54.35488233989787]
  Computational protein science is dedicated to revealing knowledge and developing applications within the protein sequence-structure-function paradigm. Recently, Language Models (pLMs) have emerged as a milestone in AI due to their unprecedented language processing & generalization capability.
  arXiv  Detail & Related papers  (2025-01-17T16:21:18Z)
• ProtCLIP: Function-Informed Protein Multi-Modal Learning [18.61302416993122]
  We develop ProtCLIP, a multi-modality foundation model that represents function-aware protein embeddings. Our ProtCLIP consistently achieves SOTA performance, with remarkable improvements of 75% on average in five cross-modal transformation benchmarks. The experimental results verify the extraordinary potential of ProtCLIP serving as the protein multi-modality foundation model.
  arXiv  Detail & Related papers  (2024-12-28T04:23:47Z)
• Long-context Protein Language Model [76.95505296417866]
  Self-supervised training of language models (LMs) has seen great success for protein sequences in learning meaningful representations and for generative drug design. Most protein LMs are based on the Transformer architecture trained on individual proteins with short context lengths. We propose LC-PLM based on an alternative protein LM architecture, BiMamba-S, built off selective structured state-space models. We also introduce its graph-contextual variant, LC-PLM-G, which contextualizes protein-protein interaction graphs for a second stage of training.
  arXiv  Detail & Related papers  (2024-10-29T16:43:28Z)
• NovoBench: Benchmarking Deep Learning-based De Novo Peptide Sequencing Methods in Proteomics [58.03989832372747]
  We present the first unified benchmark NovoBench for emphde novo peptide sequencing. It comprises diverse mass spectrum data, integrated models, and comprehensive evaluation metrics. Recent methods, including DeepNovo, PointNovo, Casanovo, InstaNovo, AdaNovo and $pi$-HelixNovo are integrated into our framework.
  arXiv  Detail & Related papers  (2024-06-16T08:23:21Z)
• Beyond ESM2: Graph-Enhanced Protein Sequence Modeling with Efficient Clustering [24.415612744612773]
  Proteins are essential to life's processes, underpinning evolution and diversity. Advances in sequencing technology have revealed millions of proteins, underscoring the need for sophisticated pre-trained protein models for biological analysis and AI development. Facebook's ESM2, the most advanced protein language model to date, leverages a masked prediction task for unsupervised learning, crafting amino acid representations with notable biochemical accuracy. Yet, it lacks in delivering functional protein insights, signaling an opportunity for enhancing representation quality. This study addresses this gap by incorporating protein family classification into ESM2's training, while a contextual prediction task fine-tunes local
  arXiv  Detail & Related papers  (2024-04-24T11:09:43Z)
• Protein Conformation Generation via Force-Guided SE(3) Diffusion Models [48.48934625235448]
  Deep generative modeling techniques have been employed to generate novel protein conformations. We propose a force-guided SE(3) diffusion model, ConfDiff, for protein conformation generation.
  arXiv  Detail & Related papers  (2024-03-21T02:44:08Z)
• xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein [76.18058946124111]
  We propose a unified protein language model, xTrimoPGLM, to address protein understanding and generation tasks simultaneously. xTrimoPGLM significantly outperforms other advanced baselines in 18 protein understanding benchmarks across four categories. It can also generate de novo protein sequences following the principles of natural ones, and can perform programmable generation after supervised fine-tuning.
  arXiv  Detail & Related papers  (2024-01-11T15:03:17Z)
• Efficiently Predicting Protein Stability Changes Upon Single-point Mutation with Large Language Models [51.57843608615827]
  The ability to precisely predict protein thermostability is pivotal for various subfields and applications in biochemistry. We introduce an ESM-assisted efficient approach that integrates protein sequence and structural features to predict the thermostability changes in protein upon single-point mutations.
  arXiv  Detail & Related papers  (2023-12-07T03:25:49Z)
• PreQuant: A Task-agnostic Quantization Approach for Pre-trained Language Models [52.09865918265002]
  We propose a novel quantize before fine-tuning'' framework, PreQuant. PreQuant is compatible with various quantization strategies, with outlier-aware fine-tuning incorporated to correct the induced quantization error. We demonstrate the effectiveness of PreQuant on the GLUE benchmark using BERT, RoBERTa, and T5.
  arXiv  Detail & Related papers  (2023-05-30T08:41:33Z)
• ProtFIM: Fill-in-Middle Protein Sequence Design via Protein Language Models [0.0]
  In real-world protein engineering, there are many cases where the amino acids in the middle of a protein sequence are optimized while maintaining other residues. Protein language models (pLMs) have been a promising tool for protein sequence design. We show that language models trained via fill-in-middle transformation, called ProtFIM, are more appropriate for protein engineering.
  arXiv  Detail & Related papers  (2023-03-29T04:35:50Z)
• Peptide conformational sampling using the Quantum Approximate Optimization Algorithm [0.03499870393443267]
  We numerically investigate the performance of a variational quantum algorithm in sampling low-energy conformations of short peptides. Results cast serious doubt on the ability of QAOA to address the protein folding problem in the near term.
  arXiv  Detail & Related papers  (2022-04-04T20:09:50Z)
• {\lambda}-Scaled-Attention: A Novel Fast Attention Mechanism for Efficient Modeling of Protein Sequences [1.9744199942774268]
  Attention-based deep networks have been successfully applied on textual data in the field of NLP. Their application on protein sequences poses additional challenges due to the weak semantics of the protein words, unlike the plain text words. We introduce a novel lambda-scaled attention technique for fast and efficient modeling of the protein sequences.
  arXiv  Detail & Related papers  (2022-01-09T03:04:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.