ProtLingo: Efficient Protein Language Modeling via Conditional Memory and Expert Routing

• URL: http://arxiv.org/abs/2609.04793v1
• Date: Fri, 04 Sep 2026 06:46:42 GMT
• Title: ProtLingo: Efficient Protein Language Modeling via Conditional Memory and Expert Routing
• Abstract summary: Protein language models (PLMs) provide a scalable approach for modeling sequence-function relationships from unlabeled sequences.<n>We introduce ProtLingo, an efficient PLM framework that augments a pretrained single-sequence backbone with conditional local memory and sparse expert routing.<n>ProtLingo achieves competitive performance with a 150M-scale backbone, including strong parameter efficiency on mutation-effect prediction.
• Score: 39.80134572747517
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Proteins perform diverse cellular functions, and even single amino-acid substitutions can alter stability, activity, or molecular interactions. Protein language models (PLMs) provide a scalable approach for modeling such sequence--function relationships from unlabeled sequences, but increasing the size of dense Transformer backbones often brings substantial computational cost without consistently improving mutation-sensitive prediction. We introduce ProtLingo, an efficient PLM framework that augments a pretrained single-sequence backbone with conditional local memory and sparse expert routing. ProtLingo maps contextual residue representations into route-specific discrete codes, composes centered local windows into latent $N$-gram addresses, and retrieves reusable residual signals associated with recurring local sequence contexts. In parallel, selected feed-forward blocks are upcycled into sparse Mixture-of-Experts layers with shared and routed experts, enabling residue-dependent computation while activating only a subset of parameters. Experiments on protein fitness prediction, FLIP benchmarks, and supervised contact prediction show that ProtLingo achieves competitive performance with a 150M-scale backbone, including strong parameter efficiency on mutation-effect prediction and preserved long-range structural representations.

Related papers

• Multi-Objective Molecular Generation with Frequency-Controlled Evolutionary Dynamics [0.0]
  SpectralMol is an algorithm that processes chemical structures as a compact matrix of Fourier coefficients.<n>It generates more docking hits and more diverse scaffolds while maintaining competitive physicochemical properties.<n>Results support the evidence that frequency-controlled evolutionary dynamics provide an interpretable, efficient, and training-free route to multi-objective molecular design.
  arXiv  Detail & Related papers  (2026-06-25T18:42:07Z)
• Contextualizing Biological Language Models across Modalities via Logit-Space Contrastive Alignment [2.249367818885166]
  LOGICA is a framework for context-conditioned prediction that performs contrastive learning directly in output-logit space.<n>It is particularly effective for mutation-local variant ranking, where comparisons reduce to context-conditioned likelihoods of mutant tokens at perturbed sites.
  arXiv  Detail & Related papers  (2026-06-17T05:30:47Z)
• PROTOCOL: Late Interaction Retrieval for Protein Homolog Search [8.797956246984388]
  ProtoCol is a model which represents proteins as sets of residue embeddings.<n>It encodes proteins independently, keeps candidate representations pre-computable, and scores candidates with MaxSim over residue embeddings.<n>On SCOPe superfamily and Pfam clan benchmarks, ProtoCol outperforms sequence-composition, alignment-based, pooled PLM, and trained single-vector baselines.
  arXiv  Detail & Related papers  (2026-05-27T22:50:48Z)
• scHelix: Asymmetric Dual-Stream Integration via Explicit Gene-Level Disentanglement [84.88065404629079]
  scHelix is a dataset-adaptive framework that fundamentally changes how features are processed.<n> scHelix utilizes a dual-stream sparse diffusion encoder equipped with stop-gradient graph caching.<n> scHelix outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-05-18T15:55:13Z)
• Efficient Protein Optimization via Structure-aware Hamiltonian Dynamics [16.336540408998598]
  HADES is a Bayesian optimization method utilizing Hamiltonian dynamics to efficiently sample from a structure-aware approximated posterior.<n>A position discretization procedure is introduced to propose discrete protein sequences from such a continuous state system.<n>Experiments demonstrate that our method outperforms state-of-the-art baselines in in-silico evaluations.
  arXiv  Detail & Related papers  (2026-01-16T05:53:53Z)
• S$^2$Drug: Bridging Protein Sequence and 3D Structure in Contrastive Representation Learning for Virtual Screening [72.89086338778098]
  We propose a two-stage framework for protein-ligand contrastive representation learning.<n>In the first stage, we perform protein sequence pretraining on ChemBL using an ESM2-based backbone.<n>In the second stage, we fine-tune on PDBBind by fusing sequence and structure information through a residue-level gating module.<n>This auxiliary task guides the model to accurately localize binding residues within the protein sequence and capture their 3D spatial arrangement.
  arXiv  Detail & Related papers  (2025-11-10T11:57:47Z)
• SeqProFT: Applying LoRA Finetuning for Sequence-only Protein Property Predictions [8.112057136324431]
  This study employs the LoRA method to perform end-to-end fine-tuning of the ESM-2 model. A multi-head attention mechanism is integrated into the downstream network to combine sequence features with contact map information.
  arXiv  Detail & Related papers  (2024-11-18T12:40:39Z)
• Pre-training Co-evolutionary Protein Representation via A Pairwise Masked Language Model [93.9943278892735]
  Key problem in protein sequence representation learning is to capture the co-evolutionary information reflected by the inter-residue co-variation in the sequences. We propose a novel method to capture this information directly by pre-training via a dedicated language model, i.e., Pairwise Masked Language Model (PMLM) Our result shows that the proposed method can effectively capture the interresidue correlations and improves the performance of contact prediction by up to 9% compared to the baseline.
  arXiv  Detail & Related papers  (2021-10-29T04:01:32Z)
• EBM-Fold: Fully-Differentiable Protein Folding Powered by Energy-based Models [53.17320541056843]
  We propose a fully-differentiable approach for protein structure optimization, guided by a data-driven generative network. Our EBM-Fold approach can efficiently produce high-quality decoys, compared against traditional Rosetta-based structure optimization routines.
  arXiv  Detail & Related papers  (2021-05-11T03:40:29Z)
• Spatially Adaptive Inference with Stochastic Feature Sampling and Interpolation [72.40827239394565]
  We propose to compute features only at sparsely sampled locations. We then densely reconstruct the feature map with an efficient procedure. The presented network is experimentally shown to save substantial computation while maintaining accuracy over a variety of computer vision tasks.
  arXiv  Detail & Related papers  (2020-03-19T15:36:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.