ProtSent: Protein Sentence Transformers

• URL: http://arxiv.org/abs/2605.06830v1
• Date: Thu, 07 May 2026 18:33:49 GMT
• Title: ProtSent: Protein Sentence Transformers
• Abstract summary: Protein language models (pLMs) produce per-residue representations that capture evolutionary and structural information.<n>We present Protein Sentence Transformers (ProtSent), a contrastive fine-tuning framework for adapting PLMs into general-purpose embedding models.
• Score: 6.657830371527077
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Protein language models (pLMs) produce per-residue representations that capture evolutionary and structural information, yet their mean-pooled sequence embeddings are not explicitly trained to reflect functional, evolutionary or structural similarity between proteins. We present Protein Sentence Transformers (ProtSent), a contrastive fine-tuning framework for adapting PLMs into general-purpose embedding models. ProtSent trains with MultipleNegativesRankingLoss across five protein-pair datasets: Pfam families, structurally derived hard negatives, AlphaFold DB structural pairs, and StringDB protein--protein interactions, and Deep Mutational Scanning data. We evaluate on 23~downstream tasks using frozen embeddings with a k-nearest-neighbor probe to measure embedding neighborhood quality. On ESM-2 150M, ProtSent improves 15 of 23 tasks, with gains of +105% on remote homology detection, +17% on variant effect prediction, and +19.9% Recall@1 on SCOPe-40 structural retrieval. The 35M variant improves 16 of 23 tasks with +40.5% on remote homology and +15.5% Recall@1 on SCOPe-40. Contrastive fine-tuning restructures the embedding space to better capture protein function and structure, without any task-specific supervision. We release the models, public data, and training recipe and code.

Related papers

• ZeroFold: Protein-RNA Binding Affinity Predictions from Pre-Structural Embeddings [7.1857665261026575]
  The accurate prediction of protein-RNA binding affinity remains an unsolved problem in structural biology.<n>Here, we show that this obstacle can be addressed by extracting pre-structural embeddings.<n>We build ZeroFold, a transformer-based model that combines pre-structural embeddings from Boltz-2 for both protein and RNA molecules.
  arXiv  Detail & Related papers  (2026-03-24T15:14:44Z)
• Rank-and-Reason: Multi-Agent Collaboration Accelerates Zero-Shot Protein Mutation Prediction [20.82228986381723]
  We propose Rank-and-Reason (VenusRAR), a two-stage agentic framework to automate the wet-lab validation process.<n>In the Rank-Stage, a Computational Expert and Virtual Biologist aggregate a context-aware multi-modal ensemble, establishing a new Spearman correlation record of 0.551 (vs. 0.518) on ProteinGym.<n>In the Reason-Stage, an agentic Expert Panel employs chain-of-thought reasoning to audit candidates against geometric and structural constraints, improving the Top-5 Hit Rate by up to 367% on ProteinGym-DMS99.
  arXiv  Detail & Related papers  (2026-01-30T10:35:46Z)
• S$^2$Drug: Bridging Protein Sequence and 3D Structure in Contrastive Representation Learning for Virtual Screening [72.89086338778098]
  We propose a two-stage framework for protein-ligand contrastive representation learning.<n>In the first stage, we perform protein sequence pretraining on ChemBL using an ESM2-based backbone.<n>In the second stage, we fine-tune on PDBBind by fusing sequence and structure information through a residue-level gating module.<n>This auxiliary task guides the model to accurately localize binding residues within the protein sequence and capture their 3D spatial arrangement.
  arXiv  Detail & Related papers  (2025-11-10T11:57:47Z)
• ProteinAE: Protein Diffusion Autoencoders for Structure Encoding [64.77182442408254]
  We introduce ProteinAE, a novel and streamlined protein diffusion autoencoder.<n>ProteinAE directly maps protein backbone coordinates from E(3) into a continuous, compact latent space.<n>We demonstrate that ProteinAE achieves state-of-the-art reconstruction quality, outperforming existing autoencoders.
  arXiv  Detail & Related papers  (2025-10-12T14:30:32Z)
• DISPROTBENCH: A Disorder-Aware, Task-Rich Benchmark for Evaluating Protein Structure Prediction in Realistic Biological Contexts [76.59606029593085]
  DisProtBench is a benchmark for evaluating protein structure prediction models (PSPMs) under structural disorder and complex biological conditions.<n>DisProtBench spans three key axes: data complexity, task diversity, and Interpretability.<n>Results reveal significant variability in model robustness under disorder, with low-confidence regions linked to functional prediction failures.
  arXiv  Detail & Related papers  (2025-06-18T23:58:22Z)
• ProteinZero: Self-Improving Protein Generation via Online Reinforcement Learning [49.2607661375311]
  We present ProteinZero, a novel framework that enables computationally scalable, automated, and continuous self-improvement of the inverse folding model.<n>ProteinZero substantially outperforms existing methods across every key metric in protein design.<n> Notably, the entire RL run on CATH-4.3 can be done with a single 8 X GPU node in under 3 days, including reward.
  arXiv  Detail & Related papers  (2025-06-09T06:08:59Z)
• ProtCLIP: Function-Informed Protein Multi-Modal Learning [18.61302416993122]
  We develop ProtCLIP, a multi-modality foundation model that represents function-aware protein embeddings.<n>Our ProtCLIP consistently achieves SOTA performance, with remarkable improvements of 75% on average in five cross-modal transformation benchmarks.<n>The experimental results verify the extraordinary potential of ProtCLIP serving as the protein multi-modality foundation model.
  arXiv  Detail & Related papers  (2024-12-28T04:23:47Z)
• Structure-informed Language Models Are Protein Designers [69.70134899296912]
  We present LM-Design, a generic approach to reprogramming sequence-based protein language models (pLMs) We conduct a structural surgery on pLMs, where a lightweight structural adapter is implanted into pLMs and endows it with structural awareness. Experiments show that our approach outperforms the state-of-the-art methods by a large margin.
  arXiv  Detail & Related papers  (2023-02-03T10:49:52Z)
• On the Robustness of AlphaFold: A COVID-19 Case Study [16.564151738086434]
  We demonstrate that AlphaFold does not exhibit such robustness despite its high accuracy. This raises the challenge of detecting and quantifying the extent to which these predicted protein structures can be trusted.
  arXiv  Detail & Related papers  (2023-01-10T17:31:39Z)
• State-specific protein-ligand complex structure prediction with a multi-scale deep generative model [68.28309982199902]
  We present NeuralPLexer, a computational approach that can directly predict protein-ligand complex structures. Our study suggests that a data-driven approach can capture the structural cooperativity between proteins and small molecules, showing promise in accelerating the design of enzymes, drug molecules, and beyond.
  arXiv  Detail & Related papers  (2022-09-30T01:46:38Z)
• Pre-training Protein Language Models with Label-Agnostic Binding Pairs Enhances Performance in Downstream Tasks [1.452875650827562]
  Less than 1% of protein sequences are structurally and functionally annotated. We present a modification to the RoBERTa model by inputting a mixture of binding and non-binding protein sequences. We suggest that Transformer's attention mechanism contributes to protein binding site discovery.
  arXiv  Detail & Related papers  (2020-12-05T17:37:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.