PFMBench: Protein Foundation Model Benchmark

• URL: http://arxiv.org/abs/2506.14796v1
• Date: Sun, 01 Jun 2025 07:40:07 GMT
• Title: PFMBench: Protein Foundation Model Benchmark
• Authors: Zhangyang Gao, Hao Wang, Cheng Tan, Chenrui Xu, Mengdi Liu, Bozhen Hu, Linlin Chao, Xiaoming Zhang, Stan Z. Li,
• Abstract summary: PFMBench is a benchmark evaluating protein foundation models across 38 tasks spanning 8 key areas of protein science.<n>It reveals the inherent correlations between tasks, identifies top-performing models, and provides a streamlined evaluation protocol.
• Score: 42.418536890859635
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This study investigates the current landscape and future directions of protein foundation model research. While recent advancements have transformed protein science and engineering, the field lacks a comprehensive benchmark for fair evaluation and in-depth understanding. Since ESM-1B, numerous protein foundation models have emerged, each with unique datasets and methodologies. However, evaluations often focus on limited tasks tailored to specific models, hindering insights into broader generalization and limitations. Specifically, researchers struggle to understand the relationships between tasks, assess how well current models perform across them, and determine the criteria in developing new foundation models. To fill this gap, we present PFMBench, a comprehensive benchmark evaluating protein foundation models across 38 tasks spanning 8 key areas of protein science. Through hundreds of experiments on 17 state-of-the-art models across 38 tasks, PFMBench reveals the inherent correlations between tasks, identifies top-performing models, and provides a streamlined evaluation protocol. Code is available at \href{https://github.com/biomap-research/PFMBench}{\textcolor{blue}{GitHub}}.

Related papers

• Protein-SE(3): Benchmarking SE(3)-based Generative Models for Protein Structure Design [35.87227562237066]
  SE(3)-based generative models have shown great promise in protein geometry modeling and effective structure design.<n>Protein-SE(3), a new benchmark based on a unified training framework, comprises protein scaffolding tasks, integrated generative models, high-level mathematical abstraction, and diverse evaluation metrics.
  arXiv  Detail & Related papers  (2025-07-27T11:53:05Z)
• PRING: Rethinking Protein-Protein Interaction Prediction from Pairs to Graphs [80.08310253195144]
  PRING is the first benchmark that evaluates protein-protein interaction prediction from a graph-level perspective.<n> PRING curates a high-quality, multi-species PPI network dataset comprising 21,484 proteins and 186,818 interactions.
  arXiv  Detail & Related papers  (2025-07-07T15:21:05Z)
• VenusX: Unlocking Fine-Grained Functional Understanding of Proteins [3.9151216020893544]
  VenusX is the first large-scale benchmark for fine-grained functional annotation and function-based protein pairing at the residue, fragment, and domain levels.<n>VenusX comprises three major task categories, including identifying critical active sites, binding sites, conserved sites motifs, domains, and pairwises.<n>Our benchmark enables a comprehensive assessment of model performance on both in-distribution and out-of-distribution scenarios.
  arXiv  Detail & Related papers  (2025-05-17T03:44:23Z)
• Multi-Scale Representation Learning for Protein Fitness Prediction [31.735234482320283]
  Previous methods have primarily relied on self-supervised models trained on vast, unlabeled protein sequence or structure datasets.<n>We introduce the Sequence-Structure-Surface Fitness (S3F) model - a novel multimodal representation learning framework that integrates protein features across several scales.<n>Our approach combines sequence representations from a protein language model with Geometric Vector Perceptron networks encoding protein backbone and detailed surface topology.
  arXiv  Detail & Related papers  (2024-12-02T04:28:10Z)
• ProteinBench: A Holistic Evaluation of Protein Foundation Models [53.59325047872512]
  We introduce ProteinBench, a holistic evaluation framework for protein foundation models. Our approach consists of three key components: (i) A taxonomic classification of tasks that broadly encompass the main challenges in the protein domain, based on the relationships between different protein modalities; (ii) A multi-metric evaluation approach that assesses performance across four key dimensions: quality, novelty, diversity, and robustness; and (iii) In-depth analyses from various user objectives, providing a holistic view of model performance.
  arXiv  Detail & Related papers  (2024-09-10T06:52:33Z)
• GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation Models [56.63218531256961]
  We introduce GenBench, a benchmarking suite specifically tailored for evaluating the efficacy of Genomic Foundation Models. GenBench offers a modular and expandable framework that encapsulates a variety of state-of-the-art methodologies. We provide a nuanced analysis of the interplay between model architecture and dataset characteristics on task-specific performance.
  arXiv  Detail & Related papers  (2024-06-01T08:01:05Z)
• Geometric Deep Learning for Structure-Based Drug Design: A Survey [83.87489798671155]
  Structure-based drug design (SBDD) leverages the three-dimensional geometry of proteins to identify potential drug candidates. Recent advancements in geometric deep learning, which effectively integrate and process 3D geometric data, have significantly propelled the field forward.
  arXiv  Detail & Related papers  (2023-06-20T14:21:58Z)
• Unravelling the Architecture of Membrane Proteins with Conditional Random Fields [11.321552104966326]
  We will show that the Conditional Random Fields (CRF) provides a template to integrate micro-level information about biological entities into a mathematical model to understand their macro-level behavior. A comparison on benchmark data sets against twenty-eight other methods shows that the CRF model leads to extremely accurate predictions.
  arXiv  Detail & Related papers  (2020-08-06T05:57:20Z)
• Rethinking Generalization of Neural Models: A Named Entity Recognition Case Study [81.11161697133095]
  We take the NER task as a testbed to analyze the generalization behavior of existing models from different perspectives. Experiments with in-depth analyses diagnose the bottleneck of existing neural NER models. As a by-product of this paper, we have open-sourced a project that involves a comprehensive summary of recent NER papers.
  arXiv  Detail & Related papers  (2020-01-12T04:33:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.