ProtDBench: A Unified Benchmark of Protein Binder Design and Evaluation

• URL: http://arxiv.org/abs/2605.04118v1
• Date: Tue, 05 May 2026 11:48:59 GMT
• Title: ProtDBench: A Unified Benchmark of Protein Binder Design and Evaluation
• Abstract summary: We introduce ProtDBench, a standardized and throughput-aware evaluation framework for protein binder design.<n>ProtDBench defines unified benchmark tasks, evaluation protocols, and success criteria, enabling systematic analysis of how evaluation design influences observed performance.
• Score: 22.919793519533957
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in de novo protein binder design have enabled increasing experimental validation, yet reported in silico metrics remain difficult to interpret or compare across studies due to non-standardized evaluation protocols. We introduce ProtDBench, a standardized and throughput-aware evaluation framework for protein binder design. ProtDBench defines unified benchmark tasks, evaluation protocols, and success criteria, enabling systematic analysis of how evaluation design influences observed performance. Using a large wet-lab annotated dataset, we analyze commonly used structure prediction models as evaluation verifiers, revealing substantial verifier-dependent bias and limited agreement under identical filtering protocols. We then benchmark representative open-source generative binder design methods across ten diverse protein targets under a fixed evaluation protocol. Beyond per-sequence success rates, ProtDBench incorporates throughput-aware metrics based on a fixed 24-hour budget, as well as cluster-level success criteria to account for structural diversity. Together, these results expose systematic differences induced by filtering rules, success definitions, and throughput-aware evaluation between computational efficiency, success rate, and structural diversity. Overall, ProtDBench provides a fair and reproducible evaluation pipeline that supports systematic and controlled comparison of protein binder design methods under realistic evaluation settings.

Related papers

• Towards Standardized Light Field Quality Assessment: Hybrid Subjective Benchmarking and Objective Metric Evaluation [3.966528468818454]
  This paper presents a standardized workflow for light field QA, developed and deployed in the context of JPEG Pleno standardization activities.<n>The benchmark is designed to encompass not only traditional coding-only artifacts but also distortions that arise in processing pipelines.<n>A hybrid subjective method is proposed enabling fine-grained assessment by combining reference-anchored quality rating with targeted pairwise refinement.
  arXiv  Detail & Related papers  (2026-07-03T17:03:45Z)
• Picid: A Modular Evaluation Infrastructure for Reproducible PHM Across Tasks and Domains [13.382059615065167]
  We introduce picid, a modular evaluation infrastructure that formalizes the PHM evaluation pipeline as an explicit, executable, and reproducible protocol.<n>picid enforces deterministic, leakage-safe dataset construction while remaining flexible across diverse PHM settings.<n>We demonstrate picid through an empirical evaluation of thirteen models on twelve datasets spanning batteries, bearings, turbofan engines, hydraulics, filtration systems, and buildings.
  arXiv  Detail & Related papers  (2026-05-27T11:50:52Z)
• Interactive Evaluation Requires a Design Science [90.62695599188204]
  Large language models (LLMs) are increasingly deployed as systems that act over time through tools, environments, users, and other agents.<n>The field has begun to build interactive benchmarks, but the resulting landscape is fragmented.<n>This position paper argues that interactive evaluation should be treated as a principled evaluation paradigm.
  arXiv  Detail & Related papers  (2026-05-18T04:03:18Z)
• PepSpecBench: A Unified Evaluation Benchmark for Peptide Tandem Mass Spectrometry Prediction [17.33669468355787]
  PepSpecBench is a unified benchmark for peptide MS/MS spectrum prediction.<n>It standardizes data preprocessing across complementary public datasets.<n>It also enforces a strict backbone-disjoint splitting strategy to eliminate sequence leakage.
  arXiv  Detail & Related papers  (2026-05-03T16:11:27Z)
• UniDial-EvalKit: A Unified Toolkit for Evaluating Multi-Faceted Conversational Abilities [70.79422099851506]
  We present UniDial-EvalKit (UDE), a unified evaluation toolkit for assessing interactive AI systems.<n>UDE standardizes heterogeneous data formats into a universal schema, streamlines complex evaluation pipelines through a modular architecture, and aligns metric calculations under a consistent scoring interface.
  arXiv  Detail & Related papers  (2026-03-24T13:01:31Z)
• Procrustean Bed for AI-Driven Retrosynthesis: A Unified Framework for Reproducible Evaluation [0.0]
  RetroCast is a unified evaluation suite that standardizes heterogeneous model outputs into a common schema.<n>We evaluate leading search-based and sequence-based algorithms on a new suite of standardized benchmarks.
  arXiv  Detail & Related papers  (2025-12-08T01:26:39Z)
• PDFBench: A Benchmark for De novo Protein Design from Function [18.373430158468874]
  PDFBench is the first comprehensive benchmark for function-guided denovo protein design.<n>Our benchmark systematically evaluates eight state-of-the-art models on 16 metrics across two key settings.
  arXiv  Detail & Related papers  (2025-05-25T05:40:15Z)
• Benchmarking Synthetic Tabular Data: A Multi-Dimensional Evaluation Framework [0.4874819476581695]
  evaluating the quality of synthetic data remains a key challenge for ensuring privacy and utility in data-driven research.<n>We present a framework that quantifies how well synthetic data replicates original distributional properties while ensuring privacy.
  arXiv  Detail & Related papers  (2025-04-02T17:10:30Z)
• Learning to Align Multi-Faceted Evaluation: A Unified and Robust Framework [61.38174427966444]
  Large Language Models (LLMs) are being used more and more extensively for automated evaluation in various scenarios.<n>Previous studies have attempted to fine-tune open-source LLMs to replicate the evaluation explanations and judgments of powerful proprietary models.<n>We propose a novel evaluation framework, ARJudge, that adaptively formulates evaluation criteria and synthesizes both text-based and code-driven analyses.
  arXiv  Detail & Related papers  (2025-02-26T06:31:45Z)
• ProteinBench: A Holistic Evaluation of Protein Foundation Models [53.59325047872512]
  We introduce ProteinBench, a holistic evaluation framework for protein foundation models. Our approach consists of three key components: (i) A taxonomic classification of tasks that broadly encompass the main challenges in the protein domain, based on the relationships between different protein modalities; (ii) A multi-metric evaluation approach that assesses performance across four key dimensions: quality, novelty, diversity, and robustness; and (iii) In-depth analyses from various user objectives, providing a holistic view of model performance.
  arXiv  Detail & Related papers  (2024-09-10T06:52:33Z)
• A Backdoor-based Explainable AI Benchmark for High Fidelity Evaluation of Attributions [60.06461883533697]
  We first identify a set of fidelity criteria that reliable benchmarks for attribution methods are expected to fulfill.<n>We then introduce a Backdoor-based eXplainable AI benchmark (BackX) that adheres to the desired fidelity criteria.<n>Our analysis also offers insights into defending against neural Trojans by utilizing the attributions.
  arXiv  Detail & Related papers  (2024-05-02T13:48:37Z)
• PDB-Struct: A Comprehensive Benchmark for Structure-based Protein Design [19.324059406159325]
  We introduce two novel metrics: refoldability-based metric and stability-based metric. ByProt, ProteinMPNN, and ESM-IF perform exceptionally well on our benchmark, while ESM-Design and AF-Design fall short. Our proposed benchmark paves the way for a fair and comprehensive evaluation of protein design methods.
  arXiv  Detail & Related papers  (2023-11-30T02:37:55Z)
• Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries [59.27273928454995]
  Current pre-trained models applied to summarization are prone to factual inconsistencies which misrepresent the source text or introduce extraneous information. We create a crowdsourcing evaluation framework for factual consistency using the rating-based Likert scale and ranking-based Best-Worst Scaling protocols. We find that ranking-based protocols offer a more reliable measure of summary quality across datasets, while the reliability of Likert ratings depends on the target dataset and the evaluation design.
  arXiv  Detail & Related papers  (2021-09-19T19:05:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.