onepot-Bench 0: towards lab-aware in silico chemistry benchmarks

• URL: http://arxiv.org/abs/2608.02595v1
• Date: Mon, 03 Aug 2026 17:58:27 GMT
• Title: onepot-Bench 0: towards lab-aware in silico chemistry benchmarks
• Abstract summary: onepot-Bench 0 is a benchmark suite for evaluating language models on synthetic chemistry capabilities relevant to wet-lab execution.<n>Onepot-Bench 0 comprises three complementary evaluations: ChemAbacus measures tool-free cheminformatics literacy and numerical reasoning.<n> SynthRefusal characterizes safety and refusal behavior across a variety of benign, controlled, and designer-drug targets.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Language models are playing an increasingly important role in laboratory science, performing tasks such as experiment planning, execution, and post-hoc analysis. However, precisely measuring their abilities is difficult, as scientific capabilities require a mixture of both problem-solving skills and domain-specific intuition. Existing evaluations rarely measure the capabilities required to make reliable decisions in a physical laboratory and often rely on public data that may have appeared in model training corpora. We introduce onepot-Bench 0, a proprietary benchmark suite for evaluating language models on synthetic chemistry capabilities relevant to wet-lab execution. onepot-Bench 0 comprises three complementary evaluations: ChemAbacus measures tool-free cheminformatics literacy and numerical reasoning; SynthRefusal characterizes safety and refusal behavior across a variety of benign, controlled, and designer-drug targets; and SynthBench evaluates reaction-outcome prediction and catalyst selection using private experimental data generated in our laboratory. Together, these evaluations probe basic competency, reliability, and deeper knowledge, all skills which are required for reliable performance in the lab.

Related papers

• LabDex: A Hierarchical Benchmark for Dexterous Manipulation in Laboratories [40.311566677309536]
  LabDex is a large-scale real-world dataset and benchmark for dexterous manipulation in chemistry laboratories.<n>It is organized around a hierarchical task taxonomy spanning atomic skills, compositional tasks, and long-horizon experiments.<n>We conduct cross-level evaluations of representative robot learning methods in both real-world and simulation environments.
  arXiv  Detail & Related papers  (2026-08-19T07:11:06Z)
• Labimus: A Simulation and Benchmark for Humanoid Dexterous Manipulation in Chemical Laboratory [47.77200587195896]
  Labimus is the first benchmark for humanoid dexterous manipulation in organic chemistry laboratories.<n>It reconstructs over 30 functionally faithful assets from real organic chemistry workstations through real-to-sim modeling.<n>We introduce a precision-aware evaluation protocol designed to jointly measure task completion, experimental precision, and long-horizon execution.
  arXiv  Detail & Related papers  (2026-06-30T02:14:28Z)
• LabBuilder: Protocol-Grounded 3D Layout Generation for Interactable and Safe Laboratory [93.63620478256063]
  We present LabBuilder, an end-to-end system that generates and verifies 3D laboratory layouts from concise textual specifications.<n>LabBuilder significantly outperforms existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-05-04T07:24:56Z)
• BABE: Biology Arena BEnchmark [51.53220868983288]
  BABE is a benchmark designed to evaluate the experimental reasoning capabilities of biological AI systems.<n>Our benchmark provides a robust framework for assessing how well AI systems can reason like practicing scientists.
  arXiv  Detail & Related papers  (2026-02-05T16:39:20Z)
• AInsteinBench: Benchmarking Coding Agents on Scientific Repositories [33.48206557020983]
  AInsteinBench is a large-scale benchmark for evaluating whether large language model (LLM) agents can operate as scientific computing development agents.<n>AInsteinBench measures a model's ability to move beyond surface-level code generation toward the core competencies required for computational scientific research.
  arXiv  Detail & Related papers  (2025-12-24T08:11:11Z)
• LabUtopia: High-Fidelity Simulation and Hierarchical Benchmark for Scientific Embodied Agents [103.65422553044816]
  LabUtopia is a comprehensive simulation and benchmarking suite designed to facilitate the development of generalizable, reasoning-capable embodied agents.<n>It supports 30 distinct tasks and includes more than 200 scene and instrument assets.<n>We demonstrate that LabUtopia offers a powerful platform for advancing the integration of perception, planning, and control in scientific-purpose agents.
  arXiv  Detail & Related papers  (2025-05-28T17:50:53Z)
• Probing the limitations of multimodal language models for chemistry and materials research [3.422786943576035]
  We introduce MaCBench, a benchmark for evaluating how vision-language models handle real-world chemistry and materials science tasks.<n>We find that while these systems show promising capabilities in basic perception tasks, they exhibit fundamental limitations in spatial reasoning, cross-modal information synthesis, and logical inference.<n>Our insights have important implications beyond chemistry and materials science, suggesting that developing reliable multimodal AI scientific assistants may require advances in curating suitable training data and approaches to training those models.
  arXiv  Detail & Related papers  (2024-11-25T21:51:45Z)
• LabSafety Bench: Benchmarking LLMs on Safety Issues in Scientific Labs [78.99703366417661]
  Large language models (LLMs) increasingly assist in tasks ranging from procedural guidance to autonomous experiment orchestration.<n>Such overreliance is particularly dangerous in high-stakes laboratory settings, where failures in hazard identification or risk assessment can result in severe accidents.<n>We propose the Laboratory Safety Benchmark (LabSafety Bench) to evaluate models on their ability to identify potential hazards, assess risks, and predict the consequences of unsafe actions in lab environments.
  arXiv  Detail & Related papers  (2024-10-18T05:21:05Z)
• ChemVise: Maximizing Out-of-Distribution Chemical Detection with the Novel Application of Zero-Shot Learning [60.02503434201552]
  This research proposes learning approximations of complex exposures from training sets of simple ones. We demonstrate this approach to synthetic sensor responses surprisingly improves the detection of out-of-distribution obscured chemical analytes.
  arXiv  Detail & Related papers  (2023-02-09T20:19:57Z)
• Metrics for Benchmarking and Uncertainty Quantification: Quality, Applicability, and a Path to Best Practices for Machine Learning in Chemistry [0.0]
  This review aims to draw attention to two issues of concern when we set out to make machine learning benchmarking work in the chemical and materials domain. They are often overlooked or underappreciated topics as chemists typically only have limited training in statistics. These metrics are also key to comparing the performance of different models and thus for developing guidelines and best practices for the successful application of machine learning in chemistry.
  arXiv  Detail & Related papers  (2020-09-30T21:19:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.