FrontierChallenge: Evaluating Scientific Workflow Completion

• URL: http://arxiv.org/abs/2608.24979v1
• Date: Tue, 25 Aug 2026 16:50:58 GMT
• Title: FrontierChallenge: Evaluating Scientific Workflow Completion
• Abstract summary: We introduce FrontierChallenge, a cross-domain benchmark comprising 300 end-to-end scientific deliverables.<n>We release and evaluate 97 of these tasks, spanning quantum chemistry, molecular dynamics, materials characterization, analytical chemistry, life science, and electrochemistry/environment.<n>We find that neither high partial scores nor confident claims of completion reliably indicate that a scientific task has been fully delivered.
• Score: 59.601983661596286
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Scientific agents increasingly analyze data, execute code, and produce research artifacts, yet most benchmarks emphasize final answers, isolated programs, or a single domain. We introduce FrontierChallenge, a cross-domain benchmark comprising 300 end-to-end scientific workflows. In this paper, we release and evaluate 97 of these tasks, spanning quantum chemistry, molecular dynamics, materials characterization, analytical chemistry, life science, and electrochemistry/environment. Each task provides fixed inputs and specifies a bundle of required scientific deliverables. We evaluate twelve frontier models with three agent scaffolds. Pass Rate measures the fraction of tasks satisfying the full-completion criterion, while Avg. Score captures partial progress. Each of the best-performing configurations completed only 20 of the 97 released tasks, yielding a Pass Rate of 20.6%. Partial progress translated especially poorly into complete delivery in analytical chemistry and electrochemistry/environment: Avg. Scores reached 87.6 and 94.9, but the highest Pass Rates were only 4% and 0%. Among non-passing Claude Code trajectories, 75.5% still ended with language claiming completion. These findings show that neither high partial scores nor confident claims of completion reliably indicate that a scientific task has been fully delivered, highlighting the need to evaluate end-to-end workflow execution and the completeness of scientific deliverables together.

Related papers

• SCIRIGOR:Evaluating Open-Ended Scientific Analysis Beyond Final Scores [11.097584790037454]
  We formulate evidence-grounded multimodal scientific analysis, requiring agents to produce executable analyses and claims supported by results and visualizations from the same run.<n>We introduce SciRIGOR, an evaluation framework and benchmark comprising 100 cases from scientific articles across six domains and 17 subfields.<n>On full-benchmark runs, claims agree with faithful and unfaithful results at nearly identical rates (91.8% versus 91.0%). Yet no system exceeds 62.6% on the soft evidence-chain score or 18.0% strict whole-chain success.
  arXiv  Detail & Related papers  (2026-09-05T17:29:15Z)
• Efficient Test-Time Adaptation through Human-AI Interaction [112.5131603022122]
  We propose test-time adaptation through human-agent interaction (TAHI)<n>We adapt agents to 30 individuals in two high-utility domains, writing and visual creation, on a total of 600 tasks.<n>Our agents improve solo task success by 4.5-20.9% within only tens of tasks.
  arXiv  Detail & Related papers  (2026-09-03T17:33:18Z)
• BixBench3: Benchmarking AI agents on research-study-scale computational biology tasks [1.1546532378724035]
  We introduce Bix3, a benchmark that measures capacity of AI agents to process biological data through to scientific results.<n>13 models achieve scores ranging from 0.00 for Gemini 3.1 Flash Lite to 0.48 for GPT 5.6 Sol.<n>On average, agents use 6.8 hours tokens, 102 million, and $43 complete each task, and $43 to the longest attempts consuming 24 hours.
  arXiv  Detail & Related papers  (2026-08-26T01:45:55Z)
• EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards [73.12811119273206]
  We introduce EarthVerse, a benchmark that evaluates scientific agents through package-scoped investigations.<n>Agents inspect heterogeneous event packages, choose compatible evidence, execute transparent calculations, reconcile source differences, and preserve provenance in the final answer.<n>Across systems, the best mean answer-unit accuracy is 84.65%, while the highest Strict@95 is only 34.81%.
  arXiv  Detail & Related papers  (2026-08-24T17:29:16Z)
• CAi Copilot: Reducing Operational Workload in Molecular Design through Intent-Driven Agentic Workflows [13.08147687719993]
  Early-stage molecular design is not just a task of generating molecules.<n>Researchers turn broad goals into design strategies, refine candidates, assess many properties, and gather evidence before synthesis and tests.<n> AI methods can generate molecules, optimize several goals, predict properties, dock compounds, and account for synthesis.<n>We present CAi Copilot, an expert-oriented agent with three linked layers.
  arXiv  Detail & Related papers  (2026-08-07T08:36:59Z)
• ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research [142.29356526274387]
  We present ResearchClawBench, a benchmark for evaluating autonomous scientific research.<n>Each task is grounded in a real published paper, provides related literature and raw data, and hides the target paper during evaluation.
  arXiv  Detail & Related papers  (2026-05-28T16:27:40Z)
• ScientistOne: Towards Human-Level Autonomous Research via Chain-of-Evidence [57.37494162084001]
  Chain-of-Evidence (CoE) is a verifiability framework requiring every claim to be traceable to its evidence source.<n>CoE Audit is a post-hoc audit whose four integrity checks -- score verification, specification violation, reference verification, and method-code alignment -- apply uniformly to all systems.
  arXiv  Detail & Related papers  (2026-05-25T21:30:27Z)
• EnactToM: An Evolving Benchmark for Functional Theory of Mind in Embodied Agents [75.01735520608075]
  Existing benchmarks mostly test literal Theory of Mind (ToM) by asking direct belief questions.<n>We introduce EnactToM, an evolving benchmark of 300 embodied multi-agent tasks set in a 3D household.<n>On the hard split, all seven evaluated frontier models score 0.0% Pass3 on functional task completion, while averaging 45.0% on literal belief probes.
  arXiv  Detail & Related papers  (2026-05-11T00:04:19Z)
• BioResearcher: Scenario-Guided Multi-Agent for Translational Medicine [1.497336135403194]
  Ingenix BioResearcher is a scenario-guided multi-agent system that maps queries to versioned research playbooks.<n>We evaluate BioResearcher across unit-level capabilities, open-ended biomedical reasoning, and end-to-end clinical discovery.
  arXiv  Detail & Related papers  (2026-05-07T10:33:43Z)
• COMPOSITE-Stem [32.17652486099204]
  COMPOSITE-STEM is a benchmark of 70 expert-written tasks in physics, biology, chemistry, and mathematics.<n>Our benchmark combines exact-match grading and criterion-based rubrics with an-as-a-jury grading protocol.<n>The top-performing model achieves 21%, demonstrating that COMPOSITE-STEM captures capabilities beyond current agent reach.
  arXiv  Detail & Related papers  (2026-04-10T19:08:50Z)
• PRBench: End-to-end Paper Reproduction in Physics Research [32.672534450424386]
  PRBench is a benchmark of 30 expert-curated tasks spanning 11 subfields of physics.<n>Agents are provided only with the task instruction and paper content, and operate in a sandboxed execution environment.<n>We evaluate a set of coding agents on PRBench and analyze their capabilities across key dimensions of scientific reasoning and execution.
  arXiv  Detail & Related papers  (2026-03-29T11:44:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.