SFBench: The SciFy Scientific Feasibility Benchmark

• URL: http://arxiv.org/abs/2606.29630v1
• Date: Sun, 28 Jun 2026 22:27:26 GMT
• Title: SFBench: The SciFy Scientific Feasibility Benchmark
• Abstract summary: We present SFBench, a benchmark dataset for evaluating systems that assess the feasibility of scientific claims.<n>SFBench includes 197 claims in materials science, each annotated with a ground-truth feasibility score on a five-point scale.
• Score: 3.9698458223568536
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present SFBench, a benchmark dataset for evaluating systems that assess the feasibility of scientific claims. SFBench includes 197 claims in materials science, each annotated with a ground-truth feasibility score on a five-point scale along with an explanation of that assessment. The collection differs from previous collections in several important ways: 1) it defines a complex task that requires reasoning over claims of varying scientific feasibility; 2) its claims are not extracted from existing scientific publications but are created de novo, greatly reducing the chances that LLMs have trained on them; 3) claims and ground truth are established by subject matter experts, not by artificial intelligence; and 4) unlike many benchmarks that ask about question/answer pairs, provide multiple choice answers, or ask questions requiring short, fixed answers, SFBench explanations are completely open-ended. We describe the benchmark design, data creation process, and evaluation metrics, and we report baseline results using recent GPT models.

Related papers

• Are LLMs Ready for Scientific Discovery? A Capability-Oriented Benchmark for AI Scientists [5.947663261819682]
  We introduce SDABench, a benchmark that reorganizes evaluation around six capabilities (descriptive, exploratory, inferential, predictive, causal, and mechanistic) across five domains (Biology, Chemistry, Environment, Geography, Physics)<n>We find that descriptive models handle analysis well but degrade sharply on tasks requiring assumption selection, latent-process modeling, or mechanistic reasoning.
  arXiv  Detail & Related papers  (2026-07-13T04:39:40Z)
• SciCustom: A Framework for Custom Evaluation of Scientific Capabilities in Large Language Models [63.357710580101894]
  Large language models (LLMs) are increasingly applied to scientific research, yet existing evaluations often fail to reflect the fine-grained capabilities required in practice.<n>We propose a new framework named SciCustom to address the problem.<n>It enables the custom construction of benchmarks from large-scale scientific data to evaluate application-specific scientific capabilities in LLMs.
  arXiv  Detail & Related papers  (2026-05-19T04:41:43Z)
• SCICONVBENCH: Benchmarking LLMs on Multi-Turn Clarification for Task Formulation in Computational Science [3.9311288356229057]
  Large Language Models (LLMs) are increasingly deployed as scientific AI as- sistants.<n>We introduce SCICONVBENCH, a benchmark for multi- turn clarification in scientific task formulation.<n>Our benchmark pairs a structured task with a rubric-based evaluation framework.
  arXiv  Detail & Related papers  (2026-05-18T16:34:45Z)
• InfiniteScienceGym: An Unbounded, Procedurally-Generated Benchmark for Scientific Analysis [19.4536238905905]
  We present InfiniteScienceGym, a procedurally generated benchmark of scientific repositories paired with a verifiable question-answering task.<n>From a seed, the simulator deterministically generates a self-contained repository with realistic directory structure, files, and tabular data, and a privileged QA generator produces both answerable and unanswerable questions with exact ground truth.<n>We find that none achieve more than 45% accuracy overall, that recognizing unanswerable questions remains a major weakness, and that stronger models tend to use tools more effectively rather than simply consuming more tokens.
  arXiv  Detail & Related papers  (2026-04-14T18:23:02Z)
• RPC-Bench: A Fine-grained Benchmark for Research Paper Comprehension [65.81339691942757]
  RPC-Bench is a large-scale question-answering benchmark built from review-rebuttal exchanges of high-quality computer science papers.<n>We design a fine-grained taxonomy aligned with the scientific research flow to assess models' ability to understand and answer why, what, and how questions in scholarly contexts.
  arXiv  Detail & Related papers  (2026-01-14T11:37:00Z)
• Encyclo-K: Evaluating LLMs with Dynamically Composed Knowledge Statements [78.87065404966002]
  Existing benchmarks predominantly curate questions at the question level.<n>We propose Encyclo-K, a statement-based benchmark that rethinks benchmark construction from the ground up.
  arXiv  Detail & Related papers  (2025-12-31T13:55:54Z)
• ReFACT: A Benchmark for Scientific Confabulation Detection with Positional Error Annotations [14.392598503431321]
  Large Language Models (LLMs) frequently confabulate scientific facts, severely undermining their trustworthiness.<n>We introduce ReFACT, a benchmark of 1,001 expert-annotated question-answer pairs spanning diverse scientific domains.<n>Each instance includes both a scientifically correct answer and a non-factual counterpart annotated with precise error spans and error types.
  arXiv  Detail & Related papers  (2025-09-30T07:06:23Z)
• Who Gets Cited Most? Benchmarking Long-Context Language Models on Scientific Articles [81.89404347890662]
  SciTrek is a novel question-answering benchmark designed to evaluate the long-context reasoning capabilities of large language models (LLMs) using scientific articles.<n>Our analysis reveals systematic shortcomings in models' abilities to perform basic numerical operations and accurately locate specific information in long contexts.
  arXiv  Detail & Related papers  (2025-09-25T11:36:09Z)
• SAS-Bench: A Fine-Grained Benchmark for Evaluating Short Answer Scoring with Large Language Models [36.10798324093408]
  SAS-Bench is a benchmark for large language models (LLMs) based Short Answer Scoring tasks.<n>It provides fine-grained, step-wise scoring, expert-annotated error categories, and a diverse range of question types.<n>We also release an open-source dataset containing 1,030 questions and 4,109 student responses, each annotated by domain experts.
  arXiv  Detail & Related papers  (2025-05-12T05:43:21Z)
• SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretation [50.061029816288936]
  We present SciFIBench, a scientific figure interpretation benchmark consisting of 2000 questions split between two tasks across 8 categories.<n>The questions are curated from arXiv paper figures and captions, using adversarial filtering to find hard negatives and human verification for quality control.<n>We evaluate 28 LMMs on SciFIBench, finding it to be a challenging benchmark.
  arXiv  Detail & Related papers  (2024-05-14T17:54:17Z)
• SCITAB: A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tables [68.76415918462418]
  We present SCITAB, a challenging evaluation dataset consisting of 1.2K expert-verified scientific claims. Through extensive evaluations, we demonstrate that SCITAB poses a significant challenge to state-of-the-art models. Our analysis uncovers several unique challenges posed by SCITAB, including table grounding, claim ambiguity, and compositional reasoning.
  arXiv  Detail & Related papers  (2023-05-22T16:13:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.