ToolGate: An Executable Acceptance Pipeline for Tool-Dependent Scientific Benchmark Construction

• URL: http://arxiv.org/abs/2609.02067v1
• Date: Wed, 02 Sep 2026 03:49:37 GMT
• Title: ToolGate: An Executable Acceptance Pipeline for Tool-Dependent Scientific Benchmark Construction
• Abstract summary: We present ToolGate, which treats every generated item as a proposal and keeps it only if three gates pass.<n>ToolGate turns repeated answer checking and difficulty screening into an auditable process while leaving domain design and final review to experts.
• Score: 5.992855612201743
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Scientific benchmarks are commonly built by domain experts who write tasks and cross-check one another's work, or who adapt existing material from textbooks, published papers, and online resources. These routes can produce strong evaluations, but they require substantial per-item labor. Language models can reduce this repeated work by proposing candidates quickly. The remaining problem is acceptance. We target scientific questions whose answers require computations with specialist software rather than unaided reasoning alone. A candidate is invalid if its script fails or returns a different answer, or trivial if a model answers it without the software. We present ToolGate, which treats every generated item as a proposal and keeps it only if three gates pass. First, an executable solution script must reproduce the proposed answer when run with the scientific software. Second, randomized no-tool screening rejects candidates that models can already solve from the prompt alone. Third, a tool-using agent must solve each survivor within a fixed time limit. We instantiate ToolGate in FEniCSx with 500 generation attempts. The local-verification gate retains 478 candidates. For final reporting, we rescreen this pool after generation: two randomized no-tool screens exclude 222 from the reported pool, and direct GPT-5.5 API calls at medium reasoning (the API default) exclude another 121. Of the remaining 135, a GPT-5.5 Codex CLI agent with access to FEniCSx solves 130; exact deduplication leaves 128 unique protocol survivors. ToolGate turns repeated answer checking and difficulty screening into an auditable process while leaving domain design and final review to experts.

Related papers

• Kozuchi Agent: A Language-Agnostic Open-Weight Agent for Software Repair [13.668549452625117]
  Kozuchi Agent is a language-agnostic open-weight repair agent and CI-operated evaluation pipeline.<n>Explicit phases, persistent state, deterministic tools, a model-independent action interface, and cross-agent test-time selection make runs auditable and repeatable.
  arXiv  Detail & Related papers  (2026-08-16T07:06:37Z)
• Apodex Discovery: Reality Benchmarks and Environments for Evaluating and Building Discoverative Artificial Intelligence [86.61561123749011]
  We introduce Apodex Discovery, a framework for building and evaluating discoverative AI through the heavy-duty solver.<n>First, a problem-scouting process surveyed 561 industries across 16 sectors, assembled 423 high-value real-world problems, and selected 20 for the initial release.<n>Second, a common environment-task-episode abstraction provides data, tools, constraints, feedback, trajectory recording, and verification of intermediate artifacts and final submissions.
  arXiv  Detail & Related papers  (2026-08-11T18:48:40Z)
• A Constrained Natural-Language Interface for Variational Multi-Physics Finite Element Simulations in FEniCS [0.0]
  We present a constrained natural-language generated finite element analysis in which the LLM is limited to front-end tasks.<n>It never writes FEniCS solver templates, derives weak forms, or writes the numerical solver core.<n>As an end-to-end demonstration, the system generates and analyzes a 3D elastoplastic L-bracket with a fillet and bolt hole from one natural-language prompt.
  arXiv  Detail & Related papers  (2026-06-09T14:38:11Z)
• Verus-SpecGym: An Agentic Environment for Evaluating Specification Autoformalization [26.123396123145415]
  We study specification autoformalization: whether LLM agents can translate informal programming problems into faithful formal specifications.<n>We introduce Verus-SpecBench, a benchmark of 581 spec-writing tasks derived from Codeforces problems.<n>Our analysis of failure modes shows that model-generated specs can accept important input assumptions, accept incorrect outputs, & reject valid ones.
  arXiv  Detail & Related papers  (2026-05-26T02:12:48Z)
• ProgramBench: Can Language Models Rebuild Programs From Scratch? [59.40748183470308]
  ProgramBench measures the ability of software engineering agents to develop software holisitically.<n>End-to-end behavioral tests are generated via agent-driven fuzzing.<n>Models favor monolithic, single-file implementations that diverge sharply from human-written code.
  arXiv  Detail & Related papers  (2026-05-05T09:17:02Z)
• Theory Under Construction: Orchestrating Language Models for Research Software Where the Specification Evolves [1.0312968200748116]
  Large language models can generate substantial code and draft research text, but research-software projects require more than either artifact alone.<n>We identify two LM-specific failure modes: hallucination accumulation and desynchronization.<n>We propose Comet-H, an iterative prompt that orchestrates ideation, implementation, evaluation, grounding, and paper-writing as coordinates of a single workspace state.
  arXiv  Detail & Related papers  (2026-04-29T21:28:17Z)
• ARGOS: Who, Where, and When in Agentic Multi-Camera Person Search [49.61286310968402]
  We introduce ARGOS, the first benchmark and framework that reformulates multi-camera person search as an interactive reasoning problem.<n>An ARGOS agent receives a vague witness statement and must decide what to ask, when to invoke spatial or temporal tools, and how to interpret ambiguous responses.<n>The benchmark comprises 2,691 tasks across 14 real-world scenarios in three progressive tracks: semantic perception (Who), spatial reasoning (Where), and temporal reasoning (When)
  arXiv  Detail & Related papers  (2026-04-14T14:06:19Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• UQ: Assessing Language Models on Unsolved Questions [149.46593270027697]
  We introduce UQ, a testbed of 500 challenging, diverse questions sourced from Stack Exchange.<n>UQ is difficult and realistic by construction: unsolved questions are often hard and naturally arise when humans seek answers.<n>The top model passes UQ-validation on only 15% of questions, and preliminary human verification has already identified correct answers.
  arXiv  Detail & Related papers  (2025-08-25T01:07:59Z)
• Self-Questioning Language Models [58.73276539661649]
  We propose an asymmetric self-play framework where a proposer is given the topic and generates a question for a solver.<n>Both the proposer and solver are trained via reinforcement learning.<n>We study this asymmetric self-play framework on three benchmarks: three-digit multiplication, algebra problems from the OMEGA benchmark, and programming problems from Codeforces.
  arXiv  Detail & Related papers  (2025-08-05T17:51:33Z)
• Proving the Coding Interview: A Benchmark for Formally Verified Code Generation [3.5319285228327417]
  We introduce the Formally Verified Automated Programming Progress Standards, or FVAPPS, a benchmark of 4715 samples for writing programs and proving their correctness.<n>We challenge the machine learning and program synthesis communities to solve both each general purpose programming problem and its associated correctness specifications.
  arXiv  Detail & Related papers  (2025-02-08T22:54:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.