ResearchEnvBench: Benchmarking Agents on Environment Synthesis for Research Code Execution

• URL: http://arxiv.org/abs/2603.06739v2
• Date: Wed, 11 Mar 2026 07:59:46 GMT
• Title: ResearchEnvBench: Benchmarking Agents on Environment Synthesis for Research Code Execution
• Abstract summary: ResearchEnvBench is a benchmark for environment synthesis in research code execution.<n>It provides a realistic testbed for advancing autonomous agents toward reproducible scientific research.
• Score: 44.652433226889976
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Autonomous agents are increasingly expected to support scientific research, and recent benchmarks report progress in code repair and autonomous experimentation. However, these evaluations typically assume a pre-configured execution environment, which requires resolving complex software dependencies, aligning hardware and framework versions, and configuring distributed execution, yet this capability remains largely unbenchmarked. We introduce ResearchEnvBench, a benchmark for environment synthesis in research code execution. Given a research repository, documentation, and a target execution setting, agents must construct an environment that successfully executes at runtime. Evaluations on diverse research repositories reveal a substantial gap in current SOTA agents, with failures dominated by incomplete dependency resolution and brittle version coupling. ResearchEnvBench provides a realistic testbed for advancing autonomous agents toward reproducible scientific research.

Related papers

• DeployBench: Benchmarking LLM Agents for Research Artifact Deployment [21.93024846646044]
  LLM agents have made rapid progress on software engineering and ML research tasks, but these advances often assume access to a working runnable environment.<n>Existing environment setup benchmarks do not cover the full scope of research artifact deployment.<n>We introduce DeployBench, a benchmark of 51 research-artifact deployment tasks spanning AI/ML, computer systems, and scientific computing.
  arXiv  Detail & Related papers  (2026-06-03T04:59:34Z)
• AI for Auto-Research: Roadmap & User Guide [107.0834449839233]
  We present an end-to-end analysis of AI across the complete research lifecycle.<n>We identify a sharp, stage-dependent boundary between reliable assistance and unreliable autonomy.<n>We show that greater automation can obscure rather than eliminate failure modes.
  arXiv  Detail & Related papers  (2026-05-18T17:08:26Z)
• AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agents [49.67355440164857]
  We introduce AIRS-Bench, a suite of 20 tasks sourced from state-of-the-art machine learning papers.<n>Airs-Bench tasks assess agentic capabilities over the full research lifecycle.<n>We open-source the AIRS-Bench task definitions and evaluation code to catalyze further development in autonomous scientific research.
  arXiv  Detail & Related papers  (2026-02-06T16:45:02Z)
• RExBench: Can coding agents autonomously implement AI research extensions? [14.147417159347448]
  Agents based on Large Language Models (LLMs) have shown promise for performing sophisticated software engineering tasks autonomously.<n>We argue that research extension and its implementation is a critical capability for such systems.<n>We introduce RExBench to support the evaluation of this capability.
  arXiv  Detail & Related papers  (2025-06-27T19:41:41Z)
• From Reproduction to Replication: Evaluating Research Agents with Progressive Code Masking [48.90371827091671]
  AutoExperiment is a benchmark that evaluates AI agents' ability to implement and run machine learning experiments.<n>We evaluate state-of-the-art agents and find that performance degrades rapidly as $n$ increases.<n>Our findings highlight critical challenges in long-horizon code generation, context retrieval, and autonomous experiment execution.
  arXiv  Detail & Related papers  (2025-06-24T15:39:20Z)
• Commit0: Library Generation from Scratch [77.38414688148006]
  Commit0 is a benchmark that challenges AI agents to write libraries from scratch.<n>Agents are provided with a specification document outlining the library's API as well as a suite of interactive unit tests.<n> Commit0 also offers an interactive environment where models receive static analysis and execution feedback on the code they generate.
  arXiv  Detail & Related papers  (2024-12-02T18:11:30Z)
• Codev-Bench: How Do LLMs Understand Developer-Centric Code Completion? [60.84912551069379]
  We present the Code-Development Benchmark (Codev-Bench), a fine-grained, real-world, repository-level, and developer-centric evaluation framework. Codev-Agent is an agent-based system that automates repository crawling, constructs execution environments, extracts dynamic calling chains from existing unit tests, and generates new test samples to avoid data leakage.
  arXiv  Detail & Related papers  (2024-10-02T09:11:10Z)
• SIERRA: A Modular Framework for Research Automation and Reproducibility [6.1678491628787455]
  We present SIERRA, a novel framework for accelerating research development and improving results. SIERRA accelerates research by automating the process of generating executable experiments from queries over independent variables. It employs a modular architecture enabling easy customization and extension for the needs of individual researchers.
  arXiv  Detail & Related papers  (2022-08-16T15:36:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.