A case study of evaluating AI agents on a neuroscience data-to-discovery pipeline

• URL: http://arxiv.org/abs/2606.07718v1
• Date: Fri, 05 Jun 2026 15:38:18 GMT
• Title: A case study of evaluating AI agents on a neuroscience data-to-discovery pipeline
• Abstract summary: We present an empirical study of general-purpose coding agents on a fly optogenetics data-to-discovery pipeline.<n>We assess agents on tasks substantially larger than existing benchmarks, datasets orders of magnitude bigger, and evaluation criteria grounded in domain expert standards.<n>We show that agents can solve several individual pipeline stages, suggesting stage-level automation is tractable.
• Score: 4.751533658796929
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agentic AI tools offer a promising path to automating software development bottlenecks in scientific research pipelines, particularly for stages that take domain experts days to months to build, where scientists care about correctness and robustness, not implementation details. We present an empirical study of general-purpose coding agents on a fly optogenetics data-to-discovery pipeline. We assess agents on tasks substantially larger than existing benchmarks, datasets orders of magnitude bigger, and evaluation criteria grounded in domain expert standards. We show that agents can solve several individual pipeline stages, suggesting stage-level automation is tractable. By analyzing agents' code iterations, we show that they struggle most when there is not a pre-defined criterion to iterate on, and they must instead use their scientific judgment to assess their current solution, a key open challenge. Mirroring scientific practice, they sometimes attempt visual inspection of intermediate outputs for self-evaluation, but largely fail to interpret what they see or act on it appropriately. Solving the end-to-end pipeline correctly requires stringing together successes across all pipeline stages, and this is beyond agents' current abilities. We identify challenges largely absent from existing benchmarks, including computational resource management and generalization to large held-out data collections. Finally, we distill principles for constructing scientific tasks and rigorous evaluation criteria for open-ended problems.

Related papers

• Benchmarking AI Agents for Addressing Scientific Challenges Across Scales [118.2204632627895]
  SciAgentArena is a systematic benchmark for evaluating AI agents in real-world scientific research scenarios.<n>It comprises approximately 200 tasks with stepwise verification and an interactive, agent-agnostic environment for assessing diverse AI agents.<n>We find that current agents can contribute effectively to well-specified data-analysis, particularly when the task structure and evaluation criteria are clear.
  arXiv  Detail & Related papers  (2026-06-10T22:55:30Z)
• TrajAudit: Automated Failure Diagnosis for Agentic Coding Systems [18.833806809109536]
  Agentic systems have been widely studied to automate software engineering jobs such as bug fixing.<n>Existing automated failure diagnosis approaches leverage task execution trajectories, yet their effectiveness degrades as trajectory length and complexity increase.<n>For repository-level coding tasks specifically, trajectories are laden with noise, such as redundant program structure and verbose code context.<n>We propose TrajAudit, the first failure diagnosis framework for repository-level coding trajectories.
  arXiv  Detail & Related papers  (2026-05-26T05:24:37Z)
• Collider-Bench: Benchmarking AI Agents with Particle Physics Analysis Reproduction [0.02446672595462589]
  We introduce Collider-Bench, a benchmark for evaluating whether language-model agents can reproduce experimental analyses from the Large Hadron Collider (LHC) using only public papers and open scientific software.<n>Agents must therefore rely on physical reasoning, domain knowledge, and trial-and-error to fill these gaps.<n>Our results show that on average no agent reliably beats the physicist-in-the-loop solution.
  arXiv  Detail & Related papers  (2026-05-13T18:00:00Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• Can Coding Agents Reproduce Findings in Computational Materials Science? [49.254975563645786]
  We present AutoMat, a benchmark for evaluating large language models' ability to reproduce scientific claims.<n>By working closely with subject matter experts, we curate a set of claims from real materials science papers to test whether coding agents can recover and execute the end-to-end workflow.<n>Results show that current LLM-based agents obtain low overall success rates on AutoMat, with the best-performing setting achieving a success rate of only 54.1%.
  arXiv  Detail & Related papers  (2026-05-01T17:42:12Z)
• Beyond Isolated Tasks: A Framework for Evaluating Coding Agents on Sequential Software Evolution [5.10403054516716]
  Existing datasets evaluate performance on isolated, single pull request (PR) tasks in a stateless manner.<n>We introduce an automated coding task generation framework, which helps generate our dataset SWE-STEPS.
  arXiv  Detail & Related papers  (2026-04-03T13:44:40Z)
• AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents [50.481033105867205]
  We introduce AgentProcessBench, the first benchmark dedicated to evaluating step-level effectiveness in realistic, tool-augmented trajectories.<n>The benchmark comprises 1,000 diverse trajectories and 8,509 human-labeled step annotations with 89.1% inter-annotator agreement.<n>It features a ternary labeling scheme to capture exploration and an error propagation rule to reduce labeling ambiguity.
  arXiv  Detail & Related papers  (2026-03-15T16:13:58Z)
• Simple Agents Outperform Experts in Biomedical Imaging Workflow Optimization [69.36509281190662]
  Adapting production-level computer vision tools to bespoke scientific datasets is a critical "last mile" bottleneck.<n>We consider using AI agents to automate this manual coding, and focus on the open question of optimal agent design.<n>We demonstrate that a simple agent framework consistently generates adaptation code that outperforms human-expert solutions.
  arXiv  Detail & Related papers  (2025-12-02T18:42:26Z)
• CoDA: Agentic Systems for Collaborative Data Visualization [57.270599188947294]
  Deep research has revolutionized data analysis, yet data scientists still devote substantial time to manually crafting visualizations.<n>Existing approaches, including simple single- or multi-agent systems, often oversimplify the task.<n>We introduce CoDA, a multi-agent system that employs specialized LLM agents for metadata analysis, task planning, code generation, and self-reflection.
  arXiv  Detail & Related papers  (2025-10-03T17:30:16Z)
• UltraHorizon: Benchmarking Agent Capabilities in Ultra Long-Horizon Scenarios [63.67884284105684]
  We introduce textbfUltraHorizon, a novel benchmark that measures the foundational capabilities essential for complex real-world challenges.<n>Agents are designed in long-horizon discovery tasks where they must iteratively uncover hidden rules.<n>Our experiments reveal that LLM-agents consistently underperform in these settings, whereas human participants achieve higher scores.
  arXiv  Detail & Related papers  (2025-09-26T02:04:00Z)
• TRAIL: Trace Reasoning and Agentic Issue Localization [5.025960714013197]
  This work articulates the need for robust and dynamic evaluation methods for agentic workflow traces.<n>We present a set of 148 large human-annotated traces (TRAIL) constructed using this taxonomy and grounded in established agentic benchmarks.<n>To ensure ecological validity, we curate traces from both single and multi-agent systems.
  arXiv  Detail & Related papers  (2025-05-13T14:55:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.