Agents Are Systems, Not Models: Rethinking Agentic Evaluation
Abstract Overview
This paper argues that agentic evaluation should treat an agent as a configurable system rather than as a fixed model. Using a new benchmark of four scientific tasks from astrophysics and genomics, the authors systematically vary five configuration axes: task information, reasoning, self-verification, time budget, and backbone model. The tasks require a coding agent to identify and operate published specialist models under controlled conditions while measuring task scores, completion, consistency, cost, runtime, and calibration. Across the experiments, the authors find substantial run-to-run variability, strong interactions between configuration choices, and a dominant role for the information provided to the agent. The authors release the benchmark alongside more than 18,000 agent trajectories for reproducibility and behavioral analysis.
Novelty
The paper frames agent evaluation as a systems problem by systematically decomposing agent performance across five configurable axes rather than attributing outcomes primarily to the backbone model. It introduces a controlled AI-for-Science benchmark focused on operating specialist models and pairs it with a large-scale trajectory release and an automated, trajectory-derived behavioral taxonomy.
Results
Among genuine attempts, approximately 54% of outcome variance stems from run-to-run noise within identical configurations rather than configuration changes. Provided task information exhibits the largest effect on performance, outranking model capability and budget while simultaneously reducing cost, shortening runtime, and improving calibration. Prompted self-verification shows minimal impact on behavior and performance, whereas equipping the agent with a system-level verification tool significantly improves task scores and calibration at the expense of higher cost and runtime.
Key Points
- The benchmark evaluates agents across four scientific tasks requiring the discovery and execution of specialist models under controlled, repeatable conditions.
- Task information is the most impactful configuration axis, improving score while reducing runtime and cost and improving calibration without trade-offs.
- Configuration axes exhibit strong interactions: increased time budget primarily benefits agents when paired with sufficient task information or a sufficiently capable model, and verification behaviors are far more effectively driven by system tools than user prompts.