EarlyEval: Cheaper Agent Evaluation via Early Outcome Prediction

• URL: http://arxiv.org/abs/2609.02783v1
• Date: Wed, 02 Sep 2026 16:15:18 GMT
• Title: EarlyEval: Cheaper Agent Evaluation via Early Outcome Prediction
• Abstract summary: We introduce early outcome prediction, a complementary axis of efficiency that instead cuts cost within each task.<n>EarlyEval can eliminate 13%-26% of agent steps and up to 44.1% input tokens and 29.4% output tokens at 89%-97% prediction accuracy.
• Score: 14.50182233464472
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Evaluating LLM agents is essential for guiding their development, yet it has grown prohibitively expensive: a single pass of a frontier model over an agentic benchmark can cost hundreds to thousands of dollars, a price paid repeatedly across iterative development cycles. Prior efforts, centered on benchmark distillation, reduce the number of evaluation tasks but leave the cost of executing each retained task untouched. In this work, we introduce early outcome prediction, a complementary axis of efficiency that instead cuts cost within each task. Our key insight is that an agent's final outcome is often evident from its intermediate behavior well before execution completes. We instantiate this idea in EarlyEval, a lightweight framework that trains a pair of LightGBM success and failure classifiers over behavioral, textual, and reference-solution features, and halts an agent run the moment either classifier crosses a calibrated confidence threshold, adding negligible per-step overhead. Across three benchmarks, SWE-bench Verified, TerminalBench, and Toolathlon, EarlyEval can eliminate 13%-26% of agent steps and up to 44.1% input tokens and 29.4% output tokens at 89%-97% prediction accuracy, while perturbing per-agent resolve rates by only one to two percentage points on average.

Related papers

• GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents [5.041247893188969]
  We introduce GAUGE, a reusable offline protocol that measures whether a gate's ranking matches a grounded verifiable reward.<n> satisfaction carries essentially no information about task success.<n>We propose a calibrate-then-trust cadence in which a judge-free completion bit is a zero-cost tripwire for truncation regressions.
  arXiv  Detail & Related papers  (2026-09-10T20:29:28Z)
• VST: Verifiable Structured Transport for Auditable Agent-to-Agent Alpha Discovery [11.790070440436294]
  Agent-to-agent (A2A) alpha discovery is slowed by repeated feedback cycles between mining and evaluation agents.<n>We first restructure this communication as a structured agent-to-agent protocol of emphtyped, causally addressable, unicast records<n>What typing provides is a state that can be schema-checked, replayed deterministically, and prevented by construction from leaking a forecast to an evaluator.
  arXiv  Detail & Related papers  (2026-09-07T05:43:50Z)
• Not Worth Another Token: Marginal Value Estimation for Efficient Deep Research Agents [69.57766200248706]
  We study marginal value estimation for context management in deep research agents.<n>We evaluate lightweight criteria and a learned value model at pre-retrieval, post-retrieval, and pre-synthesis stages.<n> Lightweights reduce token usage by up to 73% with little quality degradation, learned pruning remains competitive on selected trade-offs, and no single method dominates across quality, efficiency, and faithfulness.
  arXiv  Detail & Related papers  (2026-08-09T00:56:37Z)
• To Run or Not to Run: Analyzing the Cost-Effectiveness of Code Execution in LLM-Based Program Repair [18.653708704742353]
  Executions can be time-consuming and expensive, yet their impact on program repair agents remains underexplored.<n>Code execution is used across all agents and models analyzed, with an average of 8.8 test runs per task.<n>Current agents apply execution indiscriminately, paying its cost on instances where it provides little benefit.
  arXiv  Detail & Related papers  (2026-06-25T12:49:59Z)
• ESPO: Early-Stopping Proximal Policy Optimization [78.79610718910628]
  ESPO (Early-Stopping Proximal Policy Optimization) detects trajectory failure on-the-fly and terminates rollouts early.<n>On DeepSeek-R1-Distill-Qwen-7B trained for mathematical reasoning, ESPO surpasses PPO on AIME2024 (46.28% vs. 45.25%), AMC2023 (85.83% vs. 82.94%), and MATH-500 (87.42% vs. 85.43%)
  arXiv  Detail & Related papers  (2026-05-28T12:40:22Z)
• A1: A Fully Transparent Open-Source, Adaptive and Efficient Truncated Vision-Language-Action Model [112.9420001646428]
  Vision-Language-Action (VLA) models have emerged as a powerful paradigm for open-world robot manipulation, but their practical deployment is often constrained by cost.<n>We present A1, a fully open-source and transparent VLA framework designed for low-cost, high- throughput inference.<n>A1 achieves state-of-the-art success rates while significantly reducing inference cost.
  arXiv  Detail & Related papers  (2026-04-07T10:18:40Z)
• Efficient Benchmarking of AI Agents [0.0]
  We study whether small task subsets can preserve agent rankings at substantially lower cost.<n>We find that absolute score prediction degrades under scaffold-driven distribution shift.<n>We propose an optimization-free protocol: evaluate new agents only on tasks with intermediate historical pass rates.
  arXiv  Detail & Related papers  (2026-03-24T22:17:11Z)
• Verified Critical Step Optimization for LLM Agents [67.05296684575445]
  Critical Step Optimization focuses preference learning on verified critical steps.<n>Method starts from failed policy trajectories rather than expert demonstrations.<n>Experiments on GAIA-Text-103 and XBench-DeepSearch show that CSO achieves 37% and 26% relative improvement over the SFT baseline.
  arXiv  Detail & Related papers  (2026-02-03T11:41:02Z)
• EET: Experience-Driven Early Termination for Cost-Efficient Software Engineering Agents [22.98266662213199]
  EET is an experience-driven early termination approach for software engineering agents.<n>It reduces the cost of SE agents while preserving task performance.<n>EET consistently reduces total cost by 19%-55%, with negligible loss in resolution rate.
  arXiv  Detail & Related papers  (2026-01-09T13:01:49Z)
• ZIP-RC: Optimizing Test-Time Compute via Zero-Overhead Joint Reward-Cost Prediction [57.799425838564]
  We present ZIP-RC, an adaptive inference method that equips models with zero-overhead inference-time predictions of reward and cost.<n> ZIP-RC improves accuracy by up to 12% over majority voting at equal or lower average cost.
  arXiv  Detail & Related papers  (2025-12-01T09:44:31Z)
• Runaway is Ashamed, But Helpful: On the Early-Exit Behavior of Large Language Model-based Agents in Embodied Environments [54.67512489842682]
  Large language models (LLMs) have demonstrated strong planning and decision-making capabilities in complex embodied environments.<n>We take a first step toward exploring the early-exit behavior for LLM-based agents.
  arXiv  Detail & Related papers  (2025-05-23T08:23:36Z)
• Forecasting Frontier Language Model Agent Capabilities [0.7499722271664147]
  We evaluate six forecasting methods that predict downstream capabilities of Language Models (LMs)<n>We use "one-step" approaches that predict benchmark scores from input metrics like compute or model release date directly or "two-step" approaches that first predict an intermediate metric like the principal component of cross-benchmark performance (PC-1) and human-evaluated competitive Elo ratings.<n>Our forecast predicts that by the beginning of 2026, non-specialized LM agents with low capability elicitation will reach a success rate of 54% on SWE-Bench Verified, while state-of-the-art LM agents will reach an 87% success rate.
  arXiv  Detail & Related papers  (2025-02-21T02:34:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.