ParEvalLayer: When Partial LLM-Agent Evaluations Support a Decision

• URL: http://arxiv.org/abs/2608.02444v1
• Date: Mon, 03 Aug 2026 16:22:51 GMT
• Title: ParEvalLayer: When Partial LLM-Agent Evaluations Support a Decision
• Abstract summary: LLM-agent evaluations often produce task outcomes long before the full benchmark run is complete.<n>A partial score is tempting to report, but it does not show whether the observed tasks support the same conclusion as the completed evaluation.<n>We introduce ParEvalLayer, a decision layer that reads paired outcomes for two agent systems and a comparison policy chosen in advance.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LLM-agent evaluations often produce task outcomes long before the full benchmark run is complete. A partial score is tempting to report, but it does not show whether the observed tasks support the same conclusion as the completed evaluation. Early tasks can omit important parts of a benchmark, running cheaper tasks first can distort the observed sample, and a rule that decides only easy pairs can appear accurate while leaving many comparisons unresolved. We introduce ParEvalLayer, a decision layer that reads paired outcomes for two agent systems and a comparison policy chosen in advance. For each partial run, it records whether the tested agent system is better by the required amount, is not better by that amount, needs more evidence, or should abstain. We evaluate ParEvalLayer by replaying completed public benchmark data as if each evaluation had stopped earlier. At each point, ParEvalLayer applies the policy using only the outcomes observed so far; if it reaches one of the two comparison judgments, we check whether that judgment matches the completed data for the same system pair. With the main comparison rule, three of the public benchmarks reach the same decision as the completed evaluation after observing only 15% to 25% of task outcomes. Other benchmarks require more task outcomes. This variation shows why a partial score alone is not enough: reports should also state the decision rule and how many comparisons remain without a decision.

Related papers

• Evaluating Multiple LLM Generations with Validated Task Coverage [12.15293959091918]
  We introduce VTC-Bench, a five-domain benchmark for evaluation of finite candidate sets.<n>VTC measures how many distinct useful results are obtained within $k$ attempts.<n>Across multiple models and inference settings, the benchmark leads to different conclusions from conventional evaluation.
  arXiv  Detail & Related papers  (2026-08-25T08:32:37Z)
• When Is an Agent Evaluation Over? Outcome Finality and Cross-Unit Separation [0.0]
  Current agent evaluations score models on the state visible at the end of a stopped run which they count as one trial.<n> interpreting the score as a final result would require two conditions that the endpoint does not necessarily establish: outcome finality and cross-unit separation.<n>We argue that a final label is justified only when anything that could still change the claimed outcome is resolved, bounded, or retained as uncertainty.
  arXiv  Detail & Related papers  (2026-08-14T23:39:47Z)
• How Many Tasks Are Enough for Agent Benchmark Decisions? A Replay Analysis of Public LLM Agent Benchmarks [0.0]
  A task fraction alone does not show whether a partial run supports the same pairwise conclusion as the completed benchmark.<n>We study this question by replaying completed public task-level records from SWE-bench, AppWorld, and tau-bench.
  arXiv  Detail & Related papers  (2026-07-14T04:36:41Z)
• PACE: A Proxy for Agentic Capability Evaluation [60.3743414796937]
  We introduce PACE, a framework that constructs proxy benchmarks by selecting instances from existing non-agentic evaluations.<n>Experiments across 14 models, 4 agentic benchmarks, and 19 non-agentic benchmarks show that PACE-Bench predicts agentic scores with leave-one-out cross-validation (LOOCV) mean absolute error (MAE) under 4%, Spearman correlation above 0.80, and pairwise model-ranking accuracy around 85%, all at much less than 1% of the full agentic evaluation cost.
  arXiv  Detail & Related papers  (2026-07-02T10:59:03Z)
• Can Agent Benchmarks Support Their Scores? Evidence-Supported Bounds for Interactive-Agent Evaluation [8.049844623361725]
  Interactive agent benchmarks map an agent run to a binary outcome through outcome checks.<n>These checks rely on surface level signals or fail to capture the agent's actual action path.<n>We introduce an outcome evidence reporting layer for existing benchmarks.
  arXiv  Detail & Related papers  (2026-05-11T12:20:15Z)
• VLM Judges Can Rank but Cannot Score: Task-Dependent Uncertainty in Multimodal Evaluation [4.694504497452662]
  Vision-language models (VLMs) are increasingly used as automated judges for multimodal systems, yet their scores provide no indication of reliability.<n>We study this problem through conformal prediction, a distribution-free framework that converts a judge's point score into a calibrated prediction interval.<n>We present the first systematic analysis of conformal prediction for VLM-as-a-Judge across 3 judges and 14 visual task categories.
  arXiv  Detail & Related papers  (2026-04-28T05:30:18Z)
• Active Evaluation of General Agents: Problem Definition and Comparison of Baseline Algorithms [18.53965204068826]
  We propose a formal definition and a conceptual framework for active evaluation of agents across multiple tasks.<n>On every iteration, the ranking algorithm chooses the task and agents to sample scores from.<n>We find that the classical Elo rating system is a consistently reliable choice for efficient reduction of ranking error in practice.
  arXiv  Detail & Related papers  (2026-01-12T15:32:11Z)
• TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them [58.04324690859212]
  Large Language Models (LLMs) as automated evaluators (LLM-as-a-judge) has revealed critical inconsistencies in current evaluation frameworks.<n>We identify two fundamental types of inconsistencies: Score-Comparison Inconsistency and Pairwise Transitivity Inconsistency.<n>We propose TrustJudge, a probabilistic framework that addresses these limitations through two key innovations.
  arXiv  Detail & Related papers  (2025-09-25T13:04:29Z)
• SureMap: Simultaneous Mean Estimation for Single-Task and Multi-Task Disaggregated Evaluation [75.56845750400116]
  Disaggregated evaluation -- estimation of performance of a machine learning model on different subpopulations -- is a core task when assessing performance and group-fairness of AI systems. We develop SureMap that has high estimation accuracy for both multi-task and single-task disaggregated evaluations of blackbox models. Our method combines maximum a posteriori (MAP) estimation using a well-chosen prior together with cross-validation-free tuning via Stein's unbiased risk estimate (SURE)
  arXiv  Detail & Related papers  (2024-11-14T17:53:35Z)
• Bidirectional Decoding: Improving Action Chunking via Guided Test-Time Sampling [51.38330727868982]
  We show how action chunking impacts the divergence between a learner and a demonstrator.<n>We propose Bidirectional Decoding (BID), a test-time inference algorithm that bridges action chunking with closed-loop adaptation.<n>Our method boosts the performance of two state-of-the-art generative policies across seven simulation benchmarks and two real-world tasks.
  arXiv  Detail & Related papers  (2024-08-30T15:39:34Z)
• LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond [135.8013388183257]
  We propose a new protocol for inconsistency detection benchmark creation and implement it in a 10-domain benchmark called SummEdits. Most LLMs struggle on SummEdits, with performance close to random chance. The best-performing model, GPT-4, is still 8% below estimated human performance.
  arXiv  Detail & Related papers  (2023-05-23T21:50:06Z)
• Frustratingly Simple Few-Shot Object Detection [98.42824677627581]
  We find that fine-tuning only the last layer of existing detectors on rare classes is crucial to the few-shot object detection task. Such a simple approach outperforms the meta-learning methods by roughly 220 points on current benchmarks.
  arXiv  Detail & Related papers  (2020-03-16T00:29:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.