When Is an Agent Evaluation Over? Outcome Finality and Cross-Unit Separation

• URL: http://arxiv.org/abs/2608.14940v1
• Date: Fri, 14 Aug 2026 23:39:47 GMT
• Title: When Is an Agent Evaluation Over? Outcome Finality and Cross-Unit Separation
• Abstract summary: Current agent evaluations score models on the state visible at the end of a stopped run which they count as one trial.<n> interpreting the score as a final result would require two conditions that the endpoint does not necessarily establish: outcome finality and cross-unit separation.<n>We argue that a final label is justified only when anything that could still change the claimed outcome is resolved, bounded, or retained as uncertainty.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Current agent evaluations score models on the state visible at the end of a stopped run which they count as one trial. However, interpreting the score as a final result would require two conditions that the endpoint does not itself necessarily establish: outcome finality and cross-unit separation. These conditions are independent, since reconciling a delayed outcome can settle the label while runs still share state and isolating runs can prevent carryover while the scored outcome remains unfinished. We develop a completion argument that specifies the evidence needed for each decision and argue that a final label is justified only when anything that could still change the claimed outcome is resolved, bounded, or retained as uncertainty. First, in a controlled replay to demonstrate the mechanism where an agent's actions were held fixed, we find that the endpoint and terminal labels differ for every delayed operation, while a delayed write changes the next run's score when service state persists between runs but not after isolation or verified reset. Second, in a review of ten public protocols, we find that all protocols identify when a run stops and what is scored, while unfinished operations and the evidence for treating runs as separate trials are documented less consistently. Finally, we propose an open-effects record that lists operations or resources that may remain relevant after the endpoint, their current status, and whether they could change the scored outcome or affect another run.

Related papers

• Recoverability as a System Primitive for Long-Horizon AI Agents [5.008310199603767]
  We introduce recoverability as a system primitive that makes reuse an explicit decision.<n>Four deterministic and 20 paired file challenges demonstrate that accurate restoration and successful completion can conceal disallowed starting points.
  arXiv  Detail & Related papers  (2026-09-12T02:54:33Z)
• From Experiments to Decisions: Reusing Evidence in Autonomous Coding Research [0.0]
  We reconstruct how evidence is reused in a 400-task NeuroGolf campaign.<n>A numerical counterexample exposes an overbroad exclusion.<n>Other episodes distinguish what failed, incomplete and revised programs justify doing next.
  arXiv  Detail & Related papers  (2026-09-10T03:48:31Z)
• VeriPhy: Agentic Physical Reasoning for World Model Evaluation and Refinement [57.86962208273888]
  We present an auditable physical-verification system in which a text-only planner compiles the prompt into typed physical obligations.<n>Each action returns a provenance-carrying evidence record whose payload, when usable, is either a typed measurement or an explicitly tagged learned state.<n>We anchor evaluation in a 1,500-clip corpus of human-annotated flaw records that localize real generation failures in prompt reference, space, and time.
  arXiv  Detail & Related papers  (2026-09-02T20:36:45Z)
• LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering [47.39676953516343]
  Loop Engineering is emerging as a practice for organizing development work around coding agents.<n>We introduce LoopArena, a benchmark for evaluating how well one model can guide a separate coding agent through a long-running task.
  arXiv  Detail & Related papers  (2026-08-28T12:44:54Z)
• From Trajectories to Evidence: Auditable Experimental Records for Industrial Research Agents [47.678562263387214]
  Research agents increasingly conduct multi-round machine-learning experiments in industrial recommendation settings.<n>Yet a completed trajectory is not automatically evidence: generated artifacts may be unsupported or incomplete, executed rounds may be invalid or confounded, and later modifications may obscure earlier findings.<n>We introduce an evidence-grounded framework that couples bounded verification of consequential artifacts with post-execution claim qualification.
  arXiv  Detail & Related papers  (2026-08-05T13:37:38Z)
• ParEvalLayer: When Partial LLM-Agent Evaluations Support a Decision [0.0]
  LLM-agent evaluations often produce task outcomes long before the full benchmark run is complete.<n>A partial score is tempting to report, but it does not show whether the observed tasks support the same conclusion as the completed evaluation.<n>We introduce ParEvalLayer, a decision layer that reads paired outcomes for two agent systems and a comparison policy chosen in advance.
  arXiv  Detail & Related papers  (2026-08-03T16:22:51Z)
• PhyAgentOS: A Self-Evolving Operating System for Embodied Agents with Decoupled Cognitive Planning and Physical Execution [49.776611937968]
  We present PhyAgentOS, a foundation delivering scheduling, verification, memory, benchmarking, and safety as system-level services.<n>Its Session-Centered treats a session, not an action, as the minimum unit of scheduling, compatibility, supervised execution, evidence collection, and acceptance.<n>SessionVerifier distinguishes execution termination from semantic task completion via evidence-grounded verdicts of success, failure, or replan.<n> Benchmarking reuses the deployment session and verification path, so results trace to real execution.
  arXiv  Detail & Related papers  (2026-07-18T04:46:53Z)
• What Resolve Rate Hides: Trajectory Structure Diagnostics for Coding Agents [14.988334585821939]
  Coding agents are ranked almost entirely by resolve rate.<n>A single pass/fail label says nothing about why a run failed or why an accepted run spent extra steps, time, or tokens.<n>We present TraceProbe, a trajectory-diagnostic framework that recovers what resolve rate hides.
  arXiv  Detail & Related papers  (2026-07-07T12:09:46Z)
• Verify-Gated Completion as Admission Control in a Governed Multi-Agent Runtime: A Bounded Architecture Case Study [0.6875312133832079]
  We study verify-gated completion as an admission-control pattern for governed multi-agent runtimes.<n>We show that under observed conditions, a read-only verify gate plus packetized admission records made completion decisions inspectable and fail-closed.
  arXiv  Detail & Related papers  (2026-05-18T07:52:13Z)
• ClawForge: Generating Executable Interactive Benchmarks for Command-Line Agents [59.626170560327274]
  We present textbfClawForge, a generator-backed benchmark framework for executable command-line categories under state conflict.<n>We instantiate this framework as the ClawForge-Bench (17 scenarios, 6 ability categories)
  arXiv  Detail & Related papers  (2026-05-13T21:34:08Z)
• Can Agent Benchmarks Support Their Scores? Evidence-Supported Bounds for Interactive-Agent Evaluation [8.049844623361725]
  Interactive agent benchmarks map an agent run to a binary outcome through outcome checks.<n>These checks rely on surface level signals or fail to capture the agent's actual action path.<n>We introduce an outcome evidence reporting layer for existing benchmarks.
  arXiv  Detail & Related papers  (2026-05-11T12:20:15Z)
• Bidirectional Decoding: Improving Action Chunking via Guided Test-Time Sampling [51.38330727868982]
  We show how action chunking impacts the divergence between a learner and a demonstrator.<n>We propose Bidirectional Decoding (BID), a test-time inference algorithm that bridges action chunking with closed-loop adaptation.<n>Our method boosts the performance of two state-of-the-art generative policies across seven simulation benchmarks and two real-world tasks.
  arXiv  Detail & Related papers  (2024-08-30T15:39:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.