What Process Evaluation of Coding Agents Actually Measures: Action, Task, and Step Are Three Different Levels

• URL: http://arxiv.org/abs/2608.22960v1
• Date: Mon, 24 Aug 2026 08:27:33 GMT
• Title: What Process Evaluation of Coding Agents Actually Measures: Action, Task, and Step Are Three Different Levels
• Abstract summary: We introduce a measurement framework for process evaluation in coding agents.<n>Our framework combines prefix-conditioned identification, replay/intervention-based estimation, and controlled judge-information manipulation.
• Score: 5.46203679640347
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Coding agents are increasingly evaluated not only by whether they solve a task, but also by how they execute it. However, existing process-level evaluations often treat action prediction, task uncertainty, and step attribution as if they were the same problem, which makes it unclear what such evaluations actually measure. In this paper, we introduce a measurement framework for process evaluation in coding agents and instantiate step-level causal attribution with SCAE, a replay-based estimator derived from a structural causal model of agent execution. Our framework combines prefix-conditioned identification, replay/intervention-based estimation, and controlled judge-information manipulation to study process evaluation at the action, task, and step levels. Experiments on 499 file-localization episodes from 12 repositories show that next actions are driven primarily by execution provenance rather than code-graph transitions, execution uncertainty is structured at the task rather than step level, and full-trace judges exhibit systematic collider bias, suggesting that current process evaluation often measures semantic relevance rather than certified causal contribution.

Related papers

• Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents? [61.49434544687523]
  We introduce REFLECT, a meta-evaluation benchmark targeting fine-grained failure detection in agentic environments.<n> REFLECT defines a detailed taxonomy of process- and outcome-level failure modes, instantiated by performing controlled and localized interventions.<n>Our experiments show that even the best-performing models achieve overall accuracies below 55% across reasoning, tool-use, and report-quality failures.
  arXiv  Detail & Related papers  (2026-05-18T23:55:08Z)
• ProcCtrlBench: Evaluating Process-Level Defects and Control Preservation in LLM Coding Agents [4.908588441456849]
  ProcCtrlBench is a benchmark for execution-process evaluation in LLM coding agents.<n>It organizes recurrent execution defects into a reusable ontology covering 11 defect types in 4 categories.<n>It evaluates agent trajectories through standardized process evidence rather than final outcomes alone.
  arXiv  Detail & Related papers  (2026-05-18T08:34:48Z)
• Interactive Evaluation Requires a Design Science [90.62695599188204]
  Large language models (LLMs) are increasingly deployed as systems that act over time through tools, environments, users, and other agents.<n>The field has begun to build interactive benchmarks, but the resulting landscape is fragmented.<n>This position paper argues that interactive evaluation should be treated as a principled evaluation paradigm.
  arXiv  Detail & Related papers  (2026-05-18T04:03:18Z)
• A Benchmark for Evaluating Repository-Level Code Agents with Intermediate Reasoning on Feature Addition Task [11.218318079376365]
  RACE-bench is a reasoning-augmented benchmark for evaluating code agents on feature addition tasks.<n> RACE-bench contains 528 real-world feature addition instances from 12 open-source repositories.<n>We evaluate three representative repository-level code agents on RACE-bench.
  arXiv  Detail & Related papers  (2026-03-27T11:58:47Z)
• Beyond Task Completion: Revealing Corrupt Success in LLM Agents through Procedure-Aware Evaluation [2.102846336724103]
  Procedure-Aware Evaluation (PAE) is a framework that formalizes agent procedures as structured observations.<n>We evaluate state-of-the-art Large Language Model (LLM)-based agents on tau-bench.
  arXiv  Detail & Related papers  (2026-03-03T15:47:41Z)
• Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation [76.5533899503582]
  Large language models (LLMs) are increasingly used as judges to evaluate agent performance.<n>We show this paradigm implicitly assumes that the agent's chain-of-thought (CoT) reasoning faithfully reflects both its internal reasoning and the underlying environment state.<n>We demonstrate that manipulated reasoning alone can inflate false positive rates of state-of-the-art VLM judges by up to 90% across 800 trajectories spanning diverse web tasks.
  arXiv  Detail & Related papers  (2026-01-21T06:07:43Z)
• Auto-Eval Judge: Towards a General Agentic Framework for Task Completion Evaluation [4.08768677009363]
  We propose a generalizable, modular framework for evaluating agent task completion independent of the task domain.<n>We validate our framework by evaluating the Magentic-One Actor Agent on two benchmarks, GAIA and BigCodeBench.<n>Our Judge Agent predicts task success with closer agreement to human evaluations, achieving 4.76% and 10.52% higher alignment accuracy, respectively.
  arXiv  Detail & Related papers  (2025-08-07T15:39:48Z)
• A Procedural Framework for Assessing the Desirability of Process Deviations [0.0]
  This paper presents a procedural framework to guide process analysts in systematically assessing deviation desirability.<n>It provides a step-by-step approach for identifying which input factors to consider in what order to categorize deviations into mutually exclusive desirability categories.
  arXiv  Detail & Related papers  (2025-06-13T07:24:57Z)
• Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinement [50.481380478458945]
  Iterative step-level Process Refinement (IPR) framework provides detailed step-by-step guidance to enhance agent training. Our experiments on three complex agent tasks demonstrate that our framework outperforms a variety of strong baselines.
  arXiv  Detail & Related papers  (2024-06-17T03:29:13Z)
• IndustReal: A Dataset for Procedure Step Recognition Handling Execution Errors in Egocentric Videos in an Industrial-Like Setting [7.561148568365396]
  We present the novel task of procedure step recognition (PSR) PSR focuses on recognizing the correct completion and order of procedural steps. We also present the multi-modal IndustReal dataset.
  arXiv  Detail & Related papers  (2023-10-26T11:44:29Z)
• ReAct: Temporal Action Detection with Relational Queries [84.76646044604055]
  This work aims at advancing temporal action detection (TAD) using an encoder-decoder framework with action queries. We first propose a relational attention mechanism in the decoder, which guides the attention among queries based on their relations. Lastly, we propose to predict the localization quality of each action query at inference in order to distinguish high-quality queries.
  arXiv  Detail & Related papers  (2022-07-14T17:46:37Z)
• SVIP: Sequence VerIfication for Procedures in Videos [68.07865790764237]
  We propose a novel sequence verification task that aims to distinguish positive video pairs performing the same action sequence from negative ones with step-level transformations. Such a challenging task resides in an open-set setting without prior action detection or segmentation. We collect a scripted video dataset enumerating all kinds of step-level transformations in chemical experiments.
  arXiv  Detail & Related papers  (2021-12-13T07:03:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.