To Run or Not to Run: Analyzing the Cost-Effectiveness of Code Execution in LLM-Based Program Repair

• URL: http://arxiv.org/abs/2606.26978v1
• Date: Thu, 25 Jun 2026 12:49:59 GMT
• Title: To Run or Not to Run: Analyzing the Cost-Effectiveness of Code Execution in LLM-Based Program Repair
• Abstract summary: Executions can be time-consuming and expensive, yet their impact on program repair agents remains underexplored.<n>Code execution is used across all agents and models analyzed, with an average of 8.8 test runs per task.<n>Current agents apply execution indiscriminately, paying its cost on instances where it provides little benefit.
• Score: 18.653708704742353
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LLM-based agents for program repair are increasingly built on a "generate-run-revise" paradigm, iteratively executing tests to evaluate and refine patches. This execution-based approach has become standard practice in state-of-the-art systems. However, executions can be time-consuming and expensive, yet their impact on these agents remains underexplored. In this paper, we conduct a two-stage empirical study over execution behavior in LLM-based program repair. To characterize execution behavior at scale, we first analyze 7,745 agent traces from SWE-bench leaderboard submissions. Second, we evaluate 3,000 end-to-end repair attempts across 200 SWE-bench instances and three agents (Claude Code, Codex, and the open-source OpenCode) under four execution paradigms, which allows for a fine-grained comparison of performance and cost. Our analysis reveals three key observations: (1) Code execution is used across all agents and models analyzed, with an average of 8.8 test runs per task. Execution behavior varies substantially across agents and models, with frequency ranging from 2 to 19 per task, and late-stage executions consistently achieve higher success rates than early-stage ones. (2) Execution restrictions have little effect on repair success: on commercial agents with SOTA models the resolve-rate gap between Prohibited and Unrestricted is only 1.25 percentage points and not statistically significant, while Prohibited saves substantial token and wall-clock cost. (3) Execution benefit is concentrated rather than uniform. These patterns suggest that current agents apply execution indiscriminately, paying its cost on instances where it provides little benefit. Execution, therefore, should be treated as a resource with an explicit cost-benefit tradeoff, not a default capability.

Related papers

• Learning Robust Execution in Robotic Manipulation with Agentic Reinforcement Learning [17.169000876264796]
  We propose an agentic reinforcement learning framework that learns to restore effective execution through high-level decision-making.<n>Under execution degradation, it triggers appropriate recovery mechanisms to restore the robot to previously visited nominal states.<n>We evaluate the proposed method on the LIBERO benchmark, achieving up to a 13.7% improvement in success rate.
  arXiv  Detail & Related papers  (2026-07-15T13:25:52Z)
• Do AI Agents Know When a Task Is Simple? Toward Complexity-Aware Reasoning and Execution [1.733151600403503]
  Large language model (LLM) agents increasingly automate multi-step engineering and informatics.<n>They often follow a maximum-context-first strategy--re-reading files and dependencies they have already seen--turning a one-line edit into a small code-base audit.<n>We argue the missing capability is task-aware execution-scope estimation.
  arXiv  Detail & Related papers  (2026-07-14T17:59:31Z)
• Failure as a Process: An Anatomy of CLI Coding Agent Trajectories [21.513740407266354]
  Large language model (LLM) coding agents are increasingly deployed to autonomously perform software engineering tasks in terminal-based environments.<n>Existing empirical studies investigate why coding agents fail, yet they largely treat failure as a final outcome rather than a temporal process.<n>We present the first large-scale empirical study of CLI coding-agent failure trajectories, introducing a process-oriented framework that analyzes failure through its onset, evolution, and recovery across execution trajectories.
  arXiv  Detail & Related papers  (2026-07-10T15:25:56Z)
• PolicyTrim: Boosting Intrinsic Policy Efficiency of Vision-Language-Action Models [50.232333672172395]
  Vision-Language-Action (VLA) models provide a unified paradigm for robotic manipulation, yet their real-world deployment is often bottlenecked by execution efficiency.<n>We propose textbfPolicyTrim, a reinforcement learning-based post-training framework that extends the reliable action chunk length and reduces redundant physical steps.<n>Our framework delivers up to a 5.83$times$ end-to-end deployment speedup without compromising task success rates.
  arXiv  Detail & Related papers  (2026-06-21T14:54:07Z)
• ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox [61.862814740220806]
  $textbfComplexMCP$ is a benchmark designed to evaluate agents in rigorous conditions.<n>Built on the Model Context Protocol (MCP), $textbfComplexMCP$ provides over 300 meticulously tested tools.
  arXiv  Detail & Related papers  (2026-05-11T16:20:51Z)
• RubricRefine: Improving Tool-Use Agent Reliability with Training-Free Pre-Execution Refinement [0.36165327398913766]
  Iterative self-refinement is a popular inference-time reliability technique.<n>But its effectiveness in code-mode tool use depends heavily on the structure of the feedback signal.<n>We introduceRefine, a training-free method for pre-execution semantic contract verification.
  arXiv  Detail & Related papers  (2026-05-10T19:57:32Z)
• Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents [66.97968363332465]
  We introduce Claw-Eval, an end-to-end evaluation suite addressing all three gaps in agent benchmarks.<n>Claw-Eval comprises 300 human-verified tasks spanning 9 categories across three groups.<n>Every agent action is recorded through three independent evidence channels.
  arXiv  Detail & Related papers  (2026-04-07T17:43:18Z)
• AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents [50.481033105867205]
  We introduce AgentProcessBench, the first benchmark dedicated to evaluating step-level effectiveness in realistic, tool-augmented trajectories.<n>The benchmark comprises 1,000 diverse trajectories and 8,509 human-labeled step annotations with 89.1% inter-annotator agreement.<n>It features a ternary labeling scheme to capture exploration and an error propagation rule to reduce labeling ambiguity.
  arXiv  Detail & Related papers  (2026-03-15T16:13:58Z)
• Runaway is Ashamed, But Helpful: On the Early-Exit Behavior of Large Language Model-based Agents in Embodied Environments [54.67512489842682]
  Large language models (LLMs) have demonstrated strong planning and decision-making capabilities in complex embodied environments.<n>We take a first step toward exploring the early-exit behavior for LLM-based agents.
  arXiv  Detail & Related papers  (2025-05-23T08:23:36Z)
• SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints [59.645885492637845]
  SOPBench is an evaluation pipeline that transforms each service-specific SOP code program into a directed graph of executable functions.<n>Our approach transforms each service-specific SOP code program into a directed graph of executable functions and requires agents to call these functions based on natural language SOP descriptions.<n>We evaluate 18 leading models, and results show the task is challenging even for top-tier models.
  arXiv  Detail & Related papers  (2025-03-11T17:53:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.