LoopsBench: From Harness Engineering to Loop Engineering in Benchmarking Coding Agent

• URL: http://arxiv.org/abs/2608.00267v1
• Date: Fri, 31 Jul 2026 20:18:25 GMT
• Title: LoopsBench: From Harness Engineering to Loop Engineering in Benchmarking Coding Agent
• Abstract summary: LOOPSBENCH is a benchmark for loop engineering in coding agent evaluation.<n>It comprises 112 tasks from authentic sources spanning 8 programming languages and 9 domains.<n>Its flow-aware runtime releases tests along the ready frontier and retains completed nodes as regression obligations.
• Score: 60.27239351179265
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Coding agent infrastructure is shifting from harness engineering toward loop engineering as coding agents are deployed for sustained long-horizon software development. Existing benchmarks often center on localized tasks or end-state outcomes, offering limited insight into sustained execution. We introduce LOOPSBENCH, a long-horizon benchmark for loop engineering in coding agent evaluation. Each task is a dependency DAG over separately testable development units with source-evidenced prerequisite edges. LOOPSBENCH comprises 112 tasks from authentic sources spanning 8 programming languages and 9 domains. Its flow-aware runtime releases tests along the ready frontier and retains completed nodes as regression obligations. We evaluate frontier coding agents paired with widely used loop implementations. The strongest configuration, Opus-4.7 with Claude Code and outer continuation, resolves 25.00% of tasks. Recorded plans recover only part of the source-recovered prerequisite DAG, and regression events remain visible across the evaluated loop profiles. We open source the benchmark data and code, including all tasks, more than 5,300 development units, and executable tests, at microsoft/Loopsbench.

Related papers

• CLOSER-Bench: Evaluating Budgeted Cross-Stage Design Closure for Hardware Agents [5.604484678527336]
  CLOSER-Bench is a controlled evaluation protocol for budgeted cross-stage design closure.<n>It pairs spec-to-RTL, RTL-to-GDS, and spec-to-GDS tasks, records every simulator, synthesis, STA, and place-and-route invocation.<n>It measures final quality, anytime progress, tool cost, and cross-stage recovery.
  arXiv  Detail & Related papers  (2026-07-18T04:28:47Z)
• RigorBench: Benchmarking Engineering Process Discipline in Autonomous AI Coding Agents [0.0]
  RigorBench is the first benchmark to measure process discipline in AI coding agents.<n>It evaluates harnesses across five pillars: Planning Fidelity, Verification Coverage, Recovery Efficiency, Abstention Quality, and Atomic Transition Integrity.<n>Our results show that structured process discipline improves process quality scores by an average of 41% and raises downstream outcome correctness by 17%.
  arXiv  Detail & Related papers  (2026-06-21T21:41:34Z)
• ReCUBE: Evaluating Repository-Level Context Utilization in Code Generation [7.907933839674293]
  Large Language Models (LLMs) have emerged as capable coding assistants that operate over larges through either agentic exploration or full-context generation.<n>We introduce ReCUBE, a benchmark in which LLMs reconstruct a masked file within a real-world repository, using all remaining source files, dependency specifications, and documentation as their only source of context.<n>We propose the Caller-Centric Exploration (CCE) toolkit, a set of dependency graph-based tools that can be integrated into agentic frameworks.
  arXiv  Detail & Related papers  (2026-03-26T08:04:15Z)
• SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks [55.76734816061826]
  We introduce SlopCodeBench, a language-agnostic benchmark comprising 20 problems and 93 checkpoints.<n>We track two trajectory-level quality signals: verbosity, the fraction of redundant or duplicated code, and structural erosion.<n>No agent solves any problem end-to-end across 11 models; the highest checkpoint solve rate is 17.2%.
  arXiv  Detail & Related papers  (2026-03-25T19:26:44Z)
• ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development [72.4729759618632]
  We introduce ABC-Bench, a benchmark to evaluate agentic backend coding within a realistic, executable workflow.<n>We curated 224 practical tasks spanning 8 languages and 19 frameworks from open-source repositories.<n>Our evaluation reveals that even state-of-the-art models struggle to deliver reliable performance on these holistic tasks.
  arXiv  Detail & Related papers  (2026-01-16T08:23:52Z)
• NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents [79.29376673236142]
  Existing benchmarks fail to rigorously evaluate the long-horizon capabilities required to build complete software systems.<n>We present NL2Repo Bench, a benchmark explicitly designed to evaluate the long-horizon repository generation ability of coding agents.
  arXiv  Detail & Related papers  (2025-12-14T15:12:13Z)
• CodeTree: Agent-guided Tree Search for Code Generation with Large Language Models [106.11371409170818]
  Large language models (LLMs) can act as agents with capabilities to self-refine and improve generated code autonomously. We propose CodeTree, a framework for LLM agents to efficiently explore the search space in different stages of the code generation process. Specifically, we adopted a unified tree structure to explicitly explore different coding strategies, generate corresponding coding solutions, and subsequently refine the solutions.
  arXiv  Detail & Related papers  (2024-11-07T00:09:54Z)
• Codev-Bench: How Do LLMs Understand Developer-Centric Code Completion? [60.84912551069379]
  We present the Code-Development Benchmark (Codev-Bench), a fine-grained, real-world, repository-level, and developer-centric evaluation framework. Codev-Agent is an agent-based system that automates repository crawling, constructs execution environments, extracts dynamic calling chains from existing unit tests, and generates new test samples to avoid data leakage.
  arXiv  Detail & Related papers  (2024-10-02T09:11:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.