LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering

• URL: http://arxiv.org/abs/2608.28281v1
• Date: Fri, 28 Aug 2026 12:44:54 GMT
• Title: LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering
• Abstract summary: Loop Engineering is emerging as a practice for organizing development work around coding agents.<n>We introduce LoopArena, a benchmark for evaluating how well one model can guide a separate coding agent through a long-running task.
• Score: 47.39676953516343
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Loop Engineering is emerging as a practice for organizing development work around coding agents. Instead of writing each prompt by hand, practitioners design loops that monitor progress, assign work, run checks, and decide what the agent should do next. Even with a capable coding agent, a loop may trust a stale progress note, skip needed verification, spend its budget in the wrong direction, or stop before the task is safe to submit. Yet the final outcome of one end-to-end run cannot tell whether success or failure reflects the loop's guidance or the coding agent's ability to carry out the task. We introduce LoopArena, a benchmark for evaluating how well one model can guide a separate coding agent through a long-running task. The model under evaluation is the \textbf{Controller}: after each coding round, it receives a structured summary of the run and instructs a separate, fixed coding agent, the \textbf{Worker}, on what to do or verify next, or decides whether to stop. LoopArena evaluates this ability in three complementary settings that differ in execution scope and cost. Type I scores next-step Loop Contract selection through execution-validated questions without running the Worker at evaluation time. Type II executes repeated control over a selected slice of a full task, while Type III evaluates the paired full task from its original state. On full tasks, the best observed Strict Success Rate is \textbf{24.69\%}, leaving substantial room for improvement in long-horizon loop control. Across Controllers, the paired reduction in estimated inference cost averages \textbf{64.4\%}, and Type II produces a similar ordering under the main Core criterion (Spearman's \(ρ=\textbf{0.9747}\)). We release the benchmark data and evaluation code at https://github.com/AMAP-ML/LoopArena .

Related papers

• Iris: Climbing to the Search Frontier [21.066541702697112]
  We present Iris-mini and Iris-pro, two search agents trained at the 35B-A3B and 397B-A17B scales, together with the data pipeline and training recipe behind them.
  arXiv  Detail & Related papers  (2026-09-03T17:51:01Z)
• Verification-Aware Training for Speculative Decoding [57.84976863792784]
  We introduce Verification-Aware Training (VAT), a plug-in framework that simulates verification at every training step and turns the resulting accept and reject patterns into supervision.<n>VAT modifies only the training objective, so it can be layered on top of existing methods without changing the draft architecture, the target model, or the inference procedure.<n>VAT improves average acceptance length by up to 11.4% and wall-clock speedup by up to 8.7%, with consistent gains across math, code, and chat benchmarks.
  arXiv  Detail & Related papers  (2026-08-31T01:42:10Z)
• Task-CoEvolve: Efficient Harness Optimization via Adaptive Validation Task Selection [51.39973047614183]
  We present a novel approach to efficient LLM harness optimization through adaptive validation task selection.<n>$textbfTask-CoEvolve$, which co-evolves the validation tasks with the harness by addressing two challenges: selecting informative tasks and estimating full-set performance from partial evaluations.
  arXiv  Detail & Related papers  (2026-08-20T15:24:54Z)
• LoopsBench: From Harness Engineering to Loop Engineering in Benchmarking Coding Agent [60.27239351179265]
  LOOPSBENCH is a benchmark for loop engineering in coding agent evaluation.<n>It comprises 112 tasks from authentic sources spanning 8 programming languages and 9 domains.<n>Its flow-aware runtime releases tests along the ready frontier and retains completed nodes as regression obligations.
  arXiv  Detail & Related papers  (2026-07-31T20:18:25Z)
• OrchBench: Evaluating Multi-Agent Orchestration Plans in Isolation via Deterministic Simulation [41.885817564161975]
  We present OrchBench, a simulation-based benchmark for evaluating multi-agent orchestration plans in isolation.<n>A deterministic simulator evaluates the resulting plan without invoking worker agents and returns interpretable measures of result quality, makespan, and token cost.<n>Across diverse planners and workflow scales, we find that preserving task-critical information is more important than simply increasing the number of agents.
  arXiv  Detail & Related papers  (2026-07-28T12:43:34Z)
• Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction [57.138342889101345]
  Tencent WorkBuddy Bench is a multi-domain evaluation suite for coding agents.<n>This report documents its construction methodology, scoring protocol, and a cross-model leaderboard.
  arXiv  Detail & Related papers  (2026-07-23T04:34:06Z)
• Autoresearch with Coding Agents: Generalizers and Metric-Maximizers on Quran Recitation Data [4.191965713559235]
  Coding agents can now be left alone to improve software against a score.<n>In this pattern, the agent receives a dataset, an evaluation script, and one editable file, and iterates without supervision.<n>We ran this loop on a real production task: deciding which Quranic verses appear in a noisy speech-recognition transcript.
  arXiv  Detail & Related papers  (2026-07-20T15:32:09Z)
• CLOSER-Bench: Evaluating Budgeted Cross-Stage Design Closure for Hardware Agents [5.604484678527336]
  CLOSER-Bench is a controlled evaluation protocol for budgeted cross-stage design closure.<n>It pairs spec-to-RTL, RTL-to-GDS, and spec-to-GDS tasks, records every simulator, synthesis, STA, and place-and-route invocation.<n>It measures final quality, anytime progress, tool cost, and cross-stage recovery.
  arXiv  Detail & Related papers  (2026-07-18T04:28:47Z)
• LACUNA: Safe Agents as Recursive Program Holes [3.2613419151327343]
  LLM agents increasingly act by writing code, yet a split persists between the runtime that drives the agent and the code the model writes.<n>We present LACUNA, a programming model for agents that closes this split while preserving safety.<n>Our primitive expresses ReAct loops, sub-agents, skills, parallel decomposition, and multi-model planning as ordinary control flow.
  arXiv  Detail & Related papers  (2026-05-27T15:27:25Z)
• ExecVerify: White-Box RL with Verifiable Stepwise Rewards for Code Execution Reasoning [9.137158235106941]
  We introduce ExecVerify, which goes beyond text imitation by incorporating verifiable white-box rewards derived from execution traces.<n>Experiments demonstrate that a 7B model trained with ExecVerify achieves performance comparable to 32B models on code reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-03-11T18:49:45Z)
• AlgoVeri: An Aligned Benchmark for Verified Code Generation on Classical Algorithms [54.99368693313797]
  Existing benchmarks test only individual languages/tools, so the performance numbers are not directly comparable.<n>We address this gap with AlgoVeri, a benchmark that evaluates vericoding of $77$ classical algorithms in Dafny, Verus, and Lean.
  arXiv  Detail & Related papers  (2026-02-10T06:58:26Z)
• Rethinking Thinking Tokens: LLMs as Improvement Operators [80.12087211785949]
  Reasoning training incentivizes LLMs to produce long chains of thought (long CoT), which allows them to explore solution strategies with self-checking.<n>This results in higher accuracy, but inflates context length, token/compute cost, and answer latency.<n>We ask: Can current models leverage their metacognition to provide other combinations on this Pareto frontier?<n>We identify an interesting inference family Parallel-Distill-Refine (PDR), which performs the following: (i) generate diverse drafts in parallel; (ii) distill them into a bounded, textual workspace; and (iii) refine conditioned on this workspace
  arXiv  Detail & Related papers  (2025-10-01T17:08:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.