Related papers: RigorBench: Benchmarking Engineering Process Discipline in Autonomous AI Coding Agents

RigorBench: Benchmarking Engineering Process Discipline in Autonomous AI Coding Agents

URL: http://arxiv.org/abs/2606.22678v2
Date: Mon, 29 Jun 2026 19:02:41 GMT
Title: RigorBench: Benchmarking Engineering Process Discipline in Autonomous AI Coding Agents
Abstract summary: RigorBench is the first benchmark to measure process discipline in AI coding agents.<n>It evaluates harnesses across five pillars: Planning Fidelity, Verification Coverage, Recovery Efficiency, Abstention Quality, and Atomic Transition Integrity.<n>Our results show that structured process discipline improves process quality scores by an average of 41% and raises downstream outcome correctness by 17%.
Score: 0.0
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Abstract: Agentic coding harnesses - such as Agent-Skills, Superpowers, and Agent-Rigor - are increasingly deployed to augment underlying LLMs for real-world software engineering tasks. Existing benchmarks evaluate these agents almost exclusively on outcome correctness: whether generated code passes tests or resolves issues. We argue that this outcome-only lens is insufficient: an agent that arrives at a correct solution through reckless trial-and-error, without planning, verification, or graceful recovery, is fundamentally less reliable than one that follows sound engineering discipline. We introduce RigorBench, the first benchmark designed to measure process discipline in AI coding agents. RigorBench evaluates these harnesses across five pillars: Planning Fidelity, Verification Coverage, Recovery Efficiency, Abstention Quality, and Atomic Transition Integrity. A composite RigorScore aggregates these dimensions into a single metric via a weighted sum. We curate a suite of 30 tasks spanning five categories - Plan-Then-Build, Verify-Or-Die, Doom Loop Gauntlet, Know When to Fold, and Don't Break the Build-and evaluate leading harnesses in a controlled with/without experimental design against baseline coding assistants. Our results show that structured process discipline not only improves process quality scores by an average of 41% but also raises downstream outcome correctness by 17%, providing the first quantitative evidence that how agents code matters as much as what they produce. We release the full benchmark, scoring rubrics, and trajectory analysis tools as open-source artifacts.

Related papers

HarnessOpt-Bench: Evaluating LLMs at Harness Optimization [3.846854817984248]
We introduce a benchmark for end-to-end harness optimization under expensive evaluation regimes.<n>A trusted execution environment enforces the evaluation boundary, target-agent resource use, and preserves candidate versions for audit.<n>Experiment results show that models separate more than the coding harnesses they act through, native harnesses are not consistently superior, and gains vary substantially across tasks.
arXiv Detail & Related papers (2026-08-06T17:21:05Z)
Do AI Agents Know When a Task Is Simple? Toward Complexity-Aware Reasoning and Execution [1.733151600403503]
Large language model (LLM) agents increasingly automate multi-step engineering and informatics.<n>They often follow a maximum-context-first strategy--re-reading files and dependencies they have already seen--turning a one-line edit into a small code-base audit.<n>We argue the missing capability is task-aware execution-scope estimation.
arXiv Detail & Related papers (2026-07-14T17:59:31Z)
TTHE: Test-Time Harness Evolution [50.26245555541721]
Existing approaches optimize such harnesses before deployment, searching training or development data for a fixed agent workflow that is then frozen at test time.<n>We ask whether the harness can instead be optimized during evaluation itself, using only the unlabeled execution traces the agent produces on the test inputs.<n>During evaluation, TTHE maintains a population of candidate harnesses and refines them through an agentic proposer that reasons over their execution traces.<n>A judge then commits an improved harness from execution-derived proxy signals, and the selected program persists to govern subsequent inputs.
arXiv Detail & Related papers (2026-07-09T05:53:39Z)
Socratic-SWE: Self-Evolving Coding Agents via Trace-Derived Agent Skills [12.442806027914097]
We introduce Socratic-SWE, a closed-loop self-evolution framework that reuses the agent's historical solving traces as a source of training signal.<n>Rather than treating traces only as evidence for reward computation, Socratic-SWE distills them into structured agent skills that summarize recurring failures and effective repair patterns.
arXiv Detail & Related papers (2026-06-05T16:00:17Z)
Can Coding Agents Reproduce Findings in Computational Materials Science? [49.254975563645786]
We present AutoMat, a benchmark for evaluating large language models' ability to reproduce scientific claims.<n>By working closely with subject matter experts, we curate a set of claims from real materials science papers to test whether coding agents can recover and execute the end-to-end workflow.<n>Results show that current LLM-based agents obtain low overall success rates on AutoMat, with the best-performing setting achieving a success rate of only 54.1%.
arXiv Detail & Related papers (2026-05-01T17:42:12Z)
ELT-Bench-Verified: Benchmark Quality Issues Underestimate AI Agent Capabilities [4.5258165293324515]
Extract-Load-Transform pipelines are a labor-intensive data engineering task and a high-impact target for AI automation.<n>On ELT-Bench, the first benchmark for end-to-end ELT pipeline construction, AI agents initially showed low success rates.<n>We revisit these results and identify two factors causing a substantial underestimation of agent capabilities.
arXiv Detail & Related papers (2026-03-31T08:02:16Z)
SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks [55.76734816061826]
We introduce SlopCodeBench, a language-agnostic benchmark comprising 20 problems and 93 checkpoints.<n>We track two trajectory-level quality signals: verbosity, the fraction of redundant or duplicated code, and structural erosion.<n>No agent solves any problem end-to-end across 11 models; the highest checkpoint solve rate is 17.2%.
arXiv Detail & Related papers (2026-03-25T19:26:44Z)
Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
arXiv Detail & Related papers (2025-09-25T14:05:55Z)
Establishing Best Practices for Building Rigorous Agentic Benchmarks [94.69724201080155]
We show that many agentic benchmarks have issues in task setup or reward design.<n>Such issues can lead to under- or overestimation of agents' performance by up to 100% in relative terms.<n>We introduce the Agentic Benchmark Checklist (ABC), a set of guidelines that we synthesized from our benchmark-building experience.
arXiv Detail & Related papers (2025-07-03T17:35:31Z)
An LLM-as-Judge Metric for Bridging the Gap with Human Evaluation in SE Tasks [15.820416019287622]
SE-Jury is the first evaluation metric for LLM-as-Ensemble-Judge.<n>We evaluate SE-Jury across a diverse set of software engineering (SE) benchmarks.
arXiv Detail & Related papers (2025-05-27T08:04:34Z)
Self-Evaluation Guided Beam Search for Reasoning [61.523627290397556]
We introduce a stepwise self-evaluation mechanism to guide and calibrate the reasoning process of Large Language Model (LLM) We propose a decoding algorithm integrating the self-evaluation guidance via beam search. Our approach surpasses the corresponding Codex-backboned baselines in few-shot accuracy by $6.34%$, $9.56%$, and $5.46%$ on the GSM8K, AQuA, and StrategyQA.
arXiv Detail & Related papers (2023-05-01T02:37:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.