Coding Agents as Test-Suite Auditors: Finding What Official Suites Miss While Approaching What They Catch

• URL: http://arxiv.org/abs/2608.01715v1
• Date: Mon, 03 Aug 2026 05:24:03 GMT
• Title: Coding Agents as Test-Suite Auditors: Finding What Official Suites Miss While Approaching What They Catch
• Abstract summary: An off-the-shelf coding agent, serving as a test-suite auditor, both builds adversarial test suites to expose what official suites miss.<n>A certification chain determines whether each agent-flagged submission is genuinely buggy without relying on the official judge.<n>On post-cutoff Codeforces problems with no available official suites, the same test-building method leads all five reproduced baselines at every tested input budget.
• Score: 47.836680625916266
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Online-judge verdicts and the datasets and benchmarks built on them are treated as ground truth for evaluating and training large language models for code. Yet prior audits have sounded a warning: official suites accept buggy submissions. These audits, however, stop at the warning and offer no practical remedy. Our remedy has two parts: an off-the-shelf coding agent, serving as a test-suite auditor, both builds adversarial test suites to expose what official suites miss and supplies these suites where no official suite exists; a certification chain determines whether each agent-flagged submission is genuinely buggy without relying on the official judge: multiple independently written accepted solutions agree on the expected output for every test, brute-force solutions settle disagreements, and a per-problem validator certifies each failing input legal. One such agent identifies 589 verified accepted-but-buggy submissions among AtCoder's 20,375 audited accepted submissions; extending the same certification to all five agents yields a union floor of 906 such submissions. Five agents, scored separately, each stay within 1.7pp of official-suite coverage on logic bugs those suites catch. On post-cutoff Codeforces problems with no available official suites, the same test-building method leads all five reproduced baselines at every tested input budget. Where an official suite exists, the agent audits suite adequacy instead of assuming it; where none exists, agent suites catch the most buggy submissions among methods we reproduced and tested.

Related papers

• SaltBench: A Referee-Gated Protocol for Measuring Method Effects in Machine-Checked Software Work [0.0]
  SaltBench is a benchmark protocol for one question: How does a machine referee change the way a coding agent works?<n>In this study, the subject of the benchmark is a seat'', meaning an agent session in its standard harness.<n>We tested five systems components, all authored in Rust under a pinned Verus toolchain, with a withheld test suite as the referee for each.
  arXiv  Detail & Related papers  (2026-09-10T04:32:43Z)
• ExecCritic: Learn to Test, Test to Improve for Coding Agents [68.42713285082912]
  Execution feedback can guide coding agents toward correct repository repairs, but only when the tests capture the behavior requested by the issue.<n>We introduce ExecCritic, combining a test--verify--revise scaffold with a role-specific reinforcement learning recipe for training agents within it.<n>The scaffold separates test construction from source-code repair: a Test agent independently generates repository-native tests, a fail-closed harness qualifies and freezes them, and a Repair agent revises source code from their execution feedback without changing the tests.
  arXiv  Detail & Related papers  (2026-09-08T17:53:37Z)
• Grounding AI Agents in Contracts: An Empirical Evaluation of Spec-Driven Test Generation [4.061618191574875]
  Spec-Driven Test Generation instructs an agent to first reason about -- and explicitly document -- code pre-conditions, post-conditions, and undefined behaviors.<n>This intermediate semi-formal specification acts as a cognitive scaffold to guide subsequent test generation.
  arXiv  Detail & Related papers  (2026-08-17T22:36:12Z)
• A First Look at Coding Agents' Compliance with AI Contribution Rules in Open-Source Communities [13.02344624749338]
  We estimate real-world rule compliance of coding agents in open source repositories.<n>Our experiments show that today's agents almost never proactively retrieve the contribution rules.<n>The status reveals that verification and disclosure issues are solvable with existing mechanisms.
  arXiv  Detail & Related papers  (2026-07-29T12:14:17Z)
• Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction [57.138342889101345]
  Tencent WorkBuddy Bench is a multi-domain evaluation suite for coding agents.<n>This report documents its construction methodology, scoring protocol, and a cross-model leaderboard.
  arXiv  Detail & Related papers  (2026-07-23T04:34:06Z)
• Fantastic Adaptive Taxonomies and How to Use Them [100.20614173157708]
  AdaMAST converts execution traces into a compact, evidence-grounded failure taxonomy.<n>No code is hand-authored, no trace human-annotated.<n>In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form.
  arXiv  Detail & Related papers  (2026-07-17T17:41:16Z)
• The Prover Is the Judge: Verified Security Software from AI Coding Agents in Ada/SPARK [0.0]
  In a verifier-driven loop, AI agents wrote and verified bare-metal security software in Ada/SPARK spanning classical and post-quantum cryptography, TLS 1.3, IKEv2, X.509, and a Matrix client.<n>GNATprove discharged 49,280 proof obligations, established functional correctness for selected primitives, and proved the absence of run-time errors for the rest.<n>We report where each layer caught faults and draw the central lesson: what an agent can be trusted to establish is bounded by the strength of its feedback.
  arXiv  Detail & Related papers  (2026-07-15T20:09:05Z)
• Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits [0.30586855806896046]
  We name five classes of pipeline failure and demonstrate each in a self-audit over safety benchmarks and open-weight instruction-tuned models.<n>We position the gate as a withholding and disclosure protocol for assurance-grade evidence, supplementary to (not a replacement for) classical construct-validity evidence.
  arXiv  Detail & Related papers  (2026-07-01T05:30:07Z)
• Bayesian control for coding agents [63.64172141184361]
  We propose a cost-sensitive sequential hypothesis testing framework for coding agents.<n>A Bayesian controller maintains a belief over correctness and decides whether to gather more evidence, refine the candidate, verify it, or stop.<n>We show that the belief state yields an interpretable correctness score that outperforms token-probability and raw tool-success baselines for uncertainty quantification.
  arXiv  Detail & Related papers  (2026-06-23T11:41:32Z)
• AgentBeats: Agentifying Agent Assessment for Openness, Standardization, and Reproducibility [104.46861849039357]
  Agent systems are advancing quickly across domains, but their evaluation remains fragmented.<n>The root problem is the lack of an open, agent-agnostic assessment interface.<n>We advocate Agentified Agent Assessment (AAA), where evaluation is performed by judge agents and all participants interact through standardized protocols.
  arXiv  Detail & Related papers  (2026-06-11T17:23:54Z)
• Iterative Audit Convergence in LLM-Managed Multi-Agent Systems: A Case Study in Prompt Engineering Quality Assurance [0.0]
  This paper reports a single-system empirical case study of agent-driven auditing applied to AEGIS.<n>We report a seven-category post-hoc defect taxonomy with explicit coding rules, observed non-monotonic convergence, and audit-scope expansion.
  arXiv  Detail & Related papers  (2026-05-12T15:39:04Z)
• BenchGuard: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks [26.58983143152204]
  BenchGuard is the first automated auditing framework for task-oriented, execution-based agent benchmarks.<n>It identifies 12 author-confirmed issues in ScienceAgentBench and 83.3% of expert-identified issues on the BIXBench Verified-50 subset.<n>A full audit of 50 complex bioinformatics tasks costs under USD 15, making automated benchmark auditing a practical and valuable complement to human review.
  arXiv  Detail & Related papers  (2026-04-27T19:51:25Z)
• VulAgent: Hypothesis-Validation based Multi-Agent Vulnerability Detection [55.957275374847484]
  VulAgent is a multi-agent vulnerability detection framework based on hypothesis validation.<n>It implements a semantics-sensitive, multi-view detection pipeline, each aligned to a specific analysis perspective.<n>On average, VulAgent improves overall accuracy by 6.6%, increases the correct identification rate of vulnerable--fixed code pairs by up to 450%, and reduces the false positive rate by about 36%.
  arXiv  Detail & Related papers  (2025-09-15T02:25:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.