Baselines Before Architecture: Evaluating Coding Agents for Autonomous Penetration Testing

• URL: http://arxiv.org/abs/2607.13085v1
• Date: Mon, 13 Jul 2026 13:31:46 GMT
• Title: Baselines Before Architecture: Evaluating Coding Agents for Autonomous Penetration Testing
• Abstract summary: This paper presents a controlled study on the 104-task XBOW benchmark using default coding CLI agents as plain-agent baselines.<n>We first run Codex, OpenCode, and Pi with the same GPT-5 model, budget, target interface, and scoring rule.<n>We then compare the default Codex scaffold with published MAPTA and PentestGPT V2 results under the closest available model matches.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent autonomous penetration testing papers report high benchmark scores while adding multi-component security harnesses around frontier LLMs. Because these systems often change both architecture and backbone model, it is difficult to tell how much performance comes from the harness rather than from the underlying model. This paper presents a controlled study on the 104-task XBOW benchmark using default coding CLI agents as plain-agent baselines. We first run Codex, OpenCode, and Pi with the same GPT-5 model, budget, target interface, and scoring rule. This phase identifies the strongest same-model baseline and tests whether security-specific prompt variants improve its observed score. We then compare the default Codex scaffold with published MAPTA and PentestGPT V2 results under the closest available model matches. Finally, we repeat the plain-agent experiment with GPT-5.2 and GPT-5.5 to measure model scaling inside the same scaffold. The results show a mixed but practical picture. Specialised harnesses can add measurable benchmark lift and may improve cost efficiency, but plain coding agents already solve a large share of the benchmark; repeated plain-agent runs can match or exceed some published architecture scores in union coverage, and newer models substantially improve the same scaffold. Future evaluations should report model-matched plain-agent baselines before attributing benchmark gains to architecture design alone.

Related papers

• Exploration Structure in LLM Agents for Multi-File Change Localization [7.536387580547836]
  Most AI agents explore repositories linearly, that is, visiting one directory or file per step.<n>We compare linear sequential exploration against non-linear, domain-scoped parallel agentic exploration.<n>We construct an approach for persistent-session evaluation of GitHub issues anchored at a single base commit.
  arXiv  Detail & Related papers  (2026-06-10T11:54:14Z)
• The Meta-Agent Challenge: Are Current Agents Capable of Autonomous Agent Development? [80.24951682268332]
  We introduce the Meta-Agent Challenge (MAC), an evaluation framework designed to test the capacity of frontier models for autonomous agent development.<n>To ensure evaluation integrity, this framework is secured by multi-layer defenses against reward hacking.<n>We demonstrate that meta-agents rarely match human-engineered baseline policies, and the few that do are dominated by proprietary frontier models.
  arXiv  Detail & Related papers  (2026-06-03T04:58:17Z)
• Automating Formal Verification with Agent-Guided Tree Search [0.0]
  Formal verification offers a path to provably correct software, but writing verified code remains expensive enough that the technique is rarely used in production.<n>Recent benchmarks measure their ability to translate specifications into code and machine-checked iteration of correctness.<n>This thesis evaluates the state of such LLM-driven verified-code generation in Lean and develops search-based methods for improving performance.
  arXiv  Detail & Related papers  (2026-05-26T14:50:47Z)
• BatchBench: Toward a Workload-Aware Benchmark for Autoscaling Policies in Big Data Batch Processing -- A Proposed Framework [0.0]
  BatchBench is an open benchmarking framework designed to place rule-based, learned, and agentic autoscaling policies on equal footing.<n>We contribute: (1) a workload of six batch processing classes synthesized from published autoscaling benchmarks and publicly released cluster traces; (2) a validation methodology based on two-sample Kolmogorov-Smirnov and earth-mover distance; (3) a five-axis evaluation harness specification covering cost, SLA attainment, scaling responsiveness, scaling thrash, and decision interpretability; and (4) a standardized agent interface that lets LLM-based and reinforcement-learning autoscalers be evaluated alongside rule-based controllers with
  arXiv  Detail & Related papers  (2026-05-12T15:36:20Z)
• Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios [13.708123854369303]
  CLI-Tool-Bench is a structure-agnostic benchmark for evaluating the ground-up generation of Command-Line Interface tools.<n>It features 100 diverse real-world repositories evaluated via a black-box differential testing framework.<n>Agent-generated software is executed in sandboxes, comparing system side effects and terminal outputs against human-written oracles.
  arXiv  Detail & Related papers  (2026-04-08T07:09:10Z)
• IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation [85.56193980646981]
  We propose IF-RewardBench, a comprehensive meta-evaluation benchmark for instruction-following.<n>For each instruction, we construct a preference graph containing all pairwise preferences among multiple responses.<n>Experiments on IF-RewardBench reveal significant deficiencies in current judge models.
  arXiv  Detail & Related papers  (2026-03-05T02:21:17Z)
• Every Step Counts: Decoding Trajectories as Authorship Fingerprints of dLLMs [63.82840470917859]
  We show that the decoding mechanism of dLLMs can be used as a powerful tool for model attribution.<n>We propose a novel information extraction scheme called the Directed Decoding Map (DDM), which captures structural relationships between decoding steps and better reveals model-specific behaviors.
  arXiv  Detail & Related papers  (2025-10-02T06:25:10Z)
• SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge [7.655956608192742]
  We introduce SimpleQA Verified, a 1,000-prompt benchmark for evaluating Large Language Model (LLM) short-form factuality based on OpenAI's SimpleQA.<n>It addresses critical limitations in OpenAI's benchmark, including noisy and incorrect labels, topical biases, and question redundancy.<n>On this new benchmark, Gemini 2.5 Pro achieves a state-of-the-art F1-score of 55.6, outperforming other frontier models.
  arXiv  Detail & Related papers  (2025-09-09T17:53:58Z)
• Do Large Language Model Benchmarks Test Reliability? [66.1783478365998]
  We investigate how well current benchmarks quantify model reliability.<n>Motivated by this gap in the evaluation of reliability, we propose the concept of so-called platinum benchmarks.<n>We evaluate a wide range of models on these platinum benchmarks and find that, indeed, frontier LLMs still exhibit failures on simple tasks.
  arXiv  Detail & Related papers  (2025-02-05T18:58:19Z)
• When Liebig's Barrel Meets Facial Landmark Detection: A Practical Model [87.25037167380522]
  We propose a model that is accurate, robust, efficient, generalizable, and end-to-end trainable. In order to achieve a better accuracy, we propose two lightweight modules. DQInit dynamically initializes the queries of decoder from the inputs, enabling the model to achieve as good accuracy as the ones with multiple decoder layers. QAMem is designed to enhance the discriminative ability of queries on low-resolution feature maps by assigning separate memory values to each query rather than a shared one.
  arXiv  Detail & Related papers  (2021-05-27T13:51:42Z)
• Document Ranking with a Pretrained Sequence-to-Sequence Model [56.44269917346376]
  We show how a sequence-to-sequence model can be trained to generate relevance labels as "target words" Our approach significantly outperforms an encoder-only model in a data-poor regime.
  arXiv  Detail & Related papers  (2020-03-14T22:29:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.