Towards Optimal Agentic Architectures for Offensive Security Tasks

• URL: http://arxiv.org/abs/2604.18718v1
• Date: Mon, 20 Apr 2026 18:17:51 GMT
• Title: Towards Optimal Agentic Architectures for Offensive Security Tasks
• Abstract summary: We introduce a benchmark of 20 interactive targets (10 web/API and 10 binary)<n>Each exposing one-reachable ground-truth vulnerability, evaluated in whitebox and blackbox modes.<n> MAS-Indep attains the highest validated detection rate (64.2%), while SAS is the strongest efficiency baseline at $0.058 per validated finding.
• Score: 3.93181912653522
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agentic security systems increasingly audit live targets with tool-using LLMs, but prior systems fix a single coordination topology, leaving unclear when additional agents help and when they only add cost. We treat topology choice as an empirical systems question. We introduce a controlled benchmark of 20 interactive targets (10 web/API and 10 binary), each exposing one endpoint-reachable ground-truth vulnerability, evaluated in whitebox and blackbox modes. The core study executes 600 runs over five architecture families, three model families, and both access modes, with a separate 60-run long-context pilot reported only in the appendix. On the completed core benchmark, detection-any reaches 58.0% and validated detection reaches 49.8%. MAS-Indep attains the highest validated detection rate (64.2%), while SAS is the strongest efficiency baseline at $0.058 per validated finding. Whitebox materially outperforms blackbox (67.0% vs. 32.7% validated detection), and web materially outperforms binary (74.3% vs. 25.3%). Bootstrap confidence intervals and paired target-level deltas show that the dominant effects are observability and domain, while some leading whitebox topologies remain statistically close. The main result is a non-monotonic cost-quality frontier: broader coordination can improve coverage, but it does not dominate once latency, token cost, and exploit-validation difficulty are taken into account.

Related papers

• CockpitHAT: Dependency-Graph-Driven Hierarchical Attribution for Embodied Multi-Agent Cockpits [3.9502278152528625]
  LLM multi-agent systems suffer from Correctness Collapse, where high task-level accuracy conceals severe process-level failures.<n>We introduce CockpitHAT, a hierarchical attribution framework that replaces positional windows with dependency-distance thresholds from interaction DAGs.<n>We release CockpitBench, a benchmark of 212 annotated failure traces spanning dialogue, vehicle-state, environmental severity, and memory channels.
  arXiv  Detail & Related papers  (2026-08-03T07:13:00Z)
• Efficient Visual Pointing for Embodied AI:Agent-Driven Data Synthesis, Cross-Block Attention, and Iterative Correction [55.11480729304395]
  We describe our PointArena 2026 solution, which achieves 77.2% overall accuracy and ranks second on the benchmark.<n>The ap proach targets three failure modes. First, agent-driven syn thesis builds large semantic and anchor-relative candidate pools.<n>Second, a determinis tic steerable-data pipeline creates a verified 10,000-sample main set, plus reserve samples, using masks, templates, and path verification.
  arXiv  Detail & Related papers  (2026-06-29T06:39:03Z)
• Empirical Software Engineering TerraProbe: A Layered-Oracle Framework for Detecting Deceptive Fixes in LLM-Assisted Terraform [0.0]
  Security misconfigurations in Terraform Infrastructure-as-Code are a growing risk in cloud deployments.<n>Existing evaluations often treat a repair as successful when the targeted static-analysis finding disappears.<n>This paper presents TerraProbe, a five-layer oracle framework for evaluating LLM-assisted Terraform security repair.
  arXiv  Detail & Related papers  (2026-06-25T04:21:15Z)
• Are Frontier LLMs Ready for Cybersecurity? Evidence for Vertical Foundation Models from Dual-Mode Vulnerability Benchmarks [0.3303672705634661]
  We evaluate whether frontier LLMs are ready for cybersecurity through a dual-mode benchmark.<n>We test six frontier models (GPT-5.4, Codex5.3, Claude Opus4.6, Sonnet4.6, Gemini3.1Pro and Gemini3Flash) and two domain-specialized models across four testing paradigms.
  arXiv  Detail & Related papers  (2026-05-22T05:24:43Z)
• TRUST: A Framework for Decentralized AI Service v.0.1 [47.384270414446604]
  Large Reasoning Models (LRMs) and Multi-Agent Systems (MAS) in high-stakes domains demand reliable verification.<n>We introduce TRUST (Transparent, Robust, and Unified Services for Trustworthy AI), a decentralized framework with three innovations.<n>We prove a Safety-Profitability Theorem ensuring honest auditors profit while malicious actors incur losses.
  arXiv  Detail & Related papers  (2026-04-29T19:32:58Z)
• Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents [66.97968363332465]
  We introduce Claw-Eval, an end-to-end evaluation suite addressing all three gaps in agent benchmarks.<n>Claw-Eval comprises 300 human-verified tasks spanning 9 categories across three groups.<n>Every agent action is recorded through three independent evidence channels.
  arXiv  Detail & Related papers  (2026-04-07T17:43:18Z)
• Belief Dynamics for Detecting Behavioral Shifts in Safe Collaborative Manipulation [3.198214897627444]
  We study regime-switch detection under controlled non-stationarity in ManiSkill shared-workspace manipulation tasks.<n>Across ten detection methods and five random seeds, enabling detection reduces post-switch collisions by 52%.<n>We introduce UA-TOM, a lightweight belief-tracking module that augments frozen vision-language-action control backbones.
  arXiv  Detail & Related papers  (2026-04-04T02:41:09Z)
• AWE: Adaptive Agents for Dynamic Web Penetration Testing [0.0]
  AWE is a memory-augmented multi-agent framework for autonomous web penetration testing.<n>It embeds structured, vulnerability-specific analysis pipelines within a lightweight LLM orchestration layer.<n>AWE achieves substantial gains on injection-class vulnerabilities.
  arXiv  Detail & Related papers  (2026-03-01T07:32:42Z)
• Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active Parameters [169.7981969517903]
  Step 3.5 Flash bridges frontier-level agentic intelligence and computational efficiency.<n>We focus on what matters most when building agents: sharp reasoning and fast, reliable execution.
  arXiv  Detail & Related papers  (2026-02-11T07:53:51Z)
• Multi-Agent LLM Committees for Autonomous Software Beta Testing [0.0]
  The framework combines model diversity, persona-driven behavioral variation, and visual user interface understanding.<n>Vision-enabled agents successfully identify user interface elements, with navigation and reporting achieving 100 percent success.<n>The framework enables reproducible research and practical deployment of LLM-based software testing in CI/CD pipelines.
  arXiv  Detail & Related papers  (2025-12-21T02:06:53Z)
• Towards a Science of Scaling Agent Systems [79.64446272302287]
  We formalize a definition for agent evaluation and characterize scaling laws as the interplay between agent quantity, coordination structure, modelic, and task properties.<n>We derive a predictive model using coordination metrics, that cross-validated R2=0, enabling prediction on unseen task domains.<n>We identify three effects: (1) a tool-coordination trade-off: under fixed computational budgets, tool-heavy tasks suffer disproportionately from multi-agent overhead, and (2) a capability saturation: coordination yields diminishing or negative returns once single-agent baselines exceed 45%.
  arXiv  Detail & Related papers  (2025-12-09T06:52:21Z)
• Balanced Multi-Task Attention for Satellite Image Classification: A Systematic Approach to Achieving 97.23% Accuracy on EuroSAT Without Pre-Training [0.0]
  This work presents a systematic investigation of custom convolutional neural network architectures for satellite land use classification.<n>We achieve 97.23% test accuracy on the EuroSAT dataset without reliance on pre-trained models.<n>Our approach achieves performance within 1.34% of fine-tuned ResNet-50 (98.57%) while requiring no external data.
  arXiv  Detail & Related papers  (2025-10-17T10:59:24Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• Multi-Agent Penetration Testing AI for the Web [3.93181912653522]
  MAPTA is a multi-agent system for autonomous web application security assessment.<n>It combines large language model orchestration with tool-grounded execution and end-to-end exploit validation.<n>On the 104-challenge XBOW benchmark, MAPTA achieves 76.9% overall success.
  arXiv  Detail & Related papers  (2025-08-28T14:14:24Z)
• G$^2$uardFL: Safeguarding Federated Learning Against Backdoor Attacks through Attributed Client Graph Clustering [116.4277292854053]
  Federated Learning (FL) offers collaborative model training without data sharing. FL is vulnerable to backdoor attacks, where poisoned model weights lead to compromised system integrity. We present G$2$uardFL, a protective framework that reinterprets the identification of malicious clients as an attributed graph clustering problem.
  arXiv  Detail & Related papers  (2023-06-08T07:15:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.