Cochise: A Reference Harness for Autonomous Penetration Testing

• URL: http://arxiv.org/abs/2605.11671v1
• Date: Tue, 12 May 2026 07:28:12 GMT
• Title: Cochise: A Reference Harness for Autonomous Penetration Testing
• Abstract summary: We present cochise, a 597 LOC Python reference harness for autonomous penetration-testing.<n>Cochise connects an LLM-driven agent to a Linux host over SSH and supports controlled target environments.<n>We release replay and analysis tools: (i) cochise-replay for offline visualization of captured runs, (ii) cochise-analyzelogs and cochise-analyze-a-graphs for cost, token duration, and compromise analysis.
• Score: 1.3124479769761592
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent work on LLM-driven autonomous penetration testing reports promising results, but existing systems often combine many architectural, prompting, and tool-integration choices, making it difficult to tell what is gained over a simple agent scaffold. We present cochise, a 597 LOC Python reference harness for autonomous penetration-testing experiments. Cochise connects an LLM-driven agent to a Linux execution host over SSH and supports controlled target environments reachable from that jump host. The prototype implements a separated Planner--Executor architecture in which long-term state is maintained outside the LLM context, while a ReAct-style executor issues commands over SSH and self-corrects based on command outputs. The scenario prompt can be adapted to different target environments. To demonstrate the efficacy of our minimal harness, we evaluate it against a live third-party testbed called Game of Active Directory (GOAD). Alongside the harness, we release replay and analysis tools: (i) cochise-replay for offline visualization of captured runs, (ii) cochise-analyze-alogs and cochise-analyze-graphs for cost, token, duration, and compromise analysis, and (iii) a corpus of JSON trajectory logs from GOAD runs, allowing researchers to study agent behavior without provisioning the 48--64 GB RAM / 190 GB storage testbed themselves. Cochise is intended not as a state-of-the-art pen-testing agent, but as reusable experimental infrastructure for comparing models, agent architectures, and penetration-testing traces.

Related papers

• DuMateBench: Evaluating Autonomous Agents in Complex Real-World Workflows [69.12339622053588]
  We introduce DuMateBench, a real-session benchmark reconstructed from anonymized and privacy-screened user sessions.<n>The resulting benchmark comprises 200 tasks spanning 8 broad scenarios and 17 fine-grained capability categories.<n>We execute these tasks in isolated Docker containers injected with three forms of real-world environmental complexity: Insufficient, Unstable, and Noisy.
  arXiv  Detail & Related papers  (2026-08-27T02:36:23Z)
• ScrambleToolBench: Agents Search Exhaustively Even When Their Own Map Points to the Next Step [74.83535434786145]
  We introduce ScrambleToolBench, an interactive terminal benchmark designed to isolate behavioral reasoning.<n>The benchmark requires agents to uncover hidden tool behaviors entirely through trial-and-error interaction.<n>Our evaluation of state-of-the-art language models reveals that successful initial discovery does not translate into robust adaptation.
  arXiv  Detail & Related papers  (2026-08-03T15:07:46Z)
• PhyAgentOS: A Self-Evolving Operating System for Embodied Agents with Decoupled Cognitive Planning and Physical Execution [49.776611937968]
  We present PhyAgentOS, a foundation delivering scheduling, verification, memory, benchmarking, and safety as system-level services.<n>Its Session-Centered treats a session, not an action, as the minimum unit of scheduling, compatibility, supervised execution, evidence collection, and acceptance.<n>SessionVerifier distinguishes execution termination from semantic task completion via evidence-grounded verdicts of success, failure, or replan.<n> Benchmarking reuses the deployment session and verification path, so results trace to real execution.
  arXiv  Detail & Related papers  (2026-07-18T04:46:53Z)
• TTHE: Test-Time Harness Evolution [50.26245555541721]
  Existing approaches optimize such harnesses before deployment, searching training or development data for a fixed agent workflow that is then frozen at test time.<n>We ask whether the harness can instead be optimized during evaluation itself, using only the unlabeled execution traces the agent produces on the test inputs.<n>During evaluation, TTHE maintains a population of candidate harnesses and refines them through an agentic proposer that reasons over their execution traces.<n>A judge then commits an improved harness from execution-derived proxy signals, and the selected program persists to govern subsequent inputs.
  arXiv  Detail & Related papers  (2026-07-09T05:53:39Z)
• Automating the Design of Embodied Agent Architectures [17.294577626616917]
  Embodied agents are typically built as hand-designed compositions of perception, memory, planning, and action modules.<n>We introduce a typed-graph runtime that hosts embodied executors as editable node-and-wire programs with simulator-aware execution and episode-level logs.<n>We evaluate three AAS variants across four embodied executors spanning vision-language navigation, embodied question answering, and language-conditioned manipulation.
  arXiv  Detail & Related papers  (2026-06-29T10:45:37Z)
• Do you dare to try Test-Driven Forensics? Increasing Trust in Desktop Forensics with ADARE [0.0]
  We present test-driven forensics, a practical approach that treats forensic expectations as executable specifications.<n>We implement the methodology in ADARE, an open-source framework that runs controlled experiments in virtual machines.<n>A 25-version regression study of Autopsy reveals substantial, largely undocumented changes in exported report outputs.
  arXiv  Detail & Related papers  (2026-05-27T13:38:06Z)
• Collider-Bench: Benchmarking AI Agents with Particle Physics Analysis Reproduction [0.02446672595462589]
  We introduce Collider-Bench, a benchmark for evaluating whether language-model agents can reproduce experimental analyses from the Large Hadron Collider (LHC) using only public papers and open scientific software.<n>Agents must therefore rely on physical reasoning, domain knowledge, and trial-and-error to fill these gaps.<n>Our results show that on average no agent reliably beats the physicist-in-the-loop solution.
  arXiv  Detail & Related papers  (2026-05-13T18:00:00Z)
• Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interaction [127.64173950476702]
  We study direct corpus interaction (DCI), where an agent searches the raw corpus directly with general-purpose terminal tools.<n>This approach requires no offline indexing and adapts naturally to evolving local corpora.<n>Across IR benchmarks and end-to-end agentic search tasks, this simple setup substantially outperforms strong sparse, dense, and reranking baselines.
  arXiv  Detail & Related papers  (2026-05-03T19:13:11Z)
• Towards Multi-Agent Autonomous Reasoning in Hydrodynamics [0.06999740786886537]
  We present a multi-agent system (MAS) prototype for hydrodynamics in which specialized agents are coordinated through a Layer Execution Graph (LEG)<n>A planner agent constructs query-specific execution topologies from natural-language routings that capture domain knowledge without hard-coding it as rigid control logic.<n>A reporter agent synthesizes the final response, while the runtime logs provenance for every tool invocation to support auditability.
  arXiv  Detail & Related papers  (2026-05-01T21:17:55Z)
• Springdrift: An Auditable Persistent Runtime for LLM Agents with Case-Based Memory, Normative Safety, and Ambient Self-Perception [0.20305676256390928]
  We present Springdrift, a persistent runtime for long-lived LLM agents.<n>We introduce the term Artificial Retainer for this category.<n>This is a technical report on a systems design and deployment case study, not a benchmark-driven evaluation.
  arXiv  Detail & Related papers  (2026-04-06T13:14:37Z)
• KAIJU: An Executive Kernel for Intent-Gated Execution of LLM Agents [5.931389072124715]
  Tool-calling autonomous agents based on large language models using ReAct exhibit three limitations.<n>Recent work moves towards separating planning from execution but in each case model remains coupled to the execution mechanics.<n>We introduce a system-level abstraction for LLM agents which decouples the execution agent from the reasoning layer.
  arXiv  Detail & Related papers  (2026-03-31T21:38:28Z)
• Dynamic analysis enhances issue resolution [53.50448142467294]
  DAIRA (Dynamic Analysis-enhanced Issue Resolution Agent) is an automated repair framework that embeds dynamic analysis into the agent's reasoning cycle.<n> Driven by a Test Tracing-Driven methodology, DAIRA utilizes lightweight monitors to extract critical runtime data.<n>When powered by Gemini 3 Flash Preview, DAIRA establishes a new state-of-the-art (SOTA) performance, achieving a 79.4% resolution rate on the SWE-bench Verified dataset.
  arXiv  Detail & Related papers  (2026-03-23T14:48:54Z)
• ESAA: Event Sourcing for Autonomous Agents in LLM-Based Software Engineering [0.0]
  This paper presents the ESAA (Event Sourcing for Autonomous Agents) architecture.<n>The architecture separates the cognitive intention from the project's state mutation, inspired by the Event Sourcing pattern.<n>Two case studies validate the architecture, providing empirical evidence of the architecture's scalability beyond single-agent scenarios.
  arXiv  Detail & Related papers  (2026-02-26T16:45:59Z)
• Automated structural testing of LLM-based agents: methods, framework, and case studies [0.05254956925594667]
  LLM-based agents are rapidly being adopted across diverse domains.<n>Current testing approaches focus on acceptance-level evaluation from the user's perspective.<n>We present methods to enable structural testing of LLM-based agents.
  arXiv  Detail & Related papers  (2026-01-25T11:52:30Z)
• InspectCoder: Dynamic Analysis-Enabled Self Repair through interactive LLM-Debugger Collaboration [71.18377595277018]
  Large Language Models (LLMs) frequently generate buggy code with complex logic errors that are challenging to diagnose.<n>We present InspectCoder, the first agentic program repair system that empowers LLMs to actively conduct dynamic analysis via interactive debugger control.
  arXiv  Detail & Related papers  (2025-10-21T06:26:29Z)
• From Reproduction to Replication: Evaluating Research Agents with Progressive Code Masking [48.90371827091671]
  AutoExperiment is a benchmark that evaluates AI agents' ability to implement and run machine learning experiments.<n>We evaluate state-of-the-art agents and find that performance degrades rapidly as $n$ increases.<n>Our findings highlight critical challenges in long-horizon code generation, context retrieval, and autonomous experiment execution.
  arXiv  Detail & Related papers  (2025-06-24T15:39:20Z)
• Get my drift? Catching LLM Task Drift with Activation Deltas [55.75645403965326]
  Task drift allows attackers to exfiltrate data or influence the LLM's output for other users.<n>We show that a simple linear classifier can detect drift with near-perfect ROC AUC on an out-of-distribution test set.<n>We observe that this approach generalizes surprisingly well to unseen task domains, such as prompt injections, jailbreaks, and malicious instructions.
  arXiv  Detail & Related papers  (2024-06-02T16:53:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.