Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses

• URL: http://arxiv.org/abs/2604.25850v3
• Date: Thu, 30 Apr 2026 17:34:17 GMT
• Title: Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses
• Abstract summary: Agentic Harness Engineering (AHE) is a closed loop that automates harness engineering.<n>AHE addresses challenges through three matched observability pillars.<n>Ten AHE lift pass@1 on Terminal-Bench 2 from 69.7% to 77.0%, surpassing the human-designed harness Codex-CLI.<n>On SWE-bench-verified it tops aggregate success at 12% fewer tokens than the seed, and on Terminal-Bench 2 it yields +5.1 to +10.1pp cross-family gains.
• Score: 57.20181537213498
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Harnesses are now central to coding-agent performance, mediating how models interact with tools and execution environments. Yet harness engineering remains a manual craft, because automating it faces a heterogeneous action space across editable components, voluminous trajectories that bury actionable signal, and edits whose effect is hard to attribute. We introduce Agentic Harness Engineering (AHE), a closed loop that addresses these challenges through three matched observability pillars: (1) component observability gives every editable harness component a file-level representation so the action space is explicit and revertible; (2) experience observability distills millions of raw trajectory tokens into a layered, drill-down evidence corpus that an evolving agent can actually consume; and (3) decision observability pairs every edit with a self-declared prediction, later verified against the next round's task-level outcomes. Together, these pillars turn every edit into a falsifiable contract, so harness evolution proceeds autonomously without collapsing into trial-and-error. Empirically, ten AHE iterations lift pass@1 on Terminal-Bench 2 from 69.7% to 77.0%, surpassing the human-designed harness Codex-CLI (71.9%) and the self-evolving baselines ACE and TF-GRPO. The frozen harness transfers without re-evolution: on SWE-bench-verified it tops aggregate success at 12% fewer tokens than the seed, and on Terminal-Bench 2 it yields +5.1 to +10.1pp cross-family gains across three alternate model families, indicating the evolved components encode general engineering experience rather than benchmark-specific tuning. Ablations localize the gain to tools, middleware, and long-term memory rather than the system prompt, suggesting factual harness structure transfers while prose-level strategy does not.

Related papers

• Apodex Discovery: Reality Benchmarks and Environments for Evaluating and Building Discoverative Artificial Intelligence [86.61561123749011]
  We introduce Apodex Discovery, a framework for building and evaluating discoverative AI through the heavy-duty solver.<n>First, a problem-scouting process surveyed 561 industries across 16 sectors, assembled 423 high-value real-world problems, and selected 20 for the initial release.<n>Second, a common environment-task-episode abstraction provides data, tools, constraints, feedback, trajectory recording, and verification of intermediate artifacts and final submissions.
  arXiv  Detail & Related papers  (2026-08-11T18:48:40Z)
• One Recipe, Many Harnesses: What Self-Evolution Encodes Across Languages and Models [46.83445434865936]
  Self-evolving harnesses are closed-loop systems in which an agent inspects its own rollouts and edits its prompts, tools, and memory.<n>It remains unclear whether they encode benchmark-specific adaptations, language-specific engineering knowledge, or compensation for limitations of the underlying model.
  arXiv  Detail & Related papers  (2026-08-10T19:45:45Z)
• HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnesses [17.229636050055266]
  Harness design plays a critical role in agent performance by shaping how large language models (LLMs) perceive, reason over, and act within executable environments.<n>Recent work has proposed automatic harness evolution, which iteratively improves the harness from agent--environment interactions.<n>We propose Harness, a novel automatic harness evolution framework built around constrained evolution, proactive feedback, and component-wise optimization.
  arXiv  Detail & Related papers  (2026-08-03T08:51:50Z)
• DarwinX: Evolving Agent Harnesses Through Natural Selection [48.64258219266629]
  DarwinX treats self-evolution as selection over a population of harnesses with the model frozen.<n>General agent competence, not benchmark-specific patches, so it survives changes of task, verifier, and base model.
  arXiv  Detail & Related papers  (2026-07-31T05:34:02Z)
• Living-Harness Is an Interactive-Agent Evolver [15.081919387954578]
  Large language model (LLM) agents may recover from a failure within an episode or after a retry, yet the same execution failure can recur in later tasks.<n>We propose $textbfLiving-Harness, a self-evolving agent harness that converts each completed trajectory and its evaluator signals into posterior evidence for bounded harness updates.
  arXiv  Detail & Related papers  (2026-07-29T08:20:11Z)
• Beyond the Attention Stability Boundary: Agentic Self-Synthesizing Reasoning Protocols [6.357772907811544]
  Self- Synthesizing Reasoning Protocols (SSRP) is a metacognitive framework that implements a discrete separation between architectural planning and procedural execution.<n>We present 3 experimental tiers: a shallow recency-based retrieval pilot, a high-entropy SOP, and a semantic hijacked 3-hop Multi-Fact Synthesis task.<n>Our results empirically locate the Attention Stability Boundary, where stateless Vanilla ReAct baselines for GPT 5.4 collapse to 0.1% success while SSRP achieves a 715X Resilience Lift.
  arXiv  Detail & Related papers  (2026-04-27T14:13:30Z)
• AnomalyAgent: Agentic Industrial Anomaly Synthesis via Tool-Augmented Reinforcement Learning [7.129476412364631]
  We propose AnomalyAgent, an anomaly synthesis agent with self-reflection, knowledge retrieval, and iterative refinement capabilities.<n>Specifically, AnomalyAgent is equipped with five tools: Prompt Generation (PG), Image Generation (IG), Quality Evaluation (QE), Knowledge Retrieval (KR), and Mask Generation (MG)<n>On the MVTec-AD dataset, AnomalyAgent synthesis achieves IS/IC-L of 2.0.33 for anomaly generation, 57.0% classification accuracy using ResNet34, and 99.3%/74.2% AP at the image/pixel level using a simple UNet.
  arXiv  Detail & Related papers  (2026-04-09T07:17:00Z)
• Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents [66.97968363332465]
  We introduce Claw-Eval, an end-to-end evaluation suite addressing all three gaps in agent benchmarks.<n>Claw-Eval comprises 300 human-verified tasks spanning 9 categories across three groups.<n>Every agent action is recorded through three independent evidence channels.
  arXiv  Detail & Related papers  (2026-04-07T17:43:18Z)
• CORAL: Towards Autonomous Multi-Agent Evolution for Open-Ended Discovery [67.47220507982522]
  Large language model (LLM)-based evolution is a promising approach for openended discovery.<n>We present CORAL, the first framework for autonomous multi-agent evolution on open-ended problems.<n>CORAL replaces rigid control with long-running agents that explore, reflect, and collaborate.
  arXiv  Detail & Related papers  (2026-04-02T05:59:06Z)
• TRAJEVAL: Decomposing Code Agent Trajectories for Fine-Grained Diagnosis [23.834704102474927]
  Code agents can resolve GitHub issues, yet when they fail, current evaluation provides no visibility into where or why.<n>We introduce TRAJEVAL, a diagnostic framework that decomposes agent trajectories into three interpretable stages.<n>We validate that these diagnostics are predictive, achieving model-level Pass@1 prediction within 0.87-2.1% MAE.
  arXiv  Detail & Related papers  (2026-03-25T05:27:03Z)
• AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents [50.481033105867205]
  We introduce AgentProcessBench, the first benchmark dedicated to evaluating step-level effectiveness in realistic, tool-augmented trajectories.<n>The benchmark comprises 1,000 diverse trajectories and 8,509 human-labeled step annotations with 89.1% inter-annotator agreement.<n>It features a ternary labeling scheme to capture exploration and an error propagation rule to reduce labeling ambiguity.
  arXiv  Detail & Related papers  (2026-03-15T16:13:58Z)
• Beyond Quantity: Trajectory Diversity Scaling for Code Agents [51.71414642763219]
  Trajectory Diversity Scaling is a data synthesis framework for code agents that scales performance through diversity rather than raw volume.<n> TDScaling integrates four innovations: (1) a Business Cluster mechanism that captures real-service logical dependencies; (2) a blueprint-driven multi-agent paradigm that enforces trajectory coherence; and (3) an adaptive evolution mechanism that steers toward long-tail scenarios.
  arXiv  Detail & Related papers  (2026-02-03T07:43:03Z)
• daVinci-Agency: Unlocking Long-Horizon Agency Data-Efficiently [35.39097522391409]
  Large Language Models (LLMs) excel at short-term tasks, scaling them to long-horizon agentic synthesis remains challenging.<n>We propose daVinci-Agency, which systematically mines structured supervision from chain-of-PRs.<n>DaVinci-Agency's PR-grounded structure inherently preserves the causal dependencies and iterative refinements essential for teaching persistent goal-cycle behavior.
  arXiv  Detail & Related papers  (2026-02-02T13:23:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.