AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces
Abstract Overview
This paper presents AutoSaddler, a framework for automatically improving the external harness of LLM agents on long-horizon tasks. It formulates harness optimization as an offline mini-batch learning problem over execution traces, where failures are diagnosed and converted into candidate updates. The method combines evidence-grounded diagnosis, structured patch generation over prompts, tools, and middleware, and validation-based selection with an EvoDAG history of prior updates. The goal is to produce durable harness changes that generalize beyond single trajectories rather than one-off repairs.
Novelty
The distinctive contribution is treating harness optimization as an offline learning process driven by failure traces rather than relying on manual tuning or prompt-only optimization. AutoSaddler introduces a structured intervention space spanning prompt, tool, and middleware layers with phased patch scheduling, alongside an EvoDAG-based evolution mechanism that guides candidate proposals using reflection history.
Results
Across GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0, AutoSaddler improves over the corresponding base harnesses, reaching 62.0%, 46.9%, and 50.0% Pass@1, respectively, and outperforms the strongest automated baselines on each benchmark. On GAIA2, it reaches 72.3% development accuracy with approximately 1,000 task executions, whereas GEPA and Meta-Harness saturate at 64.6% and 61.5% despite consuming roughly 2,800 executions. Ablation experiments confirm that in-depth diagnosis, structured intervention, and generalization-aware selection are all essential for preventing regressions and sustaining performance gains.
Key Points
- AutoSaddler optimizes agent harnesses through an offline mini-batch learning loop that diagnoses failure traces and generates targeted code-level patches.
- The framework organizes modifications into structured prompt, tool, and middleware categories with phased scheduling, and tracks update history using an EvoDAG memory structure.
- Empirical evaluations demonstrate substantial gains in Pass@1 across three benchmarks along with higher optimization efficiency and lower regression rates on unseen tasks.