AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces

• URL: http://arxiv.org/abs/2608.23041v1
• Date: Mon, 24 Aug 2026 09:45:08 GMT
• Title: AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces
• Abstract summary: AutoSaddler is a framework that formulates harness improvement as an offline learning problem.<n>It iteratively updates the harness using failure signals from mini-batches.<n>Experiments on GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0 show that AutoSaddler substantially improves agent performance.
• Score: 61.51487329124135
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM agents remain unreliable on long-horizon tasks, where small local failures can compound over extended interactions and lead to overall task failure. Although external harnesses can substantially improve robustness, harness design remains a manual and expensive process that requires searching over a large space of prompts, tool configurations, and control logic. We propose AutoSaddler, an automatic harness optimization framework that formulates harness improvement as an offline learning problem and iteratively updates the harness using failure signals from mini-batches. AutoSaddler combines failure-trace diagnosis, structured patch generation that treats the harness as code, and validation-based update selection. Experiments on GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0 show that AutoSaddler substantially improves agent performance over the corresponding base harnesses, achieving gains of 9.0, 9.6, and 10.0 percentage points, respectively. Ablation studies further suggest that effective harness optimization benefits from three ingredients: deep debugging rather than shallow reflection, targeted modifications rather than unconstrained editing, and generalization-aware selection rather than trajectory-specific repair. Together, these results suggest that automatic harness optimization is a promising path toward more performant and reliable agent systems.

Related papers

• HarnessOpt-Bench: Evaluating LLMs at Harness Optimization [3.846854817984248]
  We introduce a benchmark for end-to-end harness optimization under expensive evaluation regimes.<n>A trusted execution environment enforces the evaluation boundary, target-agent resource use, and preserves candidate versions for audit.<n>Experiment results show that models separate more than the coding harnesses they act through, native harnesses are not consistently superior, and gains vary substantially across tasks.
  arXiv  Detail & Related papers  (2026-08-06T17:21:05Z)
• Co-Harness: Co-Evolving Harnesses and Model Weights for LLM Agents [37.91666123749406]
  Co-Harness is a framework that jointly optimize the agent harness and model parameters during post-training.<n>An autonomous case study shows that Co-Harness can recover from system crashes, improve inference efficiency, and discover ensemble strategies without human intervention.
  arXiv  Detail & Related papers  (2026-07-17T02:39:57Z)
• Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0 [53.71882250666667]
  Most reported gains from agent-optimization methods are one-shot.<n>This does not test the setting that matters for deployed agents.<n>We compare three approaches to agent-harness optimization.
  arXiv  Detail & Related papers  (2026-07-15T16:36:04Z)
• Evolving Agents in the Dark: Retrospective Harness Optimization via Self-Preference [51.55077364626896]
  We introduce Retrospective Harness Optimization (RHO), a self-supervised method that optimize agent harness using only past trajectories.<n>RHO selects a diverse coreset of challenging tasks from past trajectories and re-solves them in parallel.<n>A single optimization round improves the pass rate on SWE-Bench Pro from 59% to 78% without any external grading.
  arXiv  Detail & Related papers  (2026-06-04T09:26:00Z)
• Adaptive Auto-Harness: Sustained Self-Improvement for Agentic System Deployment on Open-Ended Task Streams [41.672123164003814]
  We introduce Adaptive Auto-Harness, a framework and system for such streams.<n>It decomposes the gap to an oracle harness into evolution loss and adaptation loss.<n>It outperforms five existing auto-harness baselines across prediction-market, security-competition, and event-forecasting streams.
  arXiv  Detail & Related papers  (2026-06-01T06:51:14Z)
• AdaExplore: Failure-Driven Adaptation and Diversity-Preserving Search for Efficient Kernel Generation [59.964989458924585]
  AdaExplore is an agent framework that enables self-improvement via accumulated execution feedback for performance-critical kernel code generation.<n>In the adaptation stage, the agent synthesizes tasks and converts recurring failures into a reusable memory of validity rules.<n>In the search stage, the agent organizes candidate kernels as a tree and alternates between small local refinements and larger structural regeneration.
  arXiv  Detail & Related papers  (2026-04-17T18:25:03Z)
• Meta-Harness: End-to-End Optimization of Model Harnesses [52.31507076660471]
  We introduce Meta-Harness, an outer-loop system that searches over harness code for large language model (LLM) applications.<n>On online text classification, Meta-Harness improves over a state-of-the-art context management system by 7.7 points while using 4x fewer context tokens.<n>On retrieval-augmented math reasoning, a single discovered harness improves accuracy on 200 IMO-level problems by 4.7 points on average across five held-out models.
  arXiv  Detail & Related papers  (2026-03-30T05:33:50Z)
• Rethinking the Role of Entropy in Optimizing Tool-Use Behaviors for Large Language Model Agents [54.18201810286764]
  Tool-using agents based on Large Language Models (LLMs) excel in tasks such as mathematical reasoning and multi-hop question answering.<n>In long trajectories, agents often trigger excessive and low-quality tool calls, increasing latency and degrading inference performance.<n>We propose using entropy reduction as a supervisory signal and design two reward strategies to address the differing needs of optimizing tool-use behavior.
  arXiv  Detail & Related papers  (2026-02-02T12:52:14Z)
• ROAD: Reflective Optimization via Automated Debugging for Zero-Shot Agent Alignment [1.6968020497268546]
  ROAD is a novel framework that treats optimization as a dynamic debug investigation rather than a search.<n>Road is highly sample-efficient, achieving a 5.6 percent increase in success rate and a 3.8 percent increase in search accuracy.<n>These findings suggest that mimicking the human engineering loop of failure analysis and patching offers a viable, data-efficient alternative to resource-intensive training.
  arXiv  Detail & Related papers  (2025-12-30T07:31:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.