Verify Smarter, Evolve Further: Efficient Harness Evolution through Behavior-Aware Verification

• URL: http://arxiv.org/abs/2608.27311v1
• Date: Thu, 27 Aug 2026 16:12:23 GMT
• Title: Verify Smarter, Evolve Further: Efficient Harness Evolution through Behavior-Aware Verification
• Abstract summary: Existing propose-and-verify methods score every candidate on a fixed task set.<n>We introduce HarnessLens, a budget-aware framework for automated harness evolution.
• Score: 30.941867057615223
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Agent harnesses shape how language-model agents use instructions, tools, and runtime components, but adapting these harnesses requires costly verification. Existing propose-and-verify methods typically score every candidate on a fixed task set, wasting rollouts on unrelated behaviors and allowing aggregate scores to obscure specific regressions. We introduce HarnessLens, a budget-aware framework for automated harness evolution. HarnessLens jointly explores the task space and user-configurable components, derives candidate modifications from execution trajectories, and selectively verifies each candidate on behavior-relevant tasks using an attributable-evidence gate. Across three agent harnesses and four benchmarks, HarnessLens improves average held-out performance by 7.6-13.6% while consuming substantially less evaluation budget than competing baselines. These results demonstrate that behavior-aware verification with explicit attribution enables more reliable and sample-efficient harness evolution under constrained interaction budgets. Our code is available at https://github.com/jhxu5214/HarnessLens.

Related papers

• COBRA-Skills: Contextual Bandit-Guided Evolution for Agent Skill Optimization [55.60171725587438]
  Large language model (LLM) agents can benefit from reusable skills distilled from prior task experience.<n>Existing skill optimization methods often rely on costly execution-based evaluation and substantial task data.<n>We introduce textbfCOBRA-Skills, an efficient framework that formulates skill optimization as budgeted sequential optimization over a dynamically evolving candidate space.
  arXiv  Detail & Related papers  (2026-09-10T15:12:24Z)
• HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness? [61.02699867449589]
  We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to run infrastructure.<n>In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system.<n>In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance.
  arXiv  Detail & Related papers  (2026-09-01T15:45:33Z)
• HarnessEvolve: Learning from Reference Trajectories for Reliable Agent Self-Evolution [11.431479238875712]
  We introduce HarnessEvolve, a self-evolving framework that learns from reference trajectories to achieve reliable agent self-evolution.<n>We conduct extensive experiments on several benchmarks spanning open-domain and enterprise scenarios, using different models and agent frameworks.<n>Results demonstrate that HarnessEvolve consistently outperforms state-of-the-art baselines across all benchmarks and settings.
  arXiv  Detail & Related papers  (2026-09-01T07:31:18Z)
• Task-CoEvolve: Efficient Harness Optimization via Adaptive Validation Task Selection [51.39973047614183]
  We present a novel approach to efficient LLM harness optimization through adaptive validation task selection.<n>$textbfTask-CoEvolve$, which co-evolves the validation tasks with the harness by addressing two challenges: selecting informative tasks and estimating full-set performance from partial evaluations.
  arXiv  Detail & Related papers  (2026-08-20T15:24:54Z)
• $A^2E$ : An End-to-End Agent Auditing Engine [25.281617098281277]
  $A2E$ is an end-to-end evaluation engine designed for agent harnesses.<n>It captures and generates standardized execution traces during experiments.<n>$A2E$ assesses harness capabilities using a suite of multidimensional metrics.
  arXiv  Detail & Related papers  (2026-08-07T15:44:12Z)
• MemoHarness: Agent Harnesses That Learn from Experience [46.167787606315585]
  We introduce MemoHarness, an adaptive harness optimization framework that learns from its own executions.<n>MemoHarness decomposes the harness into six editable control dimensions, stores per-case diagnoses and distilled global patterns in a dual-layer experience bank.<n>In our evaluation across shell-agent, code-generation, and analytical-reasoning benchmarks, MemoHarness improves over the fixed harnesses we compare against and shows selective transfer to unseen suites and base models.
  arXiv  Detail & Related papers  (2026-07-14T21:22:18Z)
• Rethinking the Evaluation of Harness Evolution for Agents [83.07258924910069]
  Existing harness evolution methods use unit test cases to search for harness configurations and then report final performance on the same public benchmark.<n>This protocol raises two fundamental concerns.<n>As in agentic test-time scaling, it should therefore be compared with simple task-level search baselines under matched feedback and inference budgets.
  arXiv  Detail & Related papers  (2026-07-14T00:18:42Z)
• TTHE: Test-Time Harness Evolution [50.26245555541721]
  Existing approaches optimize such harnesses before deployment, searching training or development data for a fixed agent workflow that is then frozen at test time.<n>We ask whether the harness can instead be optimized during evaluation itself, using only the unlabeled execution traces the agent produces on the test inputs.<n>During evaluation, TTHE maintains a population of candidate harnesses and refines them through an agentic proposer that reasons over their execution traces.<n>A judge then commits an improved harness from execution-derived proxy signals, and the selected program persists to govern subsequent inputs.
  arXiv  Detail & Related papers  (2026-07-09T05:53:39Z)
• TAPO: Tool-Aware Policy Optimization via Credit Transfer for Multimodal Search Agents [54.08846865906602]
  We characterize credit misassignment as a systematic failure mode of GRPO in tool-augmented multimodal search agents.<n>We propose Tool-Aware Policy Optimization (TAPO), which exploits the parameter-determinism property of information-acquisition tools.
  arXiv  Detail & Related papers  (2026-06-04T07:15:43Z)
• Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses [57.20181537213498]
  Agentic Harness Engineering (AHE) is a closed loop that automates harness engineering.<n>AHE addresses challenges through three matched observability pillars.<n>Ten AHE lift pass@1 on Terminal-Bench 2 from 69.7% to 77.0%, surpassing the human-designed harness Codex-CLI.<n>On SWE-bench-verified it tops aggregate success at 12% fewer tokens than the seed, and on Terminal-Bench 2 it yields +5.1 to +10.1pp cross-family gains.
  arXiv  Detail & Related papers  (2026-04-28T16:55:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.