HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

• URL: http://arxiv.org/abs/2609.01437v1
• Date: Tue, 01 Sep 2026 15:45:33 GMT
• Title: HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
• Abstract summary: We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to run infrastructure.<n>In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system.<n>In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance.
• Score: 61.02699867449589
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly termed the agent harness. Changing this harness while holding model weights fixed can substantially alter task performance. Current agent evaluations typically report downstream performance under a chosen harness, leaving a model's ability to develop the harness itself comparatively underexplored. We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to runnable infrastructure. HarnessDev covers two stages. In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system. In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance. We then evaluate each constructed harness on capability (task success on held-out benchmarks) and efficiency (execution-token cost). The reported Creation results cover six creator LLMs, four domains, and five downstream benchmarks totaling 2,207 unique downstream instances, with hidden evaluation tasks withheld from development. We find that generated harnesses remain substantially behind mature human-engineered references on code and on search and research, while matching or exceeding the selected references on writing and machine-learning experimentation, with large variation in execution cost. Evolution produces some performance gains, but they are unstable and transfer only partially to held-out tasks. Experiments with a fixed runtime model further show that the gains depend strongly on the model executing the harness, indicating limited transfer across models.

Related papers

• HELIX: Model-Harness Co-evolution for Recursive Self-Improvement [8.484282837156933]
  We present HELIX, a source-traceable substrate for harness evolution.<n>We show how harness evolution expands current capability and creates learning signal for the next model.
  arXiv  Detail & Related papers  (2026-08-14T04:44:53Z)
• $A^2E$ : An End-to-End Agent Auditing Engine [25.281617098281277]
  $A2E$ is an end-to-end evaluation engine designed for agent harnesses.<n>It captures and generates standardized execution traces during experiments.<n>$A2E$ assesses harness capabilities using a suite of multidimensional metrics.
  arXiv  Detail & Related papers  (2026-08-07T15:44:12Z)
• EvolveNet: Collaborative Harness Evolution for Agent Self-Improvement [70.68218215070745]
  Existing approaches assume that all execution experience can be routed to a single harness.<n>We introduce EvolveNet, a paradigm of harness evolution that moves experience extraction to the data.
  arXiv  Detail & Related papers  (2026-08-05T15:37:18Z)
• Rethinking the Evaluation of Harness Evolution for Agents [83.07258924910069]
  Existing harness evolution methods use unit test cases to search for harness configurations and then report final performance on the same public benchmark.<n>This protocol raises two fundamental concerns.<n>As in agentic test-time scaling, it should therefore be compared with simple task-level search baselines under matched feedback and inference budgets.
  arXiv  Detail & Related papers  (2026-07-14T00:18:42Z)
• TTHE: Test-Time Harness Evolution [50.26245555541721]
  Existing approaches optimize such harnesses before deployment, searching training or development data for a fixed agent workflow that is then frozen at test time.<n>We ask whether the harness can instead be optimized during evaluation itself, using only the unlabeled execution traces the agent produces on the test inputs.<n>During evaluation, TTHE maintains a population of candidate harnesses and refines them through an agentic proposer that reasons over their execution traces.<n>A judge then commits an improved harness from execution-derived proxy signals, and the selected program persists to govern subsequent inputs.
  arXiv  Detail & Related papers  (2026-07-09T05:53:39Z)
• From Question Answering to Task Completion: A Survey on Agent System and Harness Design [67.8241530947325]
  This survey examines LLM-based agents through a model-harness lens.<n>We first clarify the functional definition of agents and the implementation view of an LLM-based agent as a foundation model and an execution harness.<n>We then analyze the limits of model-centric scaling, trace four paradigms of agent engineering, and decompose the execution harness into six coupled runtime responsibilities.
  arXiv  Detail & Related papers  (2026-06-14T05:40:16Z)
• HarnessX: A Composable, Adaptive, and Evolvable Agent Harness Foundry [35.87794858139959]
  HarnessX is a foundry for composable, adaptive, and evolvable agent harnesses.<n>It assembles typed harness primitives via a substitution algebra, adapts them through AEGIS.<n>It closes the harness-model loop by turning trajectories into both harness updates and model training signal.
  arXiv  Detail & Related papers  (2026-06-12T08:27:11Z)
• Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents [82.27610290890475]
  We analyze two harness self-evolution capabilities: (i) harness-updating, the capability to produce useful persistent harness updates from execution evidence; (ii) harness-benefit, the capability to benefit from updated harnesses during task solving.<n>Our analysis reveals two findings. First, harness-updating is flat in base capability: models from different capability tiers produce harness updates that lead to surprisingly similar gains.<n>Second, harness-benefit is non-monotonic in base capability: weak-tier models benefit little from updated harnesses, mid-tier models benefit most, and strong-tier models benefit less than mid-tier.
  arXiv  Detail & Related papers  (2026-05-28T22:16:14Z)
• DemoEvolve: Overcoming Sparse Feedback in Agentic Harness Evolution with Demonstrations [12.048713464721665]
  DemoEvolve is a demonstration-bootstrapped approach to harness evolution.<n>We study this paradigm as a form of sample-efficient fast adaptation.<n>We show that DemoEvolve produces more effective and auditable harness edits.
  arXiv  Detail & Related papers  (2026-05-23T12:10:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.