Self-Evolving Agent Harnesses via Gated Semantic Quality-Diversity

• URL: http://arxiv.org/abs/2607.13683v1
• Date: Wed, 15 Jul 2026 10:26:26 GMT
• Title: Self-Evolving Agent Harnesses via Gated Semantic Quality-Diversity
• Abstract summary: In deployment the harness is often the only lever available, so improving it automatically is the natural way to raise performance without touching the weights.<n>We present a self-evolving agent-harness framework that separates proposing changes from crediting them.
• Score: 10.3661763716032
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: An LLM agent's real-task performance is shaped as much by the harness around its model as by the frozen model itself: its prompts, injected knowledge, runtime control, and configuration. In deployment the harness is often the only lever available, so improving it automatically is the natural way to raise performance without touching the weights. The hard part is not generating changes but knowing which one truly helped. Self-generated feedback is noisy, and an apparent gain can be a measurement artifact or an edit that merely overfits the tasks it was tuned on. We present a self-evolving agent-harness framework that separates proposing changes from crediting them: a language model diagnoses failures and proposes patches, while all sampling, measurement, and significance testing are owned by deterministic code, so every credited improvement is trustworthy by construction. Patches populate a gated, categorical quality-diversity archive (GSME) keyed on the (WHERE x WHY) pathology an edit addresses rather than the tasks it fixes, an anti-overfitting inductive bias; generalization is measured on a sealed test scored only after evolution. Across seven domains with a frozen open-weight model, the harness is train-selected and scored once on a sealed test; its credited gains there are +9 to +15.5pp and retain 86-147% of the training gain, evidence they generalize rather than overfit. The winning patch tracks the model's dominant pathology, not its size or family: changing the model can change the pathology and the patch, while the same pathology-to-patch match recurs across two model families. What transfers is the diagnose-and-credit loop, not any specific harness.

Related papers

• Rewarding Reasoning, Not Answers: Fixing and Bounding Test-Time Reinforcement Learning on Medical QA [40.18486858532692]
  Test-time reinforcement learning adapts a model on its own unlabeled test set using majority-vote pseudo-labels.<n>We show that this recipe collapses on medical multiple-choice QA: accuracy stagnates while output diversity rapidly declines.<n>This diagnosis motivates PROSE, Process Reward Guided Self-Training, which rewards reasoning quality instead of answer agreement.
  arXiv  Detail & Related papers  (2026-09-15T05:31:55Z)
• Co-Evolving Harnesses and Models: On-Policy Correction Helps Weaker Models Catch Up Where Imitation Fails [23.287109132562403]
  Automated harness evolution can enable smaller models to perform well on domain-specific tasks at a fraction of frontier-model cost.<n>We show that imitation transfers knowledge and increases scaffold usage, but disrupts model-harness fit.<n>We develop an on-policy expert-correction pipeline, automated by a meta-level MLE agent, that localizes the failing turn in the weaker model's own rollout and asks the expert to rewrite only that turn.
  arXiv  Detail & Related papers  (2026-09-08T17:53:49Z)
• HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness? [61.02699867449589]
  We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to run infrastructure.<n>In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system.<n>In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance.
  arXiv  Detail & Related papers  (2026-09-01T15:45:33Z)
• One Recipe, Many Harnesses: What Self-Evolution Encodes Across Languages and Models [46.83445434865936]
  Self-evolving harnesses are closed-loop systems in which an agent inspects its own rollouts and edits its prompts, tools, and memory.<n>It remains unclear whether they encode benchmark-specific adaptations, language-specific engineering knowledge, or compensation for limitations of the underlying model.
  arXiv  Detail & Related papers  (2026-08-10T19:45:45Z)
• DarwinX: Evolving Agent Harnesses Through Natural Selection [48.64258219266629]
  DarwinX treats self-evolution as selection over a population of harnesses with the model frozen.<n>General agent competence, not benchmark-specific patches, so it survives changes of task, verifier, and base model.
  arXiv  Detail & Related papers  (2026-07-31T05:34:02Z)
• TTHE: Test-Time Harness Evolution [50.26245555541721]
  Existing approaches optimize such harnesses before deployment, searching training or development data for a fixed agent workflow that is then frozen at test time.<n>We ask whether the harness can instead be optimized during evaluation itself, using only the unlabeled execution traces the agent produces on the test inputs.<n>During evaluation, TTHE maintains a population of candidate harnesses and refines them through an agentic proposer that reasons over their execution traces.<n>A judge then commits an improved harness from execution-derived proxy signals, and the selected program persists to govern subsequent inputs.
  arXiv  Detail & Related papers  (2026-07-09T05:53:39Z)
• From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws [12.757205456771295]
  This paper proposes HarnessFix, a trace-guided framework for diagnosing agent failures and repairing agent harnesses.<n> HarnessFix compiles raw execution traces and harness code into a Harness-aware Trace Intermediate Representation.<n>It attributes failures to responsible trajectory steps and harness layers, consolidates recurring diagnoses into actionable flaw records, and maps them to scoped repair operators.
  arXiv  Detail & Related papers  (2026-06-04T15:58:30Z)
• Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents [82.27610290890475]
  We analyze two harness self-evolution capabilities: (i) harness-updating, the capability to produce useful persistent harness updates from execution evidence; (ii) harness-benefit, the capability to benefit from updated harnesses during task solving.<n>Our analysis reveals two findings. First, harness-updating is flat in base capability: models from different capability tiers produce harness updates that lead to surprisingly similar gains.<n>Second, harness-benefit is non-monotonic in base capability: weak-tier models benefit little from updated harnesses, mid-tier models benefit most, and strong-tier models benefit less than mid-tier.
  arXiv  Detail & Related papers  (2026-05-28T22:16:14Z)
• DemoEvolve: Overcoming Sparse Feedback in Agentic Harness Evolution with Demonstrations [12.048713464721665]
  DemoEvolve is a demonstration-bootstrapped approach to harness evolution.<n>We study this paradigm as a form of sample-efficient fast adaptation.<n>We show that DemoEvolve produces more effective and auditable harness edits.
  arXiv  Detail & Related papers  (2026-05-23T12:10:52Z)
• Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses [57.20181537213498]
  Agentic Harness Engineering (AHE) is a closed loop that automates harness engineering.<n>AHE addresses challenges through three matched observability pillars.<n>Ten AHE lift pass@1 on Terminal-Bench 2 from 69.7% to 77.0%, surpassing the human-designed harness Codex-CLI.<n>On SWE-bench-verified it tops aggregate success at 12% fewer tokens than the seed, and on Terminal-Bench 2 it yields +5.1 to +10.1pp cross-family gains.
  arXiv  Detail & Related papers  (2026-04-28T16:55:02Z)
• STABLE: Gated Continual Learning for Large Language Models [0.0]
  STABLE is a gated continual self editing framework that constrains forgetting during sequential updates.<n>Each candidate edit is evaluated against a stability budget using one of three metrics.<n>Experiments on the Qwen-2.5-7B model show that gating effectively mitigates forgetting while preserving adaptability.
  arXiv  Detail & Related papers  (2025-10-17T16:14:05Z)
• RePaCA: Leveraging Reasoning Large Language Models for Static Automated Patch Correctness Assessment [0.0]
  We introduce RePaCA, a novel static APCA technique that leverages Large Language Models (LLMs) specialized in thinking tasks.<n>Our approach achieves state-of-the-art performance, with 83.1% accuracy and an 84.8% F1-score.
  arXiv  Detail & Related papers  (2025-07-30T11:21:09Z)
• Is Self-Repair a Silver Bullet for Code Generation? [68.02601393906083]
  Large language models have shown remarkable aptitude in code generation, but still struggle to perform complex tasks. Self-repair -- in which the model debugs and repairs its own code -- has recently become a popular way to boost performance. We analyze Code Llama, GPT-3.5 and GPT-4's ability to perform self-repair on problems taken from HumanEval and APPS.
  arXiv  Detail & Related papers  (2023-06-16T15:13:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.