One Recipe, Many Harnesses: What Self-Evolution Encodes Across Languages and Models

• URL: http://arxiv.org/abs/2608.10178v1
• Date: Mon, 10 Aug 2026 19:45:45 GMT
• Title: One Recipe, Many Harnesses: What Self-Evolution Encodes Across Languages and Models
• Abstract summary: Self-evolving harnesses are closed-loop systems in which an agent inspects its own rollouts and edits its prompts, tools, and memory.<n>It remains unclear whether they encode benchmark-specific adaptations, language-specific engineering knowledge, or compensation for limitations of the underlying model.
• Score: 46.83445434865936
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Self-evolving harnesses are closed-loop systems in which an agent inspects its own rollouts and edits its prompts, tools, and memory. They reliably improve coding agents in evaluations, but prior work reports aggregate gains rather than analyzing what the evolved artifacts encode. It therefore remains unclear whether they encode benchmark-specific adaptations, language-specific engineering knowledge, or compensation for limitations of the underlying model. We disentangle these factors by holding an evolution recipe fixed across a grid of eight programming languages (Multi-SWE-Bench) and three base models, and analyzing the resulting harnesses. The recipe routes every edit through a typed failure signal and records it as a falsifiable contract, making each modification attributable after evolution. Four findings emerge. (1)The loop improves held-out solve rates over both a minimal seed and the manually designed mini-SWE-agent scaffold in most cells, but with two null regions. (2)Gains compensate recoverable execution defects, where defect mass is near zero, and gain is near zero; which defect dominates is cell-specific. A harness closes the gap between what a policy can do and what it does. (3)Evolved harnesses share an abstract playbook across languages but instantiate it with almost disjoint language ecosystem machinery. (4)The shared core transfers and can be distilled into one universal harness, while an ecosystem margin resists both and requires native re-evolution. Together, these results recast the evolved harness as a legible compensation layer, shaped jointly by the language's engineering demands and the model's behavioral gaps, rather than an opaque benchmark-tuned scaffold.

Related papers

• SynthSentry: Detecting Synthetic Data Contamination in Language Model Training Data [2.998247765202193]
  We introduce a corpus-level, model-agnostic contamination signal requiring no access to the generating model, no generation history, and no synthetic labels.<n>We evaluate on corpora contaminated by small open-weight generators and an instruction-tuned open-weight model under a leave-one-generator-out protocol.<n>The score ranks corpora by severity with little loss when whole generator families are held out.
  arXiv  Detail & Related papers  (2026-09-11T02:25:18Z)
• HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness? [61.02699867449589]
  We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to run infrastructure.<n>In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system.<n>In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance.
  arXiv  Detail & Related papers  (2026-09-01T15:45:33Z)
• Fantastic Adaptive Taxonomies and How to Use Them [100.20614173157708]
  AdaMAST converts execution traces into a compact, evidence-grounded failure taxonomy.<n>No code is hand-authored, no trace human-annotated.<n>In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form.
  arXiv  Detail & Related papers  (2026-07-17T17:41:16Z)
• Self-Evolving Agent Harnesses via Gated Semantic Quality-Diversity [10.3661763716032]
  In deployment the harness is often the only lever available, so improving it automatically is the natural way to raise performance without touching the weights.<n>We present a self-evolving agent-harness framework that separates proposing changes from crediting them.
  arXiv  Detail & Related papers  (2026-07-15T10:26:26Z)
• Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents [82.27610290890475]
  We analyze two harness self-evolution capabilities: (i) harness-updating, the capability to produce useful persistent harness updates from execution evidence; (ii) harness-benefit, the capability to benefit from updated harnesses during task solving.<n>Our analysis reveals two findings. First, harness-updating is flat in base capability: models from different capability tiers produce harness updates that lead to surprisingly similar gains.<n>Second, harness-benefit is non-monotonic in base capability: weak-tier models benefit little from updated harnesses, mid-tier models benefit most, and strong-tier models benefit less than mid-tier.
  arXiv  Detail & Related papers  (2026-05-28T22:16:14Z)
• Learning to Build the Environment: Self-Evolving Reasoning RL via Verifiable Environment Synthesis [37.03537014865641]
  We pursue a vision for self-improving language models in which the model does not merely generate problems or traces to imitate, but constructs the environments that train it.<n>We instantiate this view in EvoEnv, a single-policy generator, solver method that synthesizes Python environments from ten seeds.
  arXiv  Detail & Related papers  (2026-05-14T05:14:45Z)
• Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses [57.20181537213498]
  Agentic Harness Engineering (AHE) is a closed loop that automates harness engineering.<n>AHE addresses challenges through three matched observability pillars.<n>Ten AHE lift pass@1 on Terminal-Bench 2 from 69.7% to 77.0%, surpassing the human-designed harness Codex-CLI.<n>On SWE-bench-verified it tops aggregate success at 12% fewer tokens than the seed, and on Terminal-Bench 2 it yields +5.1 to +10.1pp cross-family gains.
  arXiv  Detail & Related papers  (2026-04-28T16:55:02Z)
• CliffSearch: Structured Agentic Co-Evolution over Theory and Code for Scientific Algorithm Discovery [8.600891931062277]
  We present CliffSearch, an agentic evolutionary framework in which the core evolution operators are implemented as LLM agents.<n>We illustrate the framework on three benchmark-grounded studies: transformer hyper-connection evolution, discovery on a fixed nanoGPT stack, and a smaller native-optimizer ablation.
  arXiv  Detail & Related papers  (2026-04-01T17:51:26Z)
• Confident Adaptive Language Modeling [95.45272377648773]
  CALM is a framework for dynamically allocating different amounts of compute per input and generation timestep. We demonstrate the efficacy of our framework in reducing compute -- potential speedup of up to $times 3$ -- while provably maintaining high performance.
  arXiv  Detail & Related papers  (2022-07-14T17:00:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.