EVOHARNESSBENCH: Can Your Agents Keep Pace with an Evolving Harness?

• URL: http://arxiv.org/abs/2609.04280v1
• Date: Thu, 03 Sep 2026 03:24:02 GMT
• Title: EVOHARNESSBENCH: Can Your Agents Keep Pace with an Evolving Harness?
• Abstract summary: EVOHARNESSBENCH is a benchmark for evaluating agents under controlled harness evolution.<n>It contains 17 multi-stage harness streams constructed deterministically from verifier-based benchmarks.
• Score: 83.21711423118768
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Modern LLM-based agents operate through a harness of tools, reusable skills, and specialist agents that shapes what they observe and what they can do. In practice, this harness continually evolves as new capabilities are added. We introduce EVOHARNESSBENCH, a benchmark for evaluating agents under controlled harness evolution across three axes (tools, skills, and agents). Unlike existing continual-learning benchmarks for agents, which typically place non-stationarity (i.e., what changes over time) in the task stream while keeping the harness fixed, EVOHARNESSBENCH places non-stationarity in the externally supplied harness itself. It contains 17 multi-stage harness streams constructed deterministically from verifier-based benchmarks, comprising 802 tasks, 520 tools, 42 skills, and 62 agents. We evaluate two complementary settings corresponding to the central challenges of harness evolution: deployment evaluation, which isolates retention of previously accessible competence as the harness expands, and self-evolving adaptation evaluation, which tests whether accumulated experience remains useful as new capabilities are introduced. Our results reveal three persistent gaps. First, harness expansion alone can degrade performance on previously solved tasks, producing harness-induced forgetting. Second, gains from self-evolving adaptation remain inconsistent across stages of harness evolution, capability axes, and environments. Third, retention and adaptation can pull in different directions: preserving earlier competence does not necessarily improve adaptation to newly introduced capabilities, and vice versa. These results establish harness evolution as a distinct challenge for building agents that can keep pace with an evolving harness while preserving previously effective behavior.

Related papers

• ModularRSI: Modular and Generalizable Recursive Harness Self-Improvement [46.36989489677189]
  We propose a benchmark-disjoint, contrastive, and modular framework for generalizable harness evolution.<n> ModularRSI contrasts successful and failed trajectories for the same task and aggregates evidence across tasks to identify recurring behavioral deficiencies.<n>It decomposes the evolvable harness into five functional modules: Agent Loop, Tool Use, Observation Management, Context Management, and Task Completion Detection.
  arXiv  Detail & Related papers  (2026-09-14T00:10:45Z)
• HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness? [61.02699867449589]
  We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to run infrastructure.<n>In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system.<n>In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance.
  arXiv  Detail & Related papers  (2026-09-01T15:45:33Z)
• Aspire: Can Models Self-Evolve from Vague Goals? [61.65007006747275]
  We introduce ASPIRE, a benchmark for vague-goal-driven self-evolution.<n>ASPIRE supports both model-weight and agent-harness evolution in a unified interactive environment.<n>Our experiments show that vague goals redirect search effort toward goal interpretation.
  arXiv  Detail & Related papers  (2026-08-31T17:14:59Z)
• Harness Continual Learning: Continual Adaptation Beyond Model Parameters [9.54786133734043]
  We formulate a new continual learning paradigm in which the harness evolves around a frozen foundation model.<n>A candidate commits a harness only after checking current improvement, historical retention, and validity.<n>Experiments demonstrate capability accumulation and failure recovery, with relative gains exceeding 10% over corresponding baselines in multiple settings.
  arXiv  Detail & Related papers  (2026-08-19T15:12:31Z)
• Evo-Harness: Context-to-Harness Skill Compilation for Self-Evolving Agents [67.97365535829098]
  Learning from experience is critical for developing capable, self-improving large language model (LLM) agents.<n>We propose Evo-Harness, which distills noisy, single-shot executions into reusable skill harnesses for cross-domain and topic-level adaptation.<n>Our analysis demonstrates the effectiveness of Evo-Harness and provides a principled understanding of how LLM agents can effectively learn on the fly.
  arXiv  Detail & Related papers  (2026-08-15T06:43:56Z)
• Evo-Bench: Can Language Models Improve Agent Harness? [44.72089415085898]
  Evo-Bench is the first benchmark designed to evaluate models' intrinsic harness-evolving capabilities across Search, Office, and General agent domains.<n>Top models achieve massive absolute gains reaching 16.6 points, closely approaching state-of-the-art human-engineered baselines.
  arXiv  Detail & Related papers  (2026-08-10T03:49:28Z)
• Rethinking the Evaluation of Harness Evolution for Agents [83.07258924910069]
  Existing harness evolution methods use unit test cases to search for harness configurations and then report final performance on the same public benchmark.<n>This protocol raises two fundamental concerns.<n>As in agentic test-time scaling, it should therefore be compared with simple task-level search baselines under matched feedback and inference budgets.
  arXiv  Detail & Related papers  (2026-07-14T00:18:42Z)
• Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents [82.27610290890475]
  We analyze two harness self-evolution capabilities: (i) harness-updating, the capability to produce useful persistent harness updates from execution evidence; (ii) harness-benefit, the capability to benefit from updated harnesses during task solving.<n>Our analysis reveals two findings. First, harness-updating is flat in base capability: models from different capability tiers produce harness updates that lead to surprisingly similar gains.<n>Second, harness-benefit is non-monotonic in base capability: weak-tier models benefit little from updated harnesses, mid-tier models benefit most, and strong-tier models benefit less than mid-tier.
  arXiv  Detail & Related papers  (2026-05-28T22:16:14Z)
• DemoEvolve: Overcoming Sparse Feedback in Agentic Harness Evolution with Demonstrations [12.048713464721665]
  DemoEvolve is a demonstration-bootstrapped approach to harness evolution.<n>We study this paradigm as a form of sample-efficient fast adaptation.<n>We show that DemoEvolve produces more effective and auditable harness edits.
  arXiv  Detail & Related papers  (2026-05-23T12:10:52Z)
• Learning Without Losing Identity: Capability Evolution for Embodied Agents [14.412476605788482]
  We propose a capability-centric evolution paradigm for embodied agents.<n>We argue that a robot should maintain a persistent agent as its cognitive identity.<n>We present a unified framework in which capability evolution is decoupled from agent identity.
  arXiv  Detail & Related papers  (2026-04-09T04:51:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.