Evo-Bench: Can Language Models Improve Agent Harness?

• URL: http://arxiv.org/abs/2608.09096v2
• Date: Tue, 11 Aug 2026 02:53:55 GMT
• Title: Evo-Bench: Can Language Models Improve Agent Harness?
• Abstract summary: Evo-Bench is the first benchmark designed to evaluate models' intrinsic harness-evolving capabilities across Search, Office, and General agent domains.<n>Top models achieve massive absolute gains reaching 16.6 points, closely approaching state-of-the-art human-engineered baselines.
• Score: 44.72089415085898
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) have driven rapid progress in autonomous agents, yet standard evaluations remain confined to static task solving. An emerging frontier is harness evolution---the agent's capacity to autonomously optimize its own operating harness. However, systematically benchmarking this capability remains challenging, as existing evaluations fail to isolate harness improvements from base model strength, prevent task-specific overfitting, or capture long-horizon iterative research. To address these challenges, we introduce Evo-Bench, the first benchmark designed to evaluate models' intrinsic harness-evolving capabilities across Search, Office, and General agent domains. To rigorously isolate this capability, Evo-Bench employs a novel harness-guided construction framework: it leverages auxiliary-task evolution to identify tasks genuinely sensitive to framework improvements, followed by sensitivity-aware stratified splitting to ensure robust cross-suite generalization. Extensive evaluations across nine frontier and open-weight models reveal that top models achieve massive absolute gains reaching 16.6 points, closely approaching state-of-the-art human-engineered baselines. Crucially, while autonomous evolution outpeforms artificial harness in General tasks and excels in Search tasks, it struggles in Office tasks that demand highly specific processing workflows. Furthermore, our analysis exposes critical temporal anomalies like early saturation, while demonstrating that the synthesized harnesses act as highly transferable reasoning structures, consistently boosting diverse policy models.

Related papers

• ModularRSI: Modular and Generalizable Recursive Harness Self-Improvement [46.36989489677189]
  We propose a benchmark-disjoint, contrastive, and modular framework for generalizable harness evolution.<n> ModularRSI contrasts successful and failed trajectories for the same task and aggregates evidence across tasks to identify recurring behavioral deficiencies.<n>It decomposes the evolvable harness into five functional modules: Agent Loop, Tool Use, Observation Management, Context Management, and Task Completion Detection.
  arXiv  Detail & Related papers  (2026-09-14T00:10:45Z)
• Ecdysis: Efficient and Effective Training of Runtime Harnesses for LLM Agents [25.41577484327689]
  Self-evolving runtime harnesses can substantially improve the capabilities of large language model (LLM) agents.<n>Existing harness evolution methods rely on iterative search, repeatedly evaluating and revising candidate harnesses.<n>We propose Ecdysis, an efficient and effective framework that distinguishes model-specific accommodation from systematic harness deficiencies.
  arXiv  Detail & Related papers  (2026-09-10T15:09:08Z)
• EVOHARNESSBENCH: Can Your Agents Keep Pace with an Evolving Harness? [83.21711423118768]
  EVOHARNESSBENCH is a benchmark for evaluating agents under controlled harness evolution.<n>It contains 17 multi-stage harness streams constructed deterministically from verifier-based benchmarks.
  arXiv  Detail & Related papers  (2026-09-03T03:24:02Z)
• AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks? [84.49732367417067]
  We introduce AutoLab, a new benchmark for ultra long-horizon closed-loop optimization.<n>AutoLab consists of 36 realistic, expert-curated tasks spanning four diverse domains.<n>We open-source the full benchmark, evaluation harness, and task artifacts, to accelerate research toward truly capable long-horizon agents.
  arXiv  Detail & Related papers  (2026-06-03T16:36:54Z)
• Reasoning Primitives in Hybrid and Non-Hybrid LLMs [10.629439705877054]
  We study reasoning through two primitives, recall and state-tracking.<n>We use matched Olmo3 transformer and hybrid models in instruction-tuned and reasoning-augmented variants.
  arXiv  Detail & Related papers  (2026-04-23T09:13:28Z)
• EvoClaw: Evaluating AI Agents on Continuous Software Evolution [47.49468375065129]
  We introduce DeepCommit, an agentic pipeline that reconstructs verifiable Milestone DAGs from noisy commit logs.<n>These executable sequences enable EvoClaw, a novel benchmark that requires agents to sustain system integrity and limit error accumulation.
  arXiv  Detail & Related papers  (2026-03-13T03:20:40Z)
• EmboCoach-Bench: Benchmarking AI Agents on Developing Embodied Robots [68.29056647487519]
  Embodied AI is fueled by high-fidelity simulation and large-scale data collection.<n>However, this scaling capability remains bottlenecked by a reliance on labor-intensive manual oversight.<n>We introduce textscEmboCoach-Bench, a benchmark evaluating the capacity of LLM agents to autonomously engineer embodied policies.
  arXiv  Detail & Related papers  (2026-01-29T11:33:49Z)
• Deep Self-Evolving Reasoning [22.934253026226155]
  We show that reasoning limits can be substantially extended through a probabilistic paradigm we call Deep Self-Evolving Reasoning (DSER)<n>On the challenging AIME 2024-2025 benchmark, DSER solves 5 out of 9 previously unsolvable problems and boosts overall performance.<n>Our findings establish a clear research agenda for developing next-generation models with powerful, intrinsic self-evolving capabilities.
  arXiv  Detail & Related papers  (2025-10-20T12:51:42Z)
• Towards Self-Evolving Benchmarks: Synthesizing Agent Trajectories via Test-Time Exploration under Validate-by-Reproduce Paradigm [60.36837655498119]
  We propose a Trajectory-based validated-by-Reproducing Agent-benchmark Complexity Evolution framework.<n>This framework takes an original task from an existing benchmark and encourages agents to evolve it into a new task with higher difficulty.<n>Experiments on the GAIA benchmark demonstrate that the TRACE framework consistently enhances task complexity while improving the reliability of correctness.
  arXiv  Detail & Related papers  (2025-10-01T01:52:52Z)
• SEEA-R1: Tree-Structured Reinforcement Fine-Tuning for Self-Evolving Embodied Agents [58.174206358223415]
  Self-Evolving Embodied Agents-R1, or SEEA-R1, is the first reinforcement fine-tuning framework designed for self-evolving embodied agents.<n>We show that SEEA-R1 can support autonomous adaptation and reward-driven self-evolution.
  arXiv  Detail & Related papers  (2025-06-26T18:00:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.