Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0

• URL: http://arxiv.org/abs/2607.14004v1
• Date: Wed, 15 Jul 2026 16:36:04 GMT
• Title: Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0
• Abstract summary: Most reported gains from agent-optimization methods are one-shot.<n>This does not test the setting that matters for deployed agents.<n>We compare three approaches to agent-harness optimization.
• Score: 53.71882250666667
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Most reported gains from agent-optimization methods are one-shot: an agent is optimized against a fixed benchmark and the resulting improvement is reported as if it were a stable property of the method. This does not test the setting that matters for deployed agents, where optimization is applied recursively as new failures and new tasks appear over time. The central question this raises is whether optimizer-driven gains compound: after an agent has been optimized once, can it be optimized again on newly arrived tasks without eroding the gains the first round produced? We study this question with a two-phase continual-learning evaluation built from hard tasks in Terminal-Bench 2.0, comparing three approaches to agent-harness optimization (GEPA, Meta Harness, and RELAI's Verifiable Continual Learning, RELAI-VCL) under identical optimization budgets. All three methods improve over the baseline agent in the conventional, static, single-phase setting. However, once new tasks are introduced, the methods diverge sharply: GEPA's optimized agent transfers below the unoptimized baseline, Meta Harness transfers well but fails to improve further once given a second optimization budget, and RELAI-VCL is the only method that both transfers positively to unseen tasks and continues improving after those tasks are folded into the optimization objective, reaching the highest pass rate at every evaluated stage and the highest lifelong average pass rate overall (76.4% vs. 66.0% for GEPA, 64.6% for Meta Harness, and 58.7% for the baseline). Our key observation was that optimization gains compounded only when regression control was built into the optimization loop, providing an inductive bias against shortcut solutions that fail to generalize.

Related papers

• COBRA-Skills: Contextual Bandit-Guided Evolution for Agent Skill Optimization [55.60171725587438]
  Large language model (LLM) agents can benefit from reusable skills distilled from prior task experience.<n>Existing skill optimization methods often rely on costly execution-based evaluation and substantial task data.<n>We introduce textbfCOBRA-Skills, an efficient framework that formulates skill optimization as budgeted sequential optimization over a dynamically evolving candidate space.
  arXiv  Detail & Related papers  (2026-09-10T15:12:24Z)
• MAGE: Understanding Stability-Performance Trade-offs in Multi-component Prompt Optimization [3.7323766833175718]
  MAGE (Memory-Augmented Goal Prompt Evolution) is a controlled analysis framework for studying component interaction in prompt optimization.<n>Our experiments uncover a previously unreported phenomenon, the Prompt Optimization Coupling Effect (POCE), when multiple optimization signals operate within a closed reflective loop.
  arXiv  Detail & Related papers  (2026-07-11T10:05:58Z)
• Evolving Agents in the Dark: Retrospective Harness Optimization via Self-Preference [51.55077364626896]
  We introduce Retrospective Harness Optimization (RHO), a self-supervised method that optimize agent harness using only past trajectories.<n>RHO selects a diverse coreset of challenging tasks from past trajectories and re-solves them in parallel.<n>A single optimization round improves the pass rate on SWE-Bench Pro from 59% to 78% without any external grading.
  arXiv  Detail & Related papers  (2026-06-04T09:26:00Z)
• Towards Direct Evaluation of Harness Optimizers via Priority Ranking [18.644878456671886]
  It is unclear whether harness optimization is driven by agents' informed update actions or simply trial-and-error.<n> evaluating harnesss directly is non-trivial and costly due to the lack of oracle harnesses.<n>We present a simple, low-cost design directly to evaluate them, namely priority ranking.
  arXiv  Detail & Related papers  (2026-05-21T13:55:02Z)
• AdaExplore: Failure-Driven Adaptation and Diversity-Preserving Search for Efficient Kernel Generation [59.964989458924585]
  AdaExplore is an agent framework that enables self-improvement via accumulated execution feedback for performance-critical kernel code generation.<n>In the adaptation stage, the agent synthesizes tasks and converts recurring failures into a reusable memory of validity rules.<n>In the search stage, the agent organizes candidate kernels as a tree and alternates between small local refinements and larger structural regeneration.
  arXiv  Detail & Related papers  (2026-04-17T18:25:03Z)
• dTRPO: Trajectory Reduction in Policy Optimization of Diffusion Large Language Models [71.26391195427878]
  Diffusion Large Language Models (dLLMs) introduce a new paradigm for language generation.<n>In this work, we aim to improve the policy optimization for dLLMs by reducing the cost of the trajectory probability calculation.<n>We evaluate dTRPO on 7B dLLMs across instruction-following and reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-03-19T11:55:52Z)
• PepEVOLVE: Position-Aware Dynamic Peptide Optimization via Group-Relative Advantage [0.0]
  PepEVOLVE is a position-aware, dynamic framework that learns both where to edit and how to dynamically optimize peptides for multi-objective improvement.<n>On a therapeutically motivated Rev-binding macrocycle benchmark, PepEVOLVE outperformed PepINVENT by reaching higher mean scores.
  arXiv  Detail & Related papers  (2025-11-21T02:51:15Z)
• OptimAI: Optimization from Natural Language Using LLM-Powered AI Agents [8.441638148384389]
  We introduce OptimAI, a framework for solving Optimization problems described in natural language.<n>Our framework is built upon the following key roles: formulator, planner, coder and code critic.<n>Our approach attains 88.1% accuracy on the NLP4LP dataset and 82.3% on the Optibench dataset, reducing error rates by 58% and 52%, respectively, over prior best results.
  arXiv  Detail & Related papers  (2025-04-23T17:45:05Z)
• Dynamic Noise Preference Optimization for LLM Self-Improvement via Synthetic Data [51.62162460809116]
  We introduce Dynamic Noise Preference Optimization (DNPO) to ensure consistent improvements across iterations.<n>In experiments with Zephyr-7B, DNPO consistently outperforms existing methods, showing an average performance boost of 2.6%.<n> DNPO shows a significant improvement in model-generated data quality, with a 29.4% win-loss rate gap compared to the baseline in GPT-4 evaluations.
  arXiv  Detail & Related papers  (2025-02-08T01:20:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.