Towards Direct Evaluation of Harness Optimizers via Priority Ranking

• URL: http://arxiv.org/abs/2605.22505v1
• Date: Thu, 21 May 2026 13:55:02 GMT
• Title: Towards Direct Evaluation of Harness Optimizers via Priority Ranking
• Abstract summary: It is unclear whether harness optimization is driven by agents' informed update actions or simply trial-and-error.<n> evaluating harnesss directly is non-trivial and costly due to the lack of oracle harnesses.<n>We present a simple, low-cost design directly to evaluate them, namely priority ranking.
• Score: 18.644878456671886
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Harness optimization enables automated agent creation by having an optimizer agent iteratively update the harness of target agents. Despite its success, current studies evaluate optimizers solely by observing target agents' performance gains. This indirect end-improvement evaluation neglects optimizers' actions at intermediate steps, which are often erroneous and hinder agent performance. Therefore, it is unclear whether harness optimization is driven by optimizers' informed update actions or simply trial-and-error. This necessitates direct evaluation of harness optimizers. However, evaluating harness optimizers directly is non-trivial and costly due to the lack of oracle harnesses. To address this, we present a simple, low-cost design to directly evaluate them, namely priority ranking. By asking harness optimizers to rank components (e.g., tools) in a given harness by their potential to improve/hinder agent performance when updated, our design quantifies optimizer ability at the step level without expensive rollouts or manual examination. More importantly, optimizers' ranking performance correlates with their ability to improve agents in actual multi-step harness optimization, establishing priority ranking as a reliable predictor of optimization ability. Priority ranking is enabled by Shor, a collection of 182 human-verified optimization scenarios spanning across domains, designs, and time stages. Codes and data can be found at https://github.com/k59118/Harness_Optimizer_Evaluation.

Related papers

• COBRA-Skills: Contextual Bandit-Guided Evolution for Agent Skill Optimization [55.60171725587438]
  Large language model (LLM) agents can benefit from reusable skills distilled from prior task experience.<n>Existing skill optimization methods often rely on costly execution-based evaluation and substantial task data.<n>We introduce textbfCOBRA-Skills, an efficient framework that formulates skill optimization as budgeted sequential optimization over a dynamically evolving candidate space.
  arXiv  Detail & Related papers  (2026-09-10T15:12:24Z)
• CORAL: An LLM-Native Harness for Production Recommender Systems [26.7379632070749]
  We present CORAL (Constraint-d Recommender via an Agentic Loop), an agentic loop that learns from the measured effects of its decisions.<n>We formulate this as a partially observed, non-stationary, constrained optimization problem in which the policy improves in context.<n>Across two large-scale social platforms, evaluated with A/B experiments, the same harness improves engagement at no additional serving cost on one and reduces serving cost without degrading engagement on the other.
  arXiv  Detail & Related papers  (2026-09-02T15:40:36Z)
• HarnessOpt-Bench: Evaluating LLMs at Harness Optimization [3.846854817984248]
  We introduce a benchmark for end-to-end harness optimization under expensive evaluation regimes.<n>A trusted execution environment enforces the evaluation boundary, target-agent resource use, and preserves candidate versions for audit.<n>Experiment results show that models separate more than the coding harnesses they act through, native harnesses are not consistently superior, and gains vary substantially across tasks.
  arXiv  Detail & Related papers  (2026-08-06T17:21:05Z)
• Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0 [53.71882250666667]
  Most reported gains from agent-optimization methods are one-shot.<n>This does not test the setting that matters for deployed agents.<n>We compare three approaches to agent-harness optimization.
  arXiv  Detail & Related papers  (2026-07-15T16:36:04Z)
• Evolving Agents in the Dark: Retrospective Harness Optimization via Self-Preference [51.55077364626896]
  We introduce Retrospective Harness Optimization (RHO), a self-supervised method that optimize agent harness using only past trajectories.<n>RHO selects a diverse coreset of challenging tasks from past trajectories and re-solves them in parallel.<n>A single optimization round improves the pass rate on SWE-Bench Pro from 59% to 78% without any external grading.
  arXiv  Detail & Related papers  (2026-06-04T09:26:00Z)
• Escher-Loop: Mutual Evolution by Closed-Loop Self-Referential Optimization [7.89766538351877]
  Escher-Loop is a framework that operationalizes the evolution of two distinct populations: Task Agents that solve problems, and Agents that refine both the task agents and themselves.<n>We show that Escher-Loop effectively pushes past the performance ceilings of static baselines, achieving the highest absolute peak performance.<n>Remarkably, we observe that agents dynamically adapt their strategies to match the shifting demands of high-performing task agents, which explains the system's continuous improvement and superior late-stage performance.
  arXiv  Detail & Related papers  (2026-04-25T23:46:08Z)
• Self-Steering Optimization: Autonomous Preference Optimization for Large Language Models [79.84205827056907]
  We present Self-Steering Optimization ($SSO$), an algorithm that autonomously generates high-quality preference data.<n>$SSO$ employs a specialized optimization objective to build a data generator from the policy model itself, which is used to produce accurate and on-policy data.<n>Our evaluation shows that $SSO$ consistently outperforms baselines in human preference alignment and reward optimization.
  arXiv  Detail & Related papers  (2024-10-22T16:04:03Z)
• Understanding Likelihood Over-optimisation in Direct Alignment Algorithms [20.043560907227018]
  Direct Alignment Algorithms (DAAs) have emerged as alternatives to online Reinforcement Learning from Human Feedback. These algorithms aim to increase the likelihood of generating better (preferred) completions while discouraging worse (non-preferred) ones. This work explores the relationship between completion likelihood and model performance in state-of-the-art DAAs.
  arXiv  Detail & Related papers  (2024-10-15T15:14:22Z)
• Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimation [46.61909578101735]
  Adversarial Policy Optimization (AdvPO) is a novel solution to the pervasive issue of reward over-optimization in Reinforcement Learning from Human Feedback. In this paper, we introduce a lightweight way to quantify uncertainties in rewards, relying solely on the last layer embeddings of the reward model.
  arXiv  Detail & Related papers  (2024-03-08T09:20:12Z)
• Lower-Left Partial AUC: An Effective and Efficient Optimization Metric for Recommendation [52.45394284415614]
  We propose a new optimization metric, Lower-Left Partial AUC (LLPAUC), which is computationally efficient like AUC but strongly correlates with Top-K ranking metrics. LLPAUC considers only the partial area under the ROC curve in the Lower-Left corner to push the optimization focus on Top-K.
  arXiv  Detail & Related papers  (2024-02-29T13:58:33Z)
• MADA: Meta-Adaptive Optimizers through hyper-gradient Descent [73.1383658672682]
  We introduce Meta-Adaptives (MADA), a unified framework that can generalize several known convergences and dynamically learn the most suitable one during training. We empirically compare MADA to other populars on vision and language tasks, and find that MADA consistently outperforms Adam and other populars. We also propose AVGrad, a modification of AMS that replaces the maximum operator with averaging, which is more suitable for hyper-gradient optimization.
  arXiv  Detail & Related papers  (2024-01-17T00:16:46Z)
• DeepHive: A multi-agent reinforcement learning approach for automated discovery of swarm-based optimization policies [0.0]
  The state of each agent within the swarm is defined as its current position and function value within a design space. The proposed approach is tested on various benchmark optimization functions and compared to the performance of other global optimization strategies.
  arXiv  Detail & Related papers  (2023-03-29T18:08:08Z)
• Regret-Aware Black-Box Optimization with Natural Gradients, Trust-Regions and Entropy Control [17.430247457941284]
  Most successful black-boxs, such as CMA-ES, use rankings of the individual samples to obtain a new search distribution. While these algorithms typically produce a high-quality mean estimate of the search distribution, the produced samples can have poor quality as these algorithms are ignorant of the regret. In contrast, the Relative Entropy Search (MORE) algorithm directly optimize the expected fitness function without the use of rankings.
  arXiv  Detail & Related papers  (2022-05-24T16:25:15Z)
• Training Learned Optimizers with Randomly Initialized Learned Optimizers [49.67678615506608]
  We show that a population of randomly learneds can be used to train themselves from scratch in an online fashion. A form of population based training is used to orchestrate this self-training. We believe feedback loops of this type will be important and powerful in the future of machine learning.
  arXiv  Detail & Related papers  (2021-01-14T19:07:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.