Automated Discovery Has No Universally Superior Harness

• URL: http://arxiv.org/abs/2607.18235v1
• Date: Mon, 20 Jul 2026 17:59:37 GMT
• Title: Automated Discovery Has No Universally Superior Harness
• Abstract summary: We decompose OpenEvolve-style evolutionary search and the TTT-Discover search harness into its constituent components.<n>We evaluate 30 budget-matched harnesses across 12 model-problem pairs.
• Score: 63.53666712024897
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Autonomous discovery systems such as OpenEvolve and TTT-Discover are often used as general-purpose harnesses. However, in practice these are composite systems combining several design choices about archives, parent selection, exploration, and budget allocation into a single recipe. Because discovery runs are expensive and inherently stochastic, existing harnesses are often compared using too few independent trials to distinguish key methodological improvements from run-to-run variance. We systematically decompose OpenEvolve-style evolutionary search and the TTT-Discover search harness into its constituent components and systematically evaluate 30 budget-matched harnesses across 12 model-problem pairs using more than 3.1 million LLM rollouts and repeated-trial statistical analysis. Our results show that discovery harnesses have a generalization problem: No fixed harness is reliably superior across the evaluated model-problem pairs, and variants of OpenEvolve generally underperform simpler alternatives. Thus, harness choice is better viewed as a hyperparameter rather than as a universal recipe, and should be tailored to the specific problem and underlying model. We also find that early discovery progress predicts final performance, and use this property to present a budget-matched adaptive-allocation experiment that starts multiple harnesses, prunes weak partial runs, and reallocates compute to stronger survivors, outperforming both commitment to a randomly sampled fixed harness and a non-adaptive harness ensemble. Together, these results motivate shifting from fixed harness selection to online adaptation guided by early performance. We release all run pools including baseline null distributions for every model-problem pair as reusable statistical infrastructure against for future harness proposals.

Related papers

• $A^2E$ : An End-to-End Agent Auditing Engine [25.281617098281277]
  $A2E$ is an end-to-end evaluation engine designed for agent harnesses.<n>It captures and generates standardized execution traces during experiments.<n>$A2E$ assesses harness capabilities using a suite of multidimensional metrics.
  arXiv  Detail & Related papers  (2026-08-07T15:44:12Z)
• Co-Harness: Co-Evolving Harnesses and Model Weights for LLM Agents [37.91666123749406]
  Co-Harness is a framework that jointly optimize the agent harness and model parameters during post-training.<n>An autonomous case study shows that Co-Harness can recover from system crashes, improve inference efficiency, and discover ensemble strategies without human intervention.
  arXiv  Detail & Related papers  (2026-07-17T02:39:57Z)
• Rethinking the Evaluation of Harness Evolution for Agents [83.07258924910069]
  Existing harness evolution methods use unit test cases to search for harness configurations and then report final performance on the same public benchmark.<n>This protocol raises two fundamental concerns.<n>As in agentic test-time scaling, it should therefore be compared with simple task-level search baselines under matched feedback and inference budgets.
  arXiv  Detail & Related papers  (2026-07-14T00:18:42Z)
• Evolving Agents in the Dark: Retrospective Harness Optimization via Self-Preference [51.55077364626896]
  We introduce Retrospective Harness Optimization (RHO), a self-supervised method that optimize agent harness using only past trajectories.<n>RHO selects a diverse coreset of challenging tasks from past trajectories and re-solves them in parallel.<n>A single optimization round improves the pass rate on SWE-Bench Pro from 59% to 78% without any external grading.
  arXiv  Detail & Related papers  (2026-06-04T09:26:00Z)
• Squeeze Evolve: Unified Multi-Model Orchestration for Verifier-Free Evolution [81.46210789228296]
  We introduce Squeeze Evolve, a unified multi-model orchestration framework for verifier-free evolutionary inference.<n>Our approach is guided by a simple principle: allocate model capability where it has the highest marginal utility.
  arXiv  Detail & Related papers  (2026-04-09T02:14:45Z)
• Distributionally Robust Feature Selection [14.493253907785473]
  We study the problem of selecting limited features to observe such that models trained on them can perform well simultaneously across multiple subpopulations.<n>Our method frames the problem as a continuous relaxation of traditional variable selection using a noising mechanism.<n>We develop a model-agnostic framework that balances overall performance of downstream prediction across populations.
  arXiv  Detail & Related papers  (2025-10-24T03:03:30Z)
• Noise or Signal? Deconstructing Contradictions and An Adaptive Remedy for Reversible Normalization in Time Series Forecasting [4.212879006865343]
  RevIN is a key technique enabling simple linear models to achieve state-of-the-art performance in time series forecasting.<n>This paper deconstructs the perplexing performance of various normalization strategies by identifying four underlying theoretical contradictions.
  arXiv  Detail & Related papers  (2025-10-06T10:22:20Z)
• MMR1: Enhancing Multimodal Reasoning with Variance-Aware Sampling and Open Resources [113.33902847941941]
  Variance-Aware Sampling (VAS) is a data selection strategy guided by Variance Promotion Score (VPS)<n>We release large-scale, carefully curated resources containing 1.6M long CoT cold-start data and 15k RL QA pairs.<n> Experiments across mathematical reasoning benchmarks demonstrate the effectiveness of both the curated data and the proposed VAS.
  arXiv  Detail & Related papers  (2025-09-25T14:58:29Z)
• Staying in the Sweet Spot: Responsive Reasoning Evolution via Capability-Adaptive Hint Scaffolding [59.60915947702282]
  Reinforcement learning with verifiable rewards (RLVR) has achieved remarkable success in enhancing the reasoning capabilities of large language models (LLMs)<n>Existing RLVR methods often suffer from exploration inefficiency due to mismatches between the training data's difficulty and the model's capability.<n>We propose SEELE, a novel supervision-aided RLVR framework that dynamically adjusts problem difficulty to stay within the high-efficiency region.
  arXiv  Detail & Related papers  (2025-09-08T17:36:21Z)
• Benign Overfitting in Out-of-Distribution Generalization of Linear Models [19.203753135860016]
  We take an initial step towards understanding benign overfitting in the Out-of-Distribution (OOD) regime.<n>We provide non-asymptotic guarantees proving that benign overfitting occurs in standard ridge regression.<n>We also present theoretical results for a more general family of target covariance matrix.
  arXiv  Detail & Related papers  (2024-12-19T02:47:39Z)
• Multiple Robust Learning for Recommendation [13.06593469196849]
  In recommender systems, a common problem is the presence of various biases in the collected data. We propose a multiple robust (MR) estimator that can take the advantage of multiple candidate imputation and propensity models to achieve unbiasedness.
  arXiv  Detail & Related papers  (2022-07-09T13:15:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.