The Last Harness You'll Ever Build

• URL: http://arxiv.org/abs/2604.21003v2
• Date: Tue, 28 Apr 2026 02:33:56 GMT
• Title: The Last Harness You'll Ever Build
• Abstract summary: AI agents are increasingly deployed on complex, domain-specific applications.<n>We present a two-level framework that automates the harness engineering process.<n>We formalize the correspondence to meta-learning and present both algorithms.
• Score: 9.741938550862034
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI agents are increasingly deployed on complex, domain-specific workflows -- navigating enterprise web applications that require dozens of clicks and form fills, orchestrating multi-step research pipelines that span search, extraction, and synthesis, automating code review across unfamiliar repositories, and handling customer escalations that demand nuanced domain knowledge. \textbf{Each new task domain requires painstaking, expert-driven harness engineering}: designing the prompts, tools, orchestration logic, and evaluation criteria that make a foundation model effective. We present a two-level framework that automates this process. At the first level, the \textbf{Harness Evolution Loop} optimizes a worker agent's harness $\mathcal{H}$ for a single task: a Worker Agent $W_{\mathcal{H}}$ executes the task, an Evaluator Agent $V$ adversarially diagnoses failures and scores performance, and an Evolution Agent $E$ modifies the harness based on the full history of prior attempts. At the second level, the \textbf{Meta-Evolution Loop} optimizes the evolution blueprint $Λ= (W_{\mathcal{H}}, \mathcal{H}^{(0)}, V, E)$ itself across diverse tasks, \textbf{learning a blueprint $Λ^{(\text{best})}$ that enables rapid harness convergence on any new task -- so that adapting an agent to a novel domain requires no human harness engineering at all.} We formalize the correspondence to meta-learning and present both algorithms. The framework \textbf{shifts manual harness engineering into automated harness engineering}, and takes one step further -- \textbf{automating the design of the automation itself}.

Related papers

• An Interactive Agent for Requirement-Driven Candidate Sourcing [75.63902836826038]
  We present sys, the first requirements-driven candidate-sourcing agent.<n>It elicits, validates, retrieves, and verifies a vague people-request into a justified slate.<n>It dominates breadth ($100%$ coverage at $2.5times$ the yield) and is emphnear-orthogonal to the field.
  arXiv  Detail & Related papers  (2026-08-24T17:04:16Z)
• $τ_0$-VLA: a Hierarchical Robot Foundation Model with World-Model-Guided Test-Time Computation [51.05516457377339]
  Long-horizon robot manipulation requires a robot to both execute individual skills reliably and sequence them coherently over extended tasks.<n>We introduce $_0$-VLA, a hierarchical robot foundation model that formulates high-level subtask generation as a compute-scalable inference problem.<n>A low-level policy then executes the generated subtask across multiple robot embodiments.
  arXiv  Detail & Related papers  (2026-08-17T17:59:11Z)
• $A^2E$ : An End-to-End Agent Auditing Engine [25.281617098281277]
  $A2E$ is an end-to-end evaluation engine designed for agent harnesses.<n>It captures and generates standardized execution traces during experiments.<n>$A2E$ assesses harness capabilities using a suite of multidimensional metrics.
  arXiv  Detail & Related papers  (2026-08-07T15:44:12Z)
• A Matter of TASTE: Improving Coverage and Difficulty of Agent Benchmarks [25.713629634281077]
  We propose TASTE: Task Synthesis from Tool Sequence Evolution, an automatic method that generates challenging tasks with broader tool-use coverage.<n> TASTE selects representative sequences from the pool via clustering, instantiates them into complete benchmark tasks, and refines them through difficulty evolution.<n>Our results suggest high scores on existing benchmarks often reflect saturation rather than robust task-solving ability.
  arXiv  Detail & Related papers  (2026-05-27T14:45:59Z)
• AI Co-Scientist for Ranking: Discovering Novel Search Ranking Models alongside LLM-based AI Agents with Cloud Computing Access [60.37646376282544]
  We present an AI Co-Scientist framework that automates the full search ranking research pipeline.<n>This is the first study in the ranking community to utilize an AI Co-Scientist framework for algorithmic research.
  arXiv  Detail & Related papers  (2026-03-23T09:29:08Z)
• Do We Always Need Query-Level Workflows? Rethinking Agentic Workflow Generation for Multi-Agent Systems [72.3575737073235]
  Multi-Agent Systems (MAS) solve complex tasks by coordinating multiple agents through.<n>Existing approaches generates either at task level or query level, but their relative costs and benefits remain unclear.<n>We show that query-level workflow generation is not always necessary, since a small set of top-K best task-level together already covers equivalent or even more queries.
  arXiv  Detail & Related papers  (2026-01-16T10:05:51Z)
• Unison: A Fully Automatic, Task-Universal, and Low-Cost Framework for Unified Understanding and Generation [29.971317846027826]
  Unified understanding and generation is a highly appealing research direction in multimodal learning.<n>We propose Unison which adopts the two-stage scheme while preserving the capabilities of the pre-trained models well.<n>With an extremely low training cost, we cover a variety of multimodal understanding tasks, including text, image, and video understanding.<n>We also equip our model with the ability to automatically parse user intentions, determine the target task type, and accurately extract the meta-information required for the corresponding task.
  arXiv  Detail & Related papers  (2025-12-08T17:34:15Z)
• EvoFlow: Evolving Diverse Agentic Workflows On The Fly [21.82515160298748]
  EvoFlow is a niching evolutionary algorithm-based framework to automatically search a population of complexity and heterogeneous agentic.<n>We show that EvoFlow can evolve a population ranging from simple I/O tasks to complex multi-turn interactions.
  arXiv  Detail & Related papers  (2025-02-11T08:48:46Z)
• Multi-agent Architecture Search via Agentic Supernet [17.235963703597093]
  Large Language Model (LLM)-empowered multi-agent systems extend the cognitive boundaries of individual agents.<n>Despite the availability of methods to automate the design of agentic, they typically seek to identify a static, complex, one-size-fits-all system.<n>We introduce MaAS, an automated framework that samples query-dependent agentic systems from the supernet.
  arXiv  Detail & Related papers  (2025-02-06T16:12:06Z)
• RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graph [63.87660059104077]
  We present RepoGraph, a plug-in module that manages a repository-level structure for modern AI software engineering solutions.<n>RepoGraph substantially boosts the performance of all systems, leading to a new state-of-the-art among open-source frameworks.
  arXiv  Detail & Related papers  (2024-10-03T05:45:26Z)
• Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows? [73.81908518992161]
  We introduce Spider2-V, the first multimodal agent benchmark focusing on professional data science and engineering. Spider2-V features real-world tasks in authentic computer environments and incorporating 20 enterprise-level professional applications. These tasks evaluate the ability of a multimodal agent to perform data-related tasks by writing code and managing the GUI in enterprise data software systems.
  arXiv  Detail & Related papers  (2024-07-15T17:54:37Z)
• A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaboration [55.35849138235116]
  We propose automatically selecting a team of agents from candidates to collaborate in a dynamic communication structure toward different tasks and domains. Specifically, we build a framework named Dynamic LLM-Powered Agent Network ($textDyLAN$) for LLM-powered agent collaboration. We demonstrate that DyLAN outperforms strong baselines in code generation, decision-making, general reasoning, and arithmetic reasoning tasks with moderate computational cost.
  arXiv  Detail & Related papers  (2023-10-03T16:05:48Z)
• Cooperative Multi-Agent Reinforcement Learning: Asynchronous Communication and Linear Function Approximation [77.09836892653176]
  We study multi-agent reinforcement learning in the setting of episodic Markov decision processes. We propose a provably efficient algorithm based on value that enable asynchronous communication. We show that a minimal $Omega(dM)$ communication complexity is required to improve the performance through collaboration.
  arXiv  Detail & Related papers  (2023-05-10T20:29:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.