Meta-Harness: End-to-End Optimization of Model Harnesses

• URL: http://arxiv.org/abs/2603.28052v1
• Date: Mon, 30 Mar 2026 05:33:50 GMT
• Title: Meta-Harness: End-to-End Optimization of Model Harnesses
• Abstract summary: We introduce Meta-Harness, an outer-loop system that searches over harness code for large language model (LLM) applications.<n>On online text classification, Meta-Harness improves over a state-of-the-art context management system by 7.7 points while using 4x fewer context tokens.<n>On retrieval-augmented math reasoning, a single discovered harness improves accuracy on 200 IMO-level problems by 4.7 points on average across five held-out models.
• Score: 52.31507076660471
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The performance of large language model (LLM) systems depends not only on model weights, but also on their harness: the code that determines what information to store, retrieve, and present to the model. Yet harnesses are still designed largely by hand, and existing text optimizers are poorly matched to this setting because they compress feedback too aggressively. We introduce Meta-Harness, an outer-loop system that searches over harness code for LLM applications. It uses an agentic proposer that accesses the source code, scores, and execution traces of all prior candidates through a filesystem. On online text classification, Meta-Harness improves over a state-of-the-art context management system by 7.7 points while using 4x fewer context tokens. On retrieval-augmented math reasoning, a single discovered harness improves accuracy on 200 IMO-level problems by 4.7 points on average across five held-out models. On agentic coding, discovered harnesses surpass the best hand-engineered baselines on TerminalBench-2. Together, these results show that richer access to prior experience can enable automated harness engineering.

Related papers

• HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness? [61.02699867449589]
  We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to run infrastructure.<n>In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system.<n>In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance.
  arXiv  Detail & Related papers  (2026-09-01T15:45:33Z)
• WHALE: A Simple Recipe for Joint Harness-Weight Optimization [48.550149284101536]
  Weight-Harness Alternating LEarning (WHALE) is a recipe that alternates two phases: updating the model under the current harness, then searching for a better harness under the updated model.<n>Using Qwen3.5-2B/4B agents across three domains (search question answering, mathematical reasoning, and chess puzzles), WHALE outperforms weight-only, harness-only, and Fast-Slow Training by 4.15-24.38 percentage points in best mean@8 accuracy.
  arXiv  Detail & Related papers  (2026-08-31T18:12:55Z)
• AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces [61.51487329124135]
  AutoSaddler is a framework that formulates harness improvement as an offline learning problem.<n>It iteratively updates the harness using failure signals from mini-batches.<n>Experiments on GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0 show that AutoSaddler substantially improves agent performance.
  arXiv  Detail & Related papers  (2026-08-24T09:45:08Z)
• HarnessOpt-Bench: Evaluating LLMs at Harness Optimization [3.846854817984248]
  We introduce a benchmark for end-to-end harness optimization under expensive evaluation regimes.<n>A trusted execution environment enforces the evaluation boundary, target-agent resource use, and preserves candidate versions for audit.<n>Experiment results show that models separate more than the coding harnesses they act through, native harnesses are not consistently superior, and gains vary substantially across tasks.
  arXiv  Detail & Related papers  (2026-08-06T17:21:05Z)
• Fantastic Adaptive Taxonomies and How to Use Them [100.20614173157708]
  AdaMAST converts execution traces into a compact, evidence-grounded failure taxonomy.<n>No code is hand-authored, no trace human-annotated.<n>In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form.
  arXiv  Detail & Related papers  (2026-07-17T17:41:16Z)
• HarnessX: A Composable, Adaptive, and Evolvable Agent Harness Foundry [35.87794858139959]
  HarnessX is a foundry for composable, adaptive, and evolvable agent harnesses.<n>It assembles typed harness primitives via a substitution algebra, adapts them through AEGIS.<n>It closes the harness-model loop by turning trajectories into both harness updates and model training signal.
  arXiv  Detail & Related papers  (2026-06-12T08:27:11Z)
• The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation [0.0]
  We evaluate Qwen 3.6 Plus and MiniMax M2.5 across three open-source harnesses.<n> Harness choice induces up to a 40x difference in tokens per solved task.<n>We recommend selecting harness-model pairs by pass rate under token/latency budgets.
  arXiv  Detail & Related papers  (2026-06-08T20:27:06Z)
• FrontierSmith: Synthesizing Open-Ended Coding Problems at Scale [73.57353294235223]
  We introduce FrontierSmith, an automated system for evolving open-ended problems from existing closed-ended coding tasks.<n>On two open-ended coding benchmarks, training on our synthesized data yields substantial gains over the base models.<n>The synthesized problems also make agents take more turns and use more tokens, similar to human-curated ones.
  arXiv  Detail & Related papers  (2026-05-14T06:39:42Z)
• STELLAR: A Search-Based Testing Framework for Large Language Model Applications [1.0125319485475452]
  Large Language Model (LLM)-based applications are increasingly deployed across various domains, including customer service, education, and mobility.<n>These systems are prone to inaccurate, fictitious, or harmful responses, and their vast, high-dimensional input space makes systematic testing particularly challenging.<n>We present STELLAR, an automated search-based testing framework for LLM-based applications that systematically uncovers text inputs leading to inappropriate system responses.
  arXiv  Detail & Related papers  (2026-01-01T22:30:15Z)
• Framework-Aware Code Generation with API Knowledge Graph-Constructed Data: A Study on HarmonyOS [52.483888557864326]
  APIKG4SYN is a framework designed to exploit API knowledge graphs for the construction of API-oriented question-code pairs.<n>We build the first benchmark for HarmonyOS code generation using APIKG4SYN.
  arXiv  Detail & Related papers  (2025-11-29T08:13:54Z)
• Reverse Engineering User Stories from Code using Large Language Models [19.450989646157716]
  We investigate whether large language models (LLMs) can automatically recover user stories directly from source code.<n>We evaluate five state-of-the-art LLMs across six prompting strategies.<n>Results show that all models achieve, on average, an F1 score of 0.8 for code up to 200 NLOC.
  arXiv  Detail & Related papers  (2025-09-23T21:23:37Z)
• Large Language Models Versus Static Code Analysis Tools: A Systematic Benchmark for Vulnerability Detection [0.0]
  Three industry-standard rule-based static code-analysis tools (Sonar, CodeQL and Snyk Code) and three state-of-the-art large language models hosted on the GitHub Models platform (GPT-4.1, Mistral Large and DeepSeek V3) were evaluated.<n>Using a curated suite of ten real-world C# projects that embed 63 vulnerabilities, we measure classical accuracy (precision, recall, F-score), analysis latency, granularity and the developer effort required to vet true positives.<n>We recommend a hybrid pipeline: employ language models early in development for broad, context-aware detection and
  arXiv  Detail & Related papers  (2025-08-06T13:48:38Z)
• MM-RLHF: The Next Step Forward in Multimodal LLM Alignment [59.536850459059856]
  We introduce MM-RLHF, a dataset containing $mathbf120k$ fine-grained, human-annotated preference comparison pairs.<n>We propose several key innovations to improve the quality of reward models and the efficiency of alignment algorithms.<n>Our approach is rigorously evaluated across $mathbf10$ distinct dimensions and $mathbf27$ benchmarks.
  arXiv  Detail & Related papers  (2025-02-14T18:59:51Z)
• Are Decoder-Only Large Language Models the Silver Bullet for Code Search? [44.9422305001193]
  Code search is essential for code reuse, allowing developers to efficiently locate relevant code snippets.<n>Powerful decoder-only Large Language Models (LLMs) has revolutionized many code intelligence tasks.<n>This paper presents a systematic evaluation of eleven decoder-only LLMs, analyzing their performance across zero-shot and fine-tuned settings.
  arXiv  Detail & Related papers  (2024-10-29T17:05:25Z)
• SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal [64.9938658716425]
  SORRY-Bench is a proposed benchmark for evaluating large language models' (LLMs) ability to recognize and reject unsafe user requests.<n>First, existing methods often use coarse-grained taxonomy of unsafe topics, and are over-representing some fine-grained topics.<n>Second, linguistic characteristics and formatting of prompts are often overlooked, like different languages, dialects, and more -- which are only implicitly considered in many evaluations.
  arXiv  Detail & Related papers  (2024-06-20T17:56:07Z)
• Autoregressive Search Engines: Generating Substrings as Document Identifiers [53.0729058170278]
  Autoregressive language models are emerging as the de-facto standard for generating answers. Previous work has explored ways to partition the search space into hierarchical structures. In this work we propose an alternative that doesn't force any structure in the search space: using all ngrams in a passage as its possible identifiers.
  arXiv  Detail & Related papers  (2022-04-22T10:45:01Z)
• When Liebig's Barrel Meets Facial Landmark Detection: A Practical Model [87.25037167380522]
  We propose a model that is accurate, robust, efficient, generalizable, and end-to-end trainable. In order to achieve a better accuracy, we propose two lightweight modules. DQInit dynamically initializes the queries of decoder from the inputs, enabling the model to achieve as good accuracy as the ones with multiple decoder layers. QAMem is designed to enhance the discriminative ability of queries on low-resolution feature maps by assigning separate memory values to each query rather than a shared one.
  arXiv  Detail & Related papers  (2021-05-27T13:51:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.