Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows

• URL: http://arxiv.org/abs/2605.27922v1
• Date: Wed, 27 May 2026 03:47:35 GMT
• Title: Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows
• Abstract summary: We introduce Harness-Bench, a diagnostic benchmark for evaluating configuration-level harness effects in realistic agent systems.<n>The benchmark contains 106 sandboxed offline tasks constructed from practical agent-use patterns.<n>Across 5,194 execution trajectories, we observe substantial variation in completion, process quality, efficiency, and failure behavior.
• Score: 18.6534256358905
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM agents are increasingly deployed as executable systems that use tools, modify workspaces, and produce concrete artifacts. In such workflows, performance depends not only on the base model, but also on the harness: the system layer that manages context, tools, state, constraints, permissions, tracing, and recovery. However, existing benchmarks typically abstract away execution, compare complete agent systems, or hold the harness fixed, making execution-layer variation difficult to study. We introduce Harness-Bench, a diagnostic benchmark for evaluating configuration-level harness effects in realistic agent workflows. Harness-Bench evaluates representative harness configurations across multiple model backends under shared task environments, budgets, and evaluation protocols, while preserving each harness's native execution behavior. The benchmark contains 106 sandboxed offline tasks constructed from practical agent-use patterns and manually reviewed for realism, solvability, oracle-checkability, and integrity. Each run records final artifacts, execution traces, usage statistics, and validator outputs, enabling analysis beyond final completion. Across 5,194 execution trajectories, we observe substantial variation in completion, process quality, efficiency, and failure behavior across model-harness pairings. These results suggest that agent capability should be reported at the model-harness configuration level rather than attributed to the base model alone. Our analysis further identifies recurring execution-alignment failures, where plausible reasoning becomes decoupled from tool feedback, workspace state, evidence, or verifiable output contracts. Harness-Bench provides a reproducible foundation for diagnosing and improving reliable, efficient, and auditable agent execution stacks.

Related papers

• Embodied-BenchForge: A Closed-Loop Agentic Workflow for Embodied Benchmark Construction [18.03387691479464]
  Embodied-BenchForge is an agentic framework that transforms user-specified evaluation intents into complete embodied benchmark artifacts.<n>It formulates construction as Closed-Loop Benchmark Synthesis, integrating forward artifact with backward verification and repair.
  arXiv  Detail & Related papers  (2026-09-11T17:16:01Z)
• DuMateBench: Evaluating Autonomous Agents in Complex Real-World Workflows [69.12339622053588]
  We introduce DuMateBench, a real-session benchmark reconstructed from anonymized and privacy-screened user sessions.<n>The resulting benchmark comprises 200 tasks spanning 8 broad scenarios and 17 fine-grained capability categories.<n>We execute these tasks in isolated Docker containers injected with three forms of real-world environmental complexity: Insufficient, Unstable, and Noisy.
  arXiv  Detail & Related papers  (2026-08-27T02:36:23Z)
• Execution-First Synthetic Tool-Use Trace Generation for LLM Agents [10.291533584425252]
  SyntheticAgentTraceQA is an execution- first framework for generating scalable supervision data for tool- augmented agents.<n>We evaluate the framework across four tool ecosystems and use the resulting data to fine-tune and evaluate Qwen model variants.
  arXiv  Detail & Related papers  (2026-07-31T08:52:26Z)
• PhyAgentOS: A Self-Evolving Operating System for Embodied Agents with Decoupled Cognitive Planning and Physical Execution [49.776611937968]
  We present PhyAgentOS, a foundation delivering scheduling, verification, memory, benchmarking, and safety as system-level services.<n>Its Session-Centered treats a session, not an action, as the minimum unit of scheduling, compatibility, supervised execution, evidence collection, and acceptance.<n>SessionVerifier distinguishes execution termination from semantic task completion via evidence-grounded verdicts of success, failure, or replan.<n> Benchmarking reuses the deployment session and verification path, so results trace to real execution.
  arXiv  Detail & Related papers  (2026-07-18T04:46:53Z)
• MemoHarness: Agent Harnesses That Learn from Experience [46.167787606315585]
  We introduce MemoHarness, an adaptive harness optimization framework that learns from its own executions.<n>MemoHarness decomposes the harness into six editable control dimensions, stores per-case diagnoses and distilled global patterns in a dual-layer experience bank.<n>In our evaluation across shell-agent, code-generation, and analytical-reasoning benchmarks, MemoHarness improves over the fixed harnesses we compare against and shows selective transfer to unseen suites and base models.
  arXiv  Detail & Related papers  (2026-07-14T21:22:18Z)
• TTHE: Test-Time Harness Evolution [50.26245555541721]
  Existing approaches optimize such harnesses before deployment, searching training or development data for a fixed agent workflow that is then frozen at test time.<n>We ask whether the harness can instead be optimized during evaluation itself, using only the unlabeled execution traces the agent produces on the test inputs.<n>During evaluation, TTHE maintains a population of candidate harnesses and refines them through an agentic proposer that reasons over their execution traces.<n>A judge then commits an improved harness from execution-derived proxy signals, and the selected program persists to govern subsequent inputs.
  arXiv  Detail & Related papers  (2026-07-09T05:53:39Z)
• From Question Answering to Task Completion: A Survey on Agent System and Harness Design [67.8241530947325]
  This survey examines LLM-based agents through a model-harness lens.<n>We first clarify the functional definition of agents and the implementation view of an LLM-based agent as a foundation model and an execution harness.<n>We then analyze the limits of model-centric scaling, trace four paradigms of agent engineering, and decompose the execution harness into six coupled runtime responsibilities.
  arXiv  Detail & Related papers  (2026-06-14T05:40:16Z)
• Benchmarks are Not Enough: RAMP for Runtime Assessing of Agentic Models in Production Systems [2.9725171307292544]
  We present RAMP, a production-grounded infrastructure for assessing software engineering agents.<n>RAMP provides a unified runtime assessment architecture through standardized orchestration and execution interfaces.
  arXiv  Detail & Related papers  (2026-05-26T16:28:10Z)
• CausalFlow: Causal Attribution and Counterfactual Repair for LLM Agent Failures [4.061481215870679]
  We introduce CausalFlow, an interventional framework that converts failed agent traces into minimal counterfactual repairs and reusable supervision.<n>CaulFlow supports two complementary uses: targeted test-time repair that recovers from failures with minimal behavioral drift, and training-time supervision suitable for offline preference optimization or reward modeling.
  arXiv  Detail & Related papers  (2026-05-25T01:47:01Z)
• ClawForge: Generating Executable Interactive Benchmarks for Command-Line Agents [59.626170560327274]
  We present textbfClawForge, a generator-backed benchmark framework for executable command-line categories under state conflict.<n>We instantiate this framework as the ClawForge-Bench (17 scenarios, 6 ability categories)
  arXiv  Detail & Related papers  (2026-05-13T21:34:08Z)
• Constraint Decay: The Fragility of LLM Agents in Backend Code Generation [9.659020624935687]
  Large Language Model (LLM) agents demonstrate strong performance in autonomous code generation under loose specifications.<n>Production-grade software requires strict adherence to structural constraints, such as architectural patterns, databases, and object-relational mappings.<n>We present a systematic study evaluating how well agents handle structural constraints in backend generation.
  arXiv  Detail & Related papers  (2026-05-07T15:44:40Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• Claw-Eval-Live: A Live Agent Benchmark for Evolving Real-World Workflows [67.92316850084575]
  We introduce Claw-Eval-Live, a live benchmark for workflow agents.<n>Each release is constructed from public workflow-demand signals.<n>Claw-Eval-Live records execution traces, audit logs, service state, and post-run workspace artifacts.
  arXiv  Detail & Related papers  (2026-04-30T17:23:19Z)
• GTA-2: Benchmarking General Tool Agents from Atomic Tool-Use to Open-Ended Workflows [90.35728421223673]
  GTA-2 is a hierarchical benchmark for General Tool Agents (GTA)<n>Built on real-world authenticity, it leverages real user queries, deployed tools, and multimodal contexts.<n>Experiments reveal a pronounced capability cliff: while frontier models already struggle on atomic tasks, top models achieving only 14.39% success.
  arXiv  Detail & Related papers  (2026-04-17T05:36:00Z)
• TopoPilot: Reliable Conversational Workflow Automation for Topological Data Analysis and Visualization [4.62716665682001]
  TopoPilot is a reliable and agentic framework for automating complex scientific visualization.<n>TopoPilot incorporates systematic guardrails and verification mechanisms to ensure reliable operation.<n>In evaluations, TopoPilot achieves a success rate exceeding 99%, compared to under 50% for baselines without comprehensive guardrails and checks.
  arXiv  Detail & Related papers  (2026-03-26T05:56:53Z)
• Automatic Building Code Review: A Case Study [6.530899637501737]
  Building officials face labor-intensive, error-prone, and costly manual reviews of design documents as projects increase in size and complexity.<n>This study introduces a novel agent-driven framework that integrates BIM-based data extraction with automated verification.
  arXiv  Detail & Related papers  (2025-10-03T00:30:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.