Same Model, Different Harness: Different Coding-Agent Results

• URL: http://arxiv.org/abs/2608.26218v1
• Date: Wed, 26 Aug 2026 11:55:44 GMT
• Title: Same Model, Different Harness: Different Coding-Agent Results
• Abstract summary: We compare two configurations of the same harness on three coding benchmarks.<n>The control supplies the full conversation in time order, while the treatment keeps the same record but mechanically shortens older tool results.<n>Under tight context, the treatment raises mean per-task fail-to-pass fraction (F2PF) in all three pressure comparisons.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A coding agent combines a model with a harness, which decides what the model sees, which tools it can use, and how the work continues. We ask whether changing the harness changes the result when the model and task stay fixed. We compare two configurations of the same harness on three coding benchmarks. The control supplies the full conversation in time order, while the treatment keeps the same record but mechanically shortens older tool results as the context fills and responds to repeated or stalled work. Under tight context, the treatment raises mean per-task fail-to-pass fraction (F2PF) in all three pressure comparisons and increases complete solutions on SWE-bench Verified and SWE-bench Pro. The tight-window Verified comparison uses 169 tasks, a 20,480-token window, and a fixed 480-second attempt endpoint; on this cohort, treatment raises mean per-task F2PF from 28 percent to 49 percent and complete solutions from 43 to 72. Without model-specific retuning, the same frozen treatment also raises both endpoints on the same cohort for three additional models with different designs. In the wide-window Qwen3.6 comparisons, observed arm outcomes are close on Verified and Pro, while FeatureBench retains a higher mean per-task F2PF under treatment. On the wide-window Verified cohort, treatment also serves fewer prompt tokens per turn. Because changing the harness changed what unchanged model weights could accomplish, coding-agent evaluations should treat the model and harness together as the tested solver.

Related papers

• HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness? [61.02699867449589]
  We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to run infrastructure.<n>In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system.<n>In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance.
  arXiv  Detail & Related papers  (2026-09-01T15:45:33Z)
• CONTRAMEM: Learning Self-Evolving Procedural Memory from Contrasting Multi-Model Trajectories [13.649910962870896]
  CONTRAMEM is a source-flexible, training-free framework for self-evolving procedural memory.<n>On held-out GAIA2/ARE computer-use tasks, CONTRAMEM more than doubles the success rate.
  arXiv  Detail & Related papers  (2026-08-23T18:11:26Z)
• Task-CoEvolve: Efficient Harness Optimization via Adaptive Validation Task Selection [51.39973047614183]
  We present a novel approach to efficient LLM harness optimization through adaptive validation task selection.<n>$textbfTask-CoEvolve$, which co-evolves the validation tasks with the harness by addressing two challenges: selecting informative tasks and estimating full-set performance from partial evaluations.
  arXiv  Detail & Related papers  (2026-08-20T15:24:54Z)
• HELIX: Model-Harness Co-evolution for Recursive Self-Improvement [8.484282837156933]
  We present HELIX, a source-traceable substrate for harness evolution.<n>We show how harness evolution expands current capability and creates learning signal for the next model.
  arXiv  Detail & Related papers  (2026-08-14T04:44:53Z)
• DarwinX: Evolving Agent Harnesses Through Natural Selection [48.64258219266629]
  DarwinX treats self-evolution as selection over a population of harnesses with the model frozen.<n>General agent competence, not benchmark-specific patches, so it survives changes of task, verifier, and base model.
  arXiv  Detail & Related papers  (2026-07-31T05:34:02Z)
• The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation [0.0]
  We evaluate Qwen 3.6 Plus and MiniMax M2.5 across three open-source harnesses.<n> Harness choice induces up to a 40x difference in tokens per solved task.<n>We recommend selecting harness-model pairs by pass rate under token/latency budgets.
  arXiv  Detail & Related papers  (2026-06-08T20:27:06Z)
• Three Roles, One Model: Role Orchestration at Inference Time to Close the Performance Gap Between Small and Large Agents [0.4666493857924357]
  We study whether inference-time scaffolding alone, without any additional training compute, can improve the performance of a small model in complex multi-step environments.<n>We evaluate Qwen3-8B on the AppWorld benchmark under both full-precision and 4-bit quantized configurations.<n>On full-precision inference, our scaffolded 8B model surpasses DeepSeek-Coder 33B Instruct (7.1%) from the original AppWorld evaluation.
  arXiv  Detail & Related papers  (2026-04-13T13:40:33Z)
• When Agents Disagree With Themselves: Measuring Behavioral Consistency in LLM-Based Agents [0.0]
  ReAct-style agents produce 2.0--4.2 distinct action sequences per 10 runs on average, even with identical inputs.<n>Tasks with consistent behavior achieve 80--92% accuracy, while highly inconsistent tasks achieve only 25--60%.<n>Our results suggest that monitoring behavioral consistency during execution could enable early error detection and improve agent reliability.
  arXiv  Detail & Related papers  (2026-02-12T06:15:14Z)
• AlphaApollo: Orchestrating Foundation Models and Professional Tools into a Self-Evolving System for Deep Agentic Reasoning [110.57865233597762]
  We present AlphaApollo, a self-evolving agentic reasoning system.<n>It aims to address two bottlenecks in foundation model (FM) reasoning-limited model-intrinsic capacity and unreliable test-time computation.<n>AlphaApollo orchestrates multiple models with professional tools to enable deliberate, verifiable reasoning.
  arXiv  Detail & Related papers  (2025-10-05T15:42:24Z)
• Localizing Task Information for Improved Model Merging and Compression [61.16012721460561]
  We show that the information required to solve each task is still preserved after merging as different tasks mostly use non-overlapping sets of weights. We propose Consensus Merging, an algorithm that eliminates such weights and improves the general performance of existing model merging approaches.
  arXiv  Detail & Related papers  (2024-05-13T14:54:37Z)
• Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inference [119.19779637025444]
  Deep networks were recently suggested to face the odds between accuracy (on clean natural images) and robustness (on adversarially perturbed images) This paper studies multi-exit networks associated with input-adaptive inference, showing their strong promise in achieving a "sweet point" in cooptimizing model accuracy, robustness and efficiency.
  arXiv  Detail & Related papers  (2020-02-24T00:40:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.