HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnesses
- URL: http://arxiv.org/abs/2608.01918v1
- Date: Mon, 03 Aug 2026 08:51:50 GMT
- Title: HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnesses
- Abstract summary: Harness design plays a critical role in agent performance by shaping how large language models (LLMs) perceive, reason over, and act within executable environments.<n>Recent work has proposed automatic harness evolution, which iteratively improves the harness from agent--environment interactions.<n>We propose Harness, a novel automatic harness evolution framework built around constrained evolution, proactive feedback, and component-wise optimization.
- Score: 17.229636050055266
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Harness design plays a critical role in agent performance by shaping how large language models (LLMs) perceive, reason over, and act within executable environments. Recent work has proposed automatic harness evolution, which iteratively improves the harness from agent--environment interactions. However, existing methods often overfit to the evolution tasks, rely exclusively on trajectory-derived signals, and optimize harness components jointly, causing interference across components. We propose HarnessCompass, a novel automatic harness evolution framework built around constrained evolution, proactive feedback, and component-wise optimization. HarnessCompass first enforces global constraints on evolution, restricting modifications to task-agnostic harness changes that generalize beyond the evolution tasks. It then augments trajectory-derived evidence with proactive first-person feedback from the agent about harness usage, yielding richer signals for evolution. Finally, it decouples the optimization of different harness components before consolidating them into a unified harness, reducing cross-component interference while preserving component synergy. On SWE-bench Verified with GPT-5.4, HarnessCompass improves Pass@1 from 54\% to 66\% in only 5 evolution iterations, outperforming AHE in both effectiveness and evolution efficiency. In addition, the evolved harness transfers effectively to held-out tasks and other models, demonstrating substantially stronger generalization than prior automatic harness evolution methods.
Related papers
- Self-Evolving Memory for Generative Recommendation [100.15207958893029]
Generative recommendation has emerged as a promising end-to-end paradigm for personalized recommendation.<n>Existing evolving strategies, such as continual retraining and distillation-based adaptation, directly update the shared model parameters.<n>We propose a self-evolving memory paradigm for generative recommendation, aiming to enable effective evolution across heterogeneous behavioral patterns.
arXiv Detail & Related papers (2026-09-14T14:00:38Z) - ModularRSI: Modular and Generalizable Recursive Harness Self-Improvement [46.36989489677189]
We propose a benchmark-disjoint, contrastive, and modular framework for generalizable harness evolution.<n> ModularRSI contrasts successful and failed trajectories for the same task and aggregates evidence across tasks to identify recurring behavioral deficiencies.<n>It decomposes the evolvable harness into five functional modules: Agent Loop, Tool Use, Observation Management, Context Management, and Task Completion Detection.
arXiv Detail & Related papers (2026-09-14T00:10:45Z) - Co-Evolution in Agentic Systems: Toward Self-Directed Evolution Beyond Human Design [55.30329388138526]
This survey focuses on co-evolution in agentic systems, a multi-component form of self-evolution in which multiple agents and their environment impose adaptive pressure on one another.<n>To organize existing papers, we propose a progressive three-stage taxonomy that traces how the system gradually sheds human-engineered constraints.
arXiv Detail & Related papers (2026-08-10T23:10:40Z) - SHE: Trajectory-driven Safety Harness Evolution for LLM Agents [63.74895225415523]
Existing safety mechanisms treat the harness as a fixed deployment artifact, limiting their ability to evolve with emerging risks.<n>We propose Safety Harness Evolution, a framework that learns evolving safe boundaries from rollout trajectories.<n>Experiments on Agent-SafetyBench demonstrate that SHE effectively enhances safety through harness evolution, achieving a 3.1x ASR reduction compared with static SafeHarness.
arXiv Detail & Related papers (2026-08-10T17:35:08Z) - Rethinking the Evaluation of Harness Evolution for Agents [83.07258924910069]
Existing harness evolution methods use unit test cases to search for harness configurations and then report final performance on the same public benchmark.<n>This protocol raises two fundamental concerns.<n>As in agentic test-time scaling, it should therefore be compared with simple task-level search baselines under matched feedback and inference budgets.
arXiv Detail & Related papers (2026-07-14T00:18:42Z) - HarnessX: A Composable, Adaptive, and Evolvable Agent Harness Foundry [35.87794858139959]
HarnessX is a foundry for composable, adaptive, and evolvable agent harnesses.<n>It assembles typed harness primitives via a substitution algebra, adapts them through AEGIS.<n>It closes the harness-model loop by turning trajectories into both harness updates and model training signal.
arXiv Detail & Related papers (2026-06-12T08:27:11Z) - Harnessing Agentic Evolution [32.567492765277855]
Agentic evolution is a powerful paradigm for improving programs, iteration, and scientific solutions by iteratively generating candidates, evaluating them, and using feedback to guide future search.<n>Existing methods are typically instantiated either as fixed hand-designed procedures that are modular but rigid, or as general-purpose agents that flexibly integrate feedback but can drift in long-horizon evolution.<n>We introduce AEvo, a harnessed meta-editing framework in which a meta-agent observes this state and acts not by directly proposing the next candidate, but by editing the procedure or agent context that controls future evolution.
arXiv Detail & Related papers (2026-05-13T17:45:16Z) - Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses [57.20181537213498]
Agentic Harness Engineering (AHE) is a closed loop that automates harness engineering.<n>AHE addresses challenges through three matched observability pillars.<n>Ten AHE lift pass@1 on Terminal-Bench 2 from 69.7% to 77.0%, surpassing the human-designed harness Codex-CLI.<n>On SWE-bench-verified it tops aggregate success at 12% fewer tokens than the seed, and on Terminal-Bench 2 it yields +5.1 to +10.1pp cross-family gains.
arXiv Detail & Related papers (2026-04-28T16:55:02Z) - Co-Evolutionary Multi-Modal Alignment via Structured Adversarial Evolution [8.28457465201508]
We move beyond static adversarial supervision and introduce co-evolutionary alignment with evolving attacks.<n>We introduce an Evolutionary Attacker that decomposes adversarial prompts into method templates and harmful intents.<n>The Adaptive Defender is iteratively updated on the synthesized hard negatives, forming a closed-loop process that adapts alignment to evolving attacks.
arXiv Detail & Related papers (2026-03-02T12:10:46Z) - A Simple yet Effective DDG Predictor is An Unsupervised Antibody Optimizer and Explainer [53.85265022754878]
We propose a lightweight DDG predictor (Light-DDG) for fast mutation screening.<n>We also release a large-scale dataset containing millions of mutation data for pre-training Light-DDG.<n>For the target antibody, we propose a novel Mutation Explainer to learn mutation preferences.
arXiv Detail & Related papers (2025-02-10T09:26:57Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.