MemoHarness: Agent Harnesses That Learn from Experience

• URL: http://arxiv.org/abs/2607.14159v1
• Date: Tue, 14 Jul 2026 21:22:18 GMT
• Title: MemoHarness: Agent Harnesses That Learn from Experience
• Abstract summary: We introduce MemoHarness, an adaptive harness optimization framework that learns from its own executions.<n>MemoHarness decomposes the harness into six editable control dimensions, stores per-case diagnoses and distilled global patterns in a dual-layer experience bank.<n>In our evaluation across shell-agent, code-generation, and analytical-reasoning benchmarks, MemoHarness improves over the fixed harnesses we compare against and shows selective transfer to unseen suites and base models.
• Score: 46.167787606315585
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: An agent harness is the external control layer that turns a base LLM into an executable agent by managing context, tools, orchestration, memory, decoding, and output handling. While harness design strongly affects agent behavior, most automatic improvement methods optimize narrower artifacts such as prompts, pipelines, or workflows, and deployed agents usually reuse a single global harness for all cases. We introduce MemoHarness, an adaptive harness optimization framework that learns from its own executions. MemoHarness decomposes the harness into six editable control dimensions, stores per-case diagnoses and distilled global patterns in a dual-layer experience bank, and adapts the learned harness to each test case using retrieved experience without test-time labels, feedback, or additional search. In our evaluation across shell-agent, code-generation, and analytical-reasoning benchmarks, MemoHarness improves over the fixed harnesses we compare against and shows selective transfer to unseen suites and base models. Its additional context can also remain cost-competitive when much of the retrieved experience is cacheable. These results provide evidence that execution experience is a practical substrate for building agent harnesses that are more adaptive than a single static configuration, while leaving broader claims about statistical robustness and component attribution to future work.

Related papers

• HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness? [61.02699867449589]
  We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to run infrastructure.<n>In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system.<n>In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance.
  arXiv  Detail & Related papers  (2026-09-01T15:45:33Z)
• S3Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement? [66.17176838165994]
  Large language models (LLMs) increasingly interact with external environments and accumulate substantial behavioral experience.<n>It remains unclear whether an agent can actively test its behavior, judge the resulting experience, and use that experience to improve future decisions.<n>We introduce textbfStextsuperscript3Gym, an interactive benchmark for evaluating LLM self-improvement.
  arXiv  Detail & Related papers  (2026-08-31T17:05:41Z)
• Verify Smarter, Evolve Further: Efficient Harness Evolution through Behavior-Aware Verification [30.941867057615223]
  Existing propose-and-verify methods score every candidate on a fixed task set.<n>We introduce HarnessLens, a budget-aware framework for automated harness evolution.
  arXiv  Detail & Related papers  (2026-08-27T16:12:23Z)
• HarnessOpt-Bench: Evaluating LLMs at Harness Optimization [3.846854817984248]
  We introduce a benchmark for end-to-end harness optimization under expensive evaluation regimes.<n>A trusted execution environment enforces the evaluation boundary, target-agent resource use, and preserves candidate versions for audit.<n>Experiment results show that models separate more than the coding harnesses they act through, native harnesses are not consistently superior, and gains vary substantially across tasks.
  arXiv  Detail & Related papers  (2026-08-06T17:21:05Z)
• From Profiling to Synthesis: Benchmarking Implicit Behavioral Alignment in Personalized LLM Agents [56.187590188698955]
  Large Language Models have enabled increasingly capable autonomous agents, yet personalization remains critical for making such agents practically useful.<n>Recent benchmarks have begun evaluating personalization in agents, but they largely rely on static preference snapshots, fixed interaction logs, or question answering over predefined user profiles.<n>We introduce IBA-Bench, a benchmark for implicit behavioral alignment constructed from longitudinal interaction histories.<n>We propose IBA-Agent, an agent framework that reconciles conflicting priorities through broad retrieval and trajectory-level alignment.
  arXiv  Detail & Related papers  (2026-08-03T12:52:04Z)
• TTHE: Test-Time Harness Evolution [50.26245555541721]
  Existing approaches optimize such harnesses before deployment, searching training or development data for a fixed agent workflow that is then frozen at test time.<n>We ask whether the harness can instead be optimized during evaluation itself, using only the unlabeled execution traces the agent produces on the test inputs.<n>During evaluation, TTHE maintains a population of candidate harnesses and refines them through an agentic proposer that reasons over their execution traces.<n>A judge then commits an improved harness from execution-derived proxy signals, and the selected program persists to govern subsequent inputs.
  arXiv  Detail & Related papers  (2026-07-09T05:53:39Z)
• Learning to Control LLM Agent Harnesses with Offline Reinforcement Learning [0.05371337604556311]
  Large language model (LLM) agents are usually improved by changing prompts, calibrated models, or hand-written, while the execution harness around the model is treated as fixed infrastructure.<n>We formalize harness operation as a finite-horizon Harness MDP, where a lightweight controller selects structural execution actions while the LLM executor remains frozen.
  arXiv  Detail & Related papers  (2026-07-05T22:11:18Z)
• From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws [12.757205456771295]
  This paper proposes HarnessFix, a trace-guided framework for diagnosing agent failures and repairing agent harnesses.<n> HarnessFix compiles raw execution traces and harness code into a Harness-aware Trace Intermediate Representation.<n>It attributes failures to responsible trajectory steps and harness layers, consolidates recurring diagnoses into actionable flaw records, and maps them to scoped repair operators.
  arXiv  Detail & Related papers  (2026-06-04T15:58:30Z)
• Evolving Agents in the Dark: Retrospective Harness Optimization via Self-Preference [51.55077364626896]
  We introduce Retrospective Harness Optimization (RHO), a self-supervised method that optimize agent harness using only past trajectories.<n>RHO selects a diverse coreset of challenging tasks from past trajectories and re-solves them in parallel.<n>A single optimization round improves the pass rate on SWE-Bench Pro from 59% to 78% without any external grading.
  arXiv  Detail & Related papers  (2026-06-04T09:26:00Z)
• Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows [18.6534256358905]
  We introduce Harness-Bench, a diagnostic benchmark for evaluating configuration-level harness effects in realistic agent systems.<n>The benchmark contains 106 sandboxed offline tasks constructed from practical agent-use patterns.<n>Across 5,194 execution trajectories, we observe substantial variation in completion, process quality, efficiency, and failure behavior.
  arXiv  Detail & Related papers  (2026-05-27T03:47:35Z)
• HARBOR: Automated Harness Optimization [2.7324157162184157]
  We argue that harness design is a first-class machine-learning problem.<n>We give a reference solver, HARBOR, built from a block-additive SAAS surrogate, multi-fidelity cost-aware acquisition, and TuRBO trust regions.
  arXiv  Detail & Related papers  (2026-04-22T13:45:12Z)
• How Memory Management Impacts LLM Agents: An Empirical Study of Experience-Following Behavior [65.70584076918679]
  Memory is a critical component in large language model (LLM)-based agents.<n>This paper studies how memory management choices impact the LLM agents' behavior, especially their long-term performance.
  arXiv  Detail & Related papers  (2025-05-21T22:35:01Z)
• On the Role of Feedback in Test-Time Scaling of Agentic AI Workflows [71.92083784393418]
  Agentic AI (systems that autonomously plan and act) are becoming widespread, yet their task success rate on complex tasks remains low.<n>Inference-time alignment relies on three components: sampling, evaluation, and feedback.<n>We introduce Iterative Agent Decoding (IAD), a procedure that repeatedly inserts feedback extracted from different forms of critiques.
  arXiv  Detail & Related papers  (2025-04-02T17:40:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.