Learning to Control LLM Agent Harnesses with Offline Reinforcement Learning

• URL: http://arxiv.org/abs/2607.05458v1
• Date: Sun, 05 Jul 2026 22:11:18 GMT
• Title: Learning to Control LLM Agent Harnesses with Offline Reinforcement Learning
• Abstract summary: Large language model (LLM) agents are usually improved by changing prompts, calibrated models, or hand-written, while the execution harness around the model is treated as fixed infrastructure.<n>We formalize harness operation as a finite-horizon Harness MDP, where a lightweight controller selects structural execution actions while the LLM executor remains frozen.
• Score: 0.05371337604556311
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language model (LLM) agents are usually improved by changing prompts, models, or hand-written workflows, while the execution harness around the model is treated as fixed infrastructure. We argue that this harness is itself a learnable control layer. We formalize harness operation as a finite-horizon Harness MDP, where a lightweight controller selects structural execution actions while the LLM executor remains frozen. The controller is trained from offline rollouts using advantage-weighted regression with only terminal task-rubric rewards. We also separate final task quality from a post-hoc Harness Maturity Score, which measures whether the harness follows reliable execution patterns rather than only whether the final answer is correct. This separation gives a finite-buffer view of harness learning: final-quality gains require high-return support in the offline buffer, while process behavior can shift whenever it aligns with advantage-weighted actions. Across six controlled domains and two public-benchmark adapters, the learned controller consistently improves verification behavior and selectively improves final task quality, with the largest gains on adapted tau-bench retail, adapted AgentBench DB-Bench, and coding with a calibrated structural verifier. Ablations against behavior cloning and Forced CHECK show that the gains are not explained by imitation or by simply adding checks. These results identify harness control as a learnable layer for frozen LLM agents, while showing that offline support limits when better process control becomes better final answers.

Related papers

• TROVE: Adaptive Agent Skill Orchestration via Trace-Grounded Route Validation and Editing [14.424308190797683]
  We propose Trace-grounded Route Orchestration via Validation and Editing (TROVE)<n> Offline, TROVE distills evaluated workflow-search traces into atomic and composite skills and an outcome-conditioned transition graph.<n>Online, it treats a planned route as provisional, after committing one top-level skill, the controller retains a valid continuation, inserts a trace-supported local response, or replaces only the invalid suffix.
  arXiv  Detail & Related papers  (2026-09-04T11:38:13Z)
• Control-Diverse Reinforcement Fine-Tuning: Decoupling the Shared Control Bottleneck of RL Post-Training [5.633817116718103]
  Reinforcement learning post-training unlocks complex reasoning in LLMs.<n> benchmark scores reveal only whether a model improved, not what changed inside it, nor how it splits finite capability across tasks.<n>We challenge this activation-centered account by separating activation from control.
  arXiv  Detail & Related papers  (2026-08-08T16:44:47Z)
• TTHE: Test-Time Harness Evolution [50.26245555541721]
  Existing approaches optimize such harnesses before deployment, searching training or development data for a fixed agent workflow that is then frozen at test time.<n>We ask whether the harness can instead be optimized during evaluation itself, using only the unlabeled execution traces the agent produces on the test inputs.<n>During evaluation, TTHE maintains a population of candidate harnesses and refines them through an agentic proposer that reasons over their execution traces.<n>A judge then commits an improved harness from execution-derived proxy signals, and the selected program persists to govern subsequent inputs.
  arXiv  Detail & Related papers  (2026-07-09T05:53:39Z)
• LLM-as-a-Verifier: A General-Purpose Verification Framework [74.40111651545979]
  We introduce LLM-as-a-Verifier, a general-purpose verification framework.<n>It provides fine-grained feedback for agentic tasks without requiring additional training.<n>It achieves state-of-the-art performance on several benchmarks.
  arXiv  Detail & Related papers  (2026-07-06T17:59:35Z)
• Benchmarking Code Improvement with Progressive, Adaptive, and Interactive Feedback [13.976149104483449]
  PAIR-Bench is a progressive benchmark for evaluating large language models (LLMs)<n>It transforms an incorrect or incomplete program into a more correct one through feedback-guided refinement.<n>It measures whether a model repairs targeted failures, generalizes beyond the hint, preserves already-correct behavior, and how much assistance it requires.
  arXiv  Detail & Related papers  (2026-07-01T18:20:27Z)
• AI Training Manager: Bounded Closed-Loop Control of Adaptive Training Recipes [0.0]
  Manager reads structured telemetry snapshots from an active run, audits a constrained action space, and returns validated updates to training parameters.<n>On TinyStories, the manager detects and corrects overfitting, achieving a validation loss 60% lower than the baseline.<n>In a robotic manipulation reinforcement-learning task, we use the same bounded decision interface in an episodic closed-loop setting, where manager updates are applied at evaluation or checkpoint boundaries.
  arXiv  Detail & Related papers  (2026-06-29T07:07:33Z)
• From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws [12.757205456771295]
  This paper proposes HarnessFix, a trace-guided framework for diagnosing agent failures and repairing agent harnesses.<n> HarnessFix compiles raw execution traces and harness code into a Harness-aware Trace Intermediate Representation.<n>It attributes failures to responsible trajectory steps and harness layers, consolidates recurring diagnoses into actionable flaw records, and maps them to scoped repair operators.
  arXiv  Detail & Related papers  (2026-06-04T15:58:30Z)
• PYTHALAB-MERA: Validation-Grounded Memory, Retrieval, and Acceptance Control for Frozen-LLM Coding Agents [0.0]
  We introduce PYTHALAB-MERA, a lightweight external controller for local validation-conditioned code generation.<n>We evaluate the implementation as a local CLI artifact on reinforcement-learning coding tasks with strict validation gates.
  arXiv  Detail & Related papers  (2026-05-08T20:39:32Z)
• Training LLMs for Multi-Step Tool Orchestration with Constrained Data Synthesis and Graduated Rewards [76.49428173793386]
  LLMs must invoke multiple dependent APIs in the correct order while propagating intermediate outputs.<n>Training models to handle such faces two obstacles: existing environments focus on simple per-turn function calls with simulated data, and binary rewards.<n>We present a framework addressing both challenges. First, we construct a reinforcement learning environment backed by a large-scale cache of real API responses, enabling a data synthesis pipeline that samples valid multi-step orchestration traces.<n>Second, we propose a graduated reward design that decomposes correctness into atomic validity and orchestration.
  arXiv  Detail & Related papers  (2026-03-25T18:31:39Z)
• GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RL [64.8155693023222]
  Open-source native GUI agents still lag behind closed-source systems on long-horizon navigation tasks.<n>This gap stems from a shortage of high-quality, action-aligned reasoning data.<n>We present GUI-Libra, a tailored training recipe that addresses these challenges.
  arXiv  Detail & Related papers  (2026-02-25T18:34:57Z)
• Confounding Robust Continuous Control via Automatic Reward Shaping [48.93769483870838]
  We propose to automatically learn a reward shaping function for continuous control problems from offline datasets.<n>Our method builds upon the recently proposed causal Bellman equation to learn a tight upper bound on the optimal state values.<n>Our work marks a solid first step towards confounding robust continuous control from a causal perspective.
  arXiv  Detail & Related papers  (2026-02-10T21:23:12Z)
• Attribute Controlled Fine-tuning for Large Language Models: A Case Study on Detoxification [76.14641982122696]
  We propose a constraint learning schema for fine-tuning Large Language Models (LLMs) with attribute control. We show that our approach leads to an LLM that produces fewer inappropriate responses while achieving competitive performance on benchmarks and a toxicity detection task.
  arXiv  Detail & Related papers  (2024-10-07T23:38:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.