LemonHarness Technical Report

• URL: http://arxiv.org/abs/2606.24311v1
• Date: Tue, 23 Jun 2026 08:44:00 GMT
• Title: LemonHarness Technical Report
• Abstract summary: LemonHarness is an integrated execution framework for long-horizon agents.<n>It constrains state-changing operations within a clearly defined workspace.<n>It brings model invocation, tool execution, and rule knowledge within a single controlled boundary.
• Score: 40.68992799867636
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As large language model (LLM) agents are applied to longer tasks, they increasingly modify workspace state across multiple rounds of iteration. However, agents typically observe only tool outputs and log fragments, while the actual state changes occur in the file system. Without explicit workspace boundaries, state-changing operations such as file writes and temporary artifact generation may scatter changes across paths. Over time, these weakly constrained changes accumulate, making states such as modified files difficult to track. This paper presents LemonHarness, an integrated execution framework for long-horizon agents. LemonHarness establishes an explicit execution boundary by constraining state-changing operations within a clearly defined workspace and bringing model invocation, tool execution, and rule knowledge within a single controlled boundary. State-changing operations, including file writes, dependency installation, and temporary artifact creation, are executed through structured tool interfaces, with execution feedback recorded as observations available to subsequent model decisions. The system also introduces a reusable rule knowledge base, which turns recurring execution rules and acceptance criteria into runtime knowledge. LemonHarness further adds a time-aware execution mechanism that exposes elapsed and remaining budget to the model, so it can rebalance exploration, implementation, and validation effort as time pressure shifts and avoid timeouts from long waits or excessive verification. On Terminal-Bench 2.0, LemonHarness_GPT-5.3-CodeX reached 84.49% accuracy over 445 trials; pairing the same framework with the stronger GPT-5.5 backbone raised the average accuracy to 86.52% across five jobs. The results suggest that a unified runtime boundary, callable rule knowledge, and time-aware execution can improve the stability of long-horizon agent execution.

Related papers

• LocalLSTC: A Long Short-Term Control Architecture for Locally Deployed GUI Agents [13.261859864631063]
  We introduce LocalLSTC, a training-free architecture that organizes control by temporal scope, maintaining persistent cross-step state.<n>Long-Term Control maintains the active subgoal, subgoal-aligned evidence, and runtime feedback across interactions, while Short-Term Execution realizes bounded commitments for the current step.<n>With Qwen3.6-27B, LocalLSTC reaches 64.7% SR-100 on OSWorld and 65.3% on WindowsAgentArena, outperforming the strongest prior local results on both benchmarks.
  arXiv  Detail & Related papers  (2026-08-26T13:20:27Z)
• Verified Tool Calls Improve LLM Agent Reliability Under Non-Atomic Failures [0.0]
  Real-world systems exhibit non-atomic behaviors such as timeouts after dispatch, delayed visibility, and partial state updates.<n>These mismatches lead to reliability issues including duplicate actions, task success, and unnecessary tool executions.<n>A lightweight, verification-aware tool wrapper is introduced that augments tool calls with postcondition verification, verify-before-retry logic, and idempotency keys.
  arXiv  Detail & Related papers  (2026-07-31T16:16:14Z)
• PhyAgentOS: A Self-Evolving Operating System for Embodied Agents with Decoupled Cognitive Planning and Physical Execution [49.776611937968]
  We present PhyAgentOS, a foundation delivering scheduling, verification, memory, benchmarking, and safety as system-level services.<n>Its Session-Centered treats a session, not an action, as the minimum unit of scheduling, compatibility, supervised execution, evidence collection, and acceptance.<n>SessionVerifier distinguishes execution termination from semantic task completion via evidence-grounded verdicts of success, failure, or replan.<n> Benchmarking reuses the deployment session and verification path, so results trace to real execution.
  arXiv  Detail & Related papers  (2026-07-18T04:46:53Z)
• Constraint Tax in Open-Weight LLMs: An Empirical Study of Tool Calling Suppression Under Structured Output Constraints [37.92327096151557]
  This paper reports reproducible phenomenon observed in a production Agent system.<n>Multiple open-weight models cease invoking tools despite maintaining high schema compliance.<n>We propose Transparent Two-Execution, an inference-time strategy that decouples tool execution from schema-constrained response generation.
  arXiv  Detail & Related papers  (2026-06-24T09:14:18Z)
• Cordon: Semantic Transactions for Tool-Using LLM Agents [11.567293065381918]
  This paper introduces Cordon, a transactional runtime system for staging and validating agent effects before commit.<n>It reduces irreversible-effect failures while preserving benign task completion with modest approval and latency overhead.
  arXiv  Detail & Related papers  (2026-06-16T06:21:14Z)
• ClawForge: Generating Executable Interactive Benchmarks for Command-Line Agents [59.626170560327274]
  We present textbfClawForge, a generator-backed benchmark framework for executable command-line categories under state conflict.<n>We instantiate this framework as the ClawForge-Bench (17 scenarios, 6 ability categories)
  arXiv  Detail & Related papers  (2026-05-13T21:34:08Z)
• WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation [88.10947115397971]
  This work presents WildClawBench, a native-runtime benchmark of 60 human-authored, bilingual, multimodal tasks spanning six thematic categories.<n>Each task averages roughly 8 minutes of wall-clock time and over 20 tool calls.<n> Grading is hybrid, combining deterministic rule-based checks, environment-state auditing of side effects, and an LLM/VLM judge for semantic verification.
  arXiv  Detail & Related papers  (2026-05-11T17:49:43Z)
• Claw-Eval-Live: A Live Agent Benchmark for Evolving Real-World Workflows [67.92316850084575]
  We introduce Claw-Eval-Live, a live benchmark for workflow agents.<n>Each release is constructed from public workflow-demand signals.<n>Claw-Eval-Live records execution traces, audit logs, service state, and post-run workspace artifacts.
  arXiv  Detail & Related papers  (2026-04-30T17:23:19Z)
• Dynamic analysis enhances issue resolution [53.50448142467294]
  DAIRA (Dynamic Analysis-enhanced Issue Resolution Agent) is an automated repair framework that embeds dynamic analysis into the agent's reasoning cycle.<n> Driven by a Test Tracing-Driven methodology, DAIRA utilizes lightweight monitors to extract critical runtime data.<n>When powered by Gemini 3 Flash Preview, DAIRA establishes a new state-of-the-art (SOTA) performance, achieving a 79.4% resolution rate on the SWE-bench Verified dataset.
  arXiv  Detail & Related papers  (2026-03-23T14:48:54Z)
• Agents Learn Their Runtime: Interpreter Persistence as Training-Time Semantics [4.774525456207306]
  Tool-augmented LLMs are increasingly deployed as agents that interleave natural-language reasoning with executable Python actions.<n>We ask whether interpreter persistence is merely a runtime scaffold, or a property of the training data that shapes how agents learn to use the interpreter.<n>We introduce Opaque Knapsack, a procedurally generated family of partially observable optimization tasks designed to prevent one-shot solutions.
  arXiv  Detail & Related papers  (2026-03-01T18:08:02Z)
• Can Large Language Models Detect Real-World Android Software Compliance Violations? [3.7278558081099544]
  Current models struggle to detect compliance violations in Android applications across diverse legal frameworks.<n>We propose emphCompliBench, a novel evaluation framework for assessing LLMs' ability to detect compliance violations.<n>Experiments with six models, including GPT-4O and Claude-3.5, show emphCompliBench improves compliance detection.
  arXiv  Detail & Related papers  (2025-11-01T16:58:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.