Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading

• URL: http://arxiv.org/abs/2607.08964v2
• Date: Mon, 13 Jul 2026 06:00:47 GMT
• Title: Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading
• Abstract summary: Long-Horizon-Terminal-Bench is a terminal benchmark of 46 long-horizon tasks spanning nine categories.<n>Each task follows a Terminal-Bench-style setup with a reference solution or simulation engine, but is further graded into fine-grained sub-tasks.<n>We evaluate 15 frontier models and find that agents consume on average 9.9M tokens per task, with roughly 231 episodes and 85.3 minutes of execution time per run.
• Score: 46.911171278867585
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI agents have become capable of autonomously completing short, well-specified tasks. However, existing terminal benchmarks largely focus on simple problems that finish within minutes and are evaluated only by their final outcome. This setup overlooks intermediate progress and partial solutions, yielding sparse reward signals and an incomplete picture of agent capability. We introduce Long-Horizon-Terminal-Bench, a terminal benchmark of 46 long-horizon tasks spanning nine categories, including experiment reproduction, software engineering, multimodal analysis, interactive games, and scientific computing. Each task follows a Terminal-Bench-style setup with a reference solution or simulation engine, but is further decomposed into fine-grained graded subtasks. This design enables dense intermediate rewards and partial credit, allowing evaluation to capture not only whether an agent reaches the final goal, but also how far it progresses on open-ended workflows. Tasks in Long-Horizon-Terminal-Bench typically require hundreds of episodes and minutes to hours of execution, stressing long-horizon planning, long-context management, and iterative debugging rather than one-shot problem solving. We evaluate 15 frontier models and find that agents consume on average 9.9M tokens per task, with roughly 231 episodes and 85.3 minutes of execution time per run, making Long-Horizon-Terminal-Bench more demanding than prior terminal-based benchmarks. Even the strongest tested model achieves 15.2% pass@1 at a partial-reward threshold of 0.95 and 10.9% at a perfect-reward threshold of 1.0, while the mean pass rate across models is 4.3% and 1.7% under the two thresholds, respectively. These results reveal headroom for improvement. We further analyze failure modes and error patterns, and release Long-Horizon-Terminal-Bench to support future progress on long-horizon terminal agents.

Related papers

• Recursive Synthesis for Long-Horizon Terminal Tasks [50.39352383646813]
  Recursive Synthetic Terminal Tasks (RST) is a framework for constructing long-horizon terminal-agent tasks at scale.<n>RST produces 37,484 synthesized terminal-agent tasks at roughly $0.05 per task.<n>Fine-tuning on trajectories improves Qwen3.5-27B and Qwen3.5-122B-A10B by up to 10 points on Terminal-Bench 2, Terminal-Bench Hard, and Long-Horizon Terminal Bench.
  arXiv  Detail & Related papers  (2026-08-05T23:24:26Z)
• LongHorizon-Harness: Advancing Long-Horizon Agents for Real-World Tasks [22.929016343684395]
  LongHorizon-Harness maintains the task state explicitly outside execution and updates it only with facts independently verified from the environment.<n>A lightweight AgentAdapter supports interchangeable model and harness backends without modifying their native agent loops.
  arXiv  Detail & Related papers  (2026-08-03T09:32:21Z)
• Desktop-Delta Bench: Do Computer-Use Models Understand Desktop GUI Transitions? [3.9561795512466413]
  Current benchmarks primarily measure end-task success or single-frame grounding.<n>This is difficult because inference, remote input, app rendering, and screenshot capture are asynchronous.<n>We introduce Desktop-Delta Bench (DDB), an offline step-level benchmark with 2,013 human-verified instances from novel, multi-app Linux trajectories.
  arXiv  Detail & Related papers  (2026-07-28T17:49:51Z)
• OSWorld2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks [98.07146747892239]
  OSWorld 2.0 is a benchmark of 108 long-horizon computer-use across everyday and professional tasks.<n>Each task represents a realistic end-to-end workflow that takes human users a median of about 1.6 hours to complete.
  arXiv  Detail & Related papers  (2026-06-28T17:59:17Z)
• SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work? [11.093716663660203]
  SWE-Marathon is a benchmark of 20 long-horizon tasks spanning software engineering and adjacent technical domains.<n>Each task consists of a unique executable environment, a human-written reference solution, and a multi-layer verification suite.<n>Logged agent attempts average 27.2M total tokens, making SWE-Marathon substantially longer-horizon than existing SWE and command-line agent benchmarks.
  arXiv  Detail & Related papers  (2026-06-05T00:39:44Z)
• Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks [67.44219836008348]
  We introduce Odysseys: a benchmark of 200 long-horizon web tasks derived from real world browsing sessions evaluated on the live Internet.<n>We find that binary pass/fail evaluation is inadequate for long-horizon settings and introduce a rubric-based evaluation, annotating each Odysseys task with an average of 6.1 graded rubrics.<n>We find that the strongest models achieve a success rate of 44.5%, which leaves substantial room for future improvements.
  arXiv  Detail & Related papers  (2026-04-27T20:05:41Z)
• Scaling Test-Time Compute for Agentic Coding [126.72747643609274]
  We propose a test-time scaling framework for agentic coding based on compact representations of rollout trajectories.<n>Our framework converts each rollout into a structured summary that preserves its salient hypotheses, progress, and failure modes.<n>Our method consistently improves the performance of frontier coding agents across SWE-Bench Verified and Terminal-Bench v2.0.
  arXiv  Detail & Related papers  (2026-04-16T17:39:33Z)
• LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces [65.11019654023978]
  LongCLI-Bench is a benchmark designed to evaluate agentic capabilities across long-horizon, realistic tasks.<n>We curated 20 high-quality, long-horizon tasks from over 1,000 computer science assignments and real-world tasks.<n>Experiments reveal that even state-of-the-art agents achieve pass rates below 20% in LongCLI-Bench.
  arXiv  Detail & Related papers  (2026-02-15T23:12:57Z)
• Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces [126.23612941699565]
  Terminal-Bench 2.0 is a benchmark composed of 89 tasks in computer terminal environments inspired by problems from real world.<n>We show that frontier models and agents score less than 65% on the benchmark.<n>We publish the dataset and evaluation harness to assist developers and researchers in future work at https://www.tbench.ai/.
  arXiv  Detail & Related papers  (2026-01-17T01:29:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.