TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agents

• URL: http://arxiv.org/abs/2606.28480v1
• Date: Fri, 26 Jun 2026 17:59:51 GMT
• Title: TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agents
• Abstract summary: We introduce TUA-Bench, a general-purpose benchmark for terminal-use agents.<n> TUA-Bench includes 120 real-world tasks across five task families.<n>We find that the strongest frontier agent achieves 65.8% overall performance.
• Score: 37.00898251261264
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: As large language models and harness frameworks continue to advance, agents operating in terminals are increasingly capable of performing a broader range of general computer-use tasks beyond coding. However, existing benchmarks do not adequately evaluate general-purpose terminal computer-use agents (TUAs): general computer-use benchmarks primarily target graphical user interfaces (GUIs), whereas terminal-based benchmarks largely emphasize technical and programming-centric workflows historically native to the shell. We introduce TUA-Bench, a general-purpose benchmark for terminal-use agents. TUA-Bench includes 120 real-world tasks across five task families, covering routine digital activities-including document editing, email management, and live-web information seeking-as well as scientific and engineering workflows co-designed with PhD-level domain experts that require specialized software. This breadth distinguishes TUA-Bench from prior shell-focused or domain-specific benchmarks. Each task is manually designed, runs in a real terminal with a deterministic setup script, and is evaluated by an execution-based scoring protocol. We find that the strongest frontier agent, Claude Code with Claude Opus 4.8 max reasoning effort, achieves 65.8% overall performance, with substantial gaps across both tracks. By providing a broad and realistic evaluation of terminal-use capabilities, TUA-Bench aims to accelerate the transition from narrow, task-specific assistants to general-purpose agents capable of operating reliably across diverse digital environments.

Related papers

• OSWorld2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks [98.07146747892239]
  OSWorld 2.0 is a benchmark of 108 long-horizon computer-use across everyday and professional tasks.<n>Each task represents a realistic end-to-end workflow that takes human users a median of about 1.6 hours to complete.
  arXiv  Detail & Related papers  (2026-06-28T17:59:17Z)
• WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces [27.11445886768225]
  We introduce WeaveBench, a long-horizon hybrid-interface benchmark with 114 tasks across 8 real-world work domains.<n>We evaluate these tasks on a real Ubuntu desktop inside deployed CLI-agent runtimes, augmented with a minimal desktop-control plugin.<n>Across frontier model-runtime pairings, the best PassRate reaches only 41.2%, showing the benchmark remains far from saturated.
  arXiv  Detail & Related papers  (2026-06-08T12:39:23Z)
• TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks [23.863417507169697]
  TerminalWorld is a scalable data engine that automatically reverse-engineers high-fidelity evaluation tasks from "in-the-wild" terminal recordings.<n>The engine yields a full benchmark of 1,530 validated tasks, spanning 18 real-world categories.<n>TerminalWorld captures real-world terminal capabilities distinct from existing expert-Verified benchmarks.
  arXiv  Detail & Related papers  (2026-05-21T14:24:43Z)
• WindowsWorld: A Process-Centric Benchmark of Autonomous GUI Agents in Professional Cross-Application Environments [34.06270058972]
  WindowsWorld is designed to assess GUI Agents on complex multi-step tasks that mirror real-world professional activities.<n>Our methodology uses a multi-agent framework steered by 16 occupations to generate four difficulty-level tasks.<n>The resulting benchmark contains 181 tasks with an average of 5.0 sub-goals across 17 common desktop applications.
  arXiv  Detail & Related papers  (2026-04-30T12:13:27Z)
• Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces [126.23612941699565]
  Terminal-Bench 2.0 is a benchmark composed of 89 tasks in computer terminal environments inspired by problems from real world.<n>We show that frontier models and agents score less than 65% on the benchmark.<n>We publish the dataset and evaluation harness to assist developers and researchers in future work at https://www.tbench.ai/.
  arXiv  Detail & Related papers  (2026-01-17T01:29:30Z)
• DeepAgent: A General Reasoning Agent with Scalable Toolsets [111.6384541877723]
  DeepAgent is an end-to-end deep reasoning agent that performs autonomous thinking, tool discovery, and action execution.<n>To address the challenges of long-horizon interactions, we introduce an autonomous memory folding mechanism that compresses past interactions into structured episodic, working, and tool memories.<n>We develop an end-to-end reinforcement learning strategy, namely ToolPO, that leverages LLM-simulated APIs and applies tool-call advantage attribution to assign fine-grained credit to the tool invocation tokens.
  arXiv  Detail & Related papers  (2025-10-24T16:24:01Z)
• Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis [57.371814877372515]
  Graphical user interface (GUI) grounding remains a critical bottleneck in computer use agent development.<n>We introduce OSWorld-G, a comprehensive benchmark comprising 564 finely annotated samples across diverse task types.<n>We synthesize and release the largest computer use grounding dataset Jedi, which contains 4 million examples.
  arXiv  Detail & Related papers  (2025-05-19T15:09:23Z)
• Programming with Pixels: Can Computer-Use Agents do Software Engineering? [24.011063667060792]
  $textttProgramming with Pixels$ (PwP) is the first comprehensive computer-use environment for software engineering.<n>PwP establishes software engineering as a natural domain for benchmarking whether generalist computer-use agents can reach specialist-level performance.
  arXiv  Detail & Related papers  (2025-02-24T18:41:33Z)
• CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents [49.68117560675367]
  Crab is the first benchmark framework designed to support cross-environment tasks.<n>Our framework supports multiple devices and can be easily extended to any environment with a Python interface.<n>The experimental results demonstrate that the single agent with GPT-4o achieves the best completion ratio of 38.01%.
  arXiv  Detail & Related papers  (2024-07-01T17:55:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.