OpenComputer: Verifiable Software Worlds for Computer-Use Agents

• URL: http://arxiv.org/abs/2605.19769v1
• Date: Tue, 19 May 2026 12:40:29 GMT
• Title: OpenComputer: Verifiable Software Worlds for Computer-Use Agents
• Abstract summary: OpenComputer is a verifier-grounded framework for constructing verifiable software worlds for computer-use agents.<n>In its current form, OpenComputer covers 33 desktop applications and 1,000 finalized tasks spanning browsers, office tools, creative software, development environments, file managers, and communication applications.
• Score: 50.93029275079928
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present OpenComputer, a verifier-grounded framework for constructing verifiable software worlds for computer-use agents. OpenComputer integrates four components: (1) app-specific state verifiers that expose structured inspection endpoints over real applications, (2) a self-evolving verification layer that improves verifier reliability using execution-grounded feedback, (3) a task-generation pipeline that synthesizes realistic and machine-checkable desktop tasks, and (4) an evaluation harness that records full trajectories and computes auditable partial-credit rewards. In its current form, OpenComputer covers 33 desktop applications and 1,000 finalized tasks spanning browsers, office tools, creative software, development environments, file managers, and communication applications. Experiments show that OpenComputer's hard-coded verifiers align more closely with human adjudication than LLM-as-judge evaluation, especially when success depends on fine-grained application state. Frontier agents struggle with end-to-end completion despite partial progress, and open-source models exhibit sharp drops from their OSWorld-Verified scores, exposing a persistent gap in robust computer automation.

Related papers

• UI-Venus-2 Technical Report [51.54957558722245]
  We present UI-Venus-2, a general-purpose foundation GUI agent designed to operate across mobile, web, and desktop environments.<n>To bridge the gap toward practical deployment, we jointly scale three critical dimensions: (1) Environments, expanding coverage to more than 170 multilingual mobile apps and native desktop operating systems; (2) Tasks, employing a deep-research pipeline for function-grounded instruction generation; and (3) Verification, adopting trace-level and sample-level evaluators with visual keypoints and multi-model voting to ensure reliable RL signals for training.
  arXiv  Detail & Related papers  (2026-08-27T17:15:33Z)
• Terminal Agents: A Survey of AI Agents in Command-Line Environments [102.48222742145474]
  Large language model agents increasingly act through terminals, yet existing surveys disperse terminal-mediated behavior across software engineering, tool use, and computer-use research.<n>We regard terminal agents as systems whose dominant progress-bearing action--observation loop is mediated by terminal command execution, textual feedback, and stateful environment interaction.<n>Our synthesis shows that realized behavior is jointly shaped by the model, interface, harness, runtime, and environment.
  arXiv  Detail & Related papers  (2026-08-20T18:16:44Z)
• Computer-Using World Model [58.59112582915026]
  We introduce the Computer-Using World Model (CUWM), a world model for desktop software that predicts the next user interface (UI) state.<n> CUWM first predicts a textual description of agent-relevant state changes, and then realizes these changes visually to synthesize the next screenshot.<n>We evaluate CUWM via test-time action search, where a frozen agent uses the world model to simulate and compare candidate actions before execution.
  arXiv  Detail & Related papers  (2026-02-19T13:48:29Z)
• OpenCUA: Open Foundations for Computer-Use Agents [74.61449905487565]
  Vision-language models have demonstrated impressive capabilities as computer-use agents (CUAs)<n>We propose OpenCUA, a comprehensive open-source framework for scaling CUA data and foundation models.<n>Our end-to-end agent models demonstrate strong performance across CUA benchmarks.
  arXiv  Detail & Related papers  (2025-08-12T17:52:32Z)
• OS-MAP: How Far Can Computer-Using Agents Go in Breadth and Depth? [30.788287791669458]
  OS-MAP is a benchmark for daily computer-using automation.<n>It organizes its 416 realistic tasks across 15 applications along two key dimensions.<n>It captures varying levels of required agent autonomy and generalization.
  arXiv  Detail & Related papers  (2025-07-25T10:14:53Z)
• UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction [16.731754927372585]
  We introduce UI-Vision, the first comprehensive, license-permissive benchmark for offline, fine-grained evaluation of computer use agents.<n>Unlike online benchmarks, UI-Vision provides dense, high-quality annotations of human demonstrations.<n>Our evaluation reveals critical limitations in state-of-the-art models like UI-TARS-72B.
  arXiv  Detail & Related papers  (2025-03-19T19:26:17Z)
• OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments [87.41051677852231]
  We introduce OSWorld, the first-of-its-kind scalable, real computer environment for multimodal agents. OSWorld can serve as a unified, integrated computer environment for assessing open-ended computer tasks. We create a benchmark of 369 computer tasks involving real web and desktop apps in open domains, OS file I/O, and spanning multiple applications.
  arXiv  Detail & Related papers  (2024-04-11T17:56:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.