MacAgentBench: Benchmarking AI Agents on Real-World macOS Desktop

• URL: http://arxiv.org/abs/2606.22557v1
• Date: Sun, 21 Jun 2026 15:32:57 GMT
• Title: MacAgentBench: Benchmarking AI Agents on Real-World macOS Desktop
• Abstract summary: Computer use agents (CUAs) have advanced rapidly in desktop automation, and a growing number of users deploy CUAs for always-on automation.<n>Existing benchmarks, including those for metrics, evaluate agents without framework augmentation and rely on binary evaluation.<n>We present MacAgentBench, a comprehensive agent benchmark comprising 676 tasks across 25 applications.
• Score: 32.14321067864185
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Computer use agents (CUAs) have advanced rapidly in desktop automation, and a growing number of users deploy CUAs such as OpenClaw on Mac Mini for always-on automation. However, existing benchmarks, including those for macOS, evaluate agents without framework augmentation and rely on binary evaluation. As a result, they fail to capture both the framework capabilities leveraged by modern CUAs and the partial progress on long-horizon, multi-application tasks. We present MacAgentBench, a comprehensive macOS agent benchmark comprising 676 tasks across 25 applications, with nearly 60% involving both GUI and CLI interaction. The benchmark adopts deterministic rule-based evaluation and introduces fine-grained multi-checkpoint scoring with capability annotations for multi-application tasks. Experiments across three frameworks and 16 models show that the best configuration, Claude Opus 4.6 on OpenClaw, attains 73.7% Pass@1, while this advantage is primarily driven by the skill library rather than by framework design. Fine-grained metrics further reveal that models with similar Pass@1 can differ substantially in sub-goal completion. Our code and data are publicly available at https://github.com/JetAstra/MacAgentBench.

Related papers

• VisualClaw: A Real-Time, Personalized Agent for the Physical World [76.89211120690028]
  We present VisualClaw, a self-evolving multimodal agent built around two principles.<n>First, hybrid encoding reduces deployment cost by filtering less informative streaming frames with a cascaded gate.<n>Second, skill evolution lets the agent learn from failures, producing skill-bank updates that help future questions.
  arXiv  Detail & Related papers  (2026-06-15T06:58:22Z)
• WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces [27.11445886768225]
  We introduce WeaveBench, a long-horizon hybrid-interface benchmark with 114 tasks across 8 real-world work domains.<n>We evaluate these tasks on a real Ubuntu desktop inside deployed CLI-agent runtimes, augmented with a minimal desktop-control plugin.<n>Across frontier model-runtime pairings, the best PassRate reaches only 41.2%, showing the benchmark remains far from saturated.
  arXiv  Detail & Related papers  (2026-06-08T12:39:23Z)
• WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation [88.10947115397971]
  This work presents WildClawBench, a native-runtime benchmark of 60 human-authored, bilingual, multimodal tasks spanning six thematic categories.<n>Each task averages roughly 8 minutes of wall-clock time and over 20 tool calls.<n> Grading is hybrid, combining deterministic rule-based checks, environment-state auditing of side effects, and an LLM/VLM judge for semantic verification.
  arXiv  Detail & Related papers  (2026-05-11T17:49:43Z)
• ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents [77.22389710754452]
  We introduce bench, a benchmark for coworker agents built around multi-turn multi-day tasks.<n>The current release contains 100 tasks across 13 professional scenarios, executed against five stateful sandboxed services.<n>The strongest model reaches 75.8 weighted score, but the best strict Task Success is only 20.0%, indicating that partial progress is common.
  arXiv  Detail & Related papers  (2026-04-26T16:05:02Z)
• OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents [49.34040731113563]
  We present OSWorld-MCP, the first comprehensive and fair benchmark for assessing computer-use agents' tool invocation, GUI operation, and decision-making abilities.<n> Rigorous manual validation yields 158 high-quality tools, each verified for correct functionality, practical applicability, and versatility.<n> OSWorld-MCP deepens understanding of multimodal agents and sets a new standard for evaluating performance in complex, tool-assisted environments.
  arXiv  Detail & Related papers  (2025-10-28T15:56:36Z)
• MCPWorld: A Unified Benchmarking Testbed for API, GUI, and Hybrid Computer Use Agents [14.736516215309768]
  We propose MCPWorld, the first automatic CUA testbed for API, GUI, and API-GUI hybrid agents.<n>A key principle of MCPWorld is the use of "white-box apps", i.e., those with source code availability and can be revised/re-compiled as needed.<n> MCPWorld includes 201 well curated and annotated user tasks, covering diversified use cases and difficulty levels.
  arXiv  Detail & Related papers  (2025-06-09T11:50:33Z)
• macOSWorld: A Multilingual Interactive Benchmark for GUI Agents [56.25808761731417]
  Graphical User Interface (GUI) agents show promising capabilities for automating computer-use tasks.<n>Existing interactive benchmarks are mostly English-only, covering web-use or Linux, and Android environments.<n>We present GUIWorld, the first comprehensive benchmark for evaluating GUI agents on Windows.
  arXiv  Detail & Related papers  (2025-06-04T16:26:56Z)
• InfantAgent-Next: A Multimodal Generalist Agent for Automated Computer Interaction [35.285466934451904]
  This paper introduces textscInfantAgent-Next, a generalist agent capable of interacting with computers in a multimodal manner.<n>Unlike existing approaches that either build intricate around a single large model or only provide modularity, our agent integrates tool-based and pure vision agents.
  arXiv  Detail & Related papers  (2025-05-16T05:43:27Z)
• CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents [49.68117560675367]
  Crab is the first benchmark framework designed to support cross-environment tasks.<n>Our framework supports multiple devices and can be easily extended to any environment with a Python interface.<n>The experimental results demonstrate that the single agent with GPT-4o achieves the best completion ratio of 38.01%.
  arXiv  Detail & Related papers  (2024-07-01T17:55:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.