MacArena: Benchmarking Computer Use Agents on an Online macOS Environment

• URL: http://arxiv.org/abs/2606.06560v1
• Date: Thu, 04 Jun 2026 14:01:32 GMT
• Title: MacArena: Benchmarking Computer Use Agents on an Online macOS Environment
• Abstract summary: Computer-use agents (CUAs) operate graphical user interfaces (GUIs) through vision and control primitives.<n>Only existing benchmark, OSWorld, covers a narrow slice of first-party applications with simpler tasks.<n>We introduce MacArena, a benchmark of 421 manually verified tasks spanning 50 applications.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Computer-use agents (CUAs) operate graphical user interfaces (GUIs) through vision and control primitives, and their capabilities have advanced rapidly, driven in part by standardized online evaluation benchmarks such as OSWorld, which serve both as evaluation tools and as training environments for reinforcement learning. However, macOS remains underserved in this landscape: the only existing benchmark, macOSWorld, covers a narrow slice of first-party applications with simpler tasks, and runs on x86 virtual machines incompatible with Apple Silicon. We introduce MacArena, a benchmark of 421 manually verified tasks spanning 50 applications that combines a curated port of OSWorld tasks, content sourced from macOSWorld, and 49 new macOS-native tasks, all running on Apple's native Virtualization framework on Apple Silicon. We argue that macOS presents distinct GUI challenges beyond what Linux-based benchmarks capture, and our evaluation supports this claim: strong model performance on existing benchmarks can reflect familiarity with task distributions rather than genuine cross-platform GUI competence. Notably, model rankings invert between ported and macOS-native tasks, with a leading model trailing by over 26% on the MacArena subset, suggesting that macOS poses a genuinely harder environment for current GUI agents.

Related papers

• JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition [51.15720718264743]
  We introduce JarvisGUI, a dynamic benchmark that evaluates GUI agents on cross-device requiring coordinated interaction across platforms.<n>JarvisGUI formulates GUI tasks as input-output transformations under a lightweight type system, which allows us to automatically compose multi-step, cross-device transformations.<n>We show that state-of-the-art open-source GUI agents struggle with the state-transfer awareness, cross-platform contextual reasoning, and long-horizon dependency management required for real-world usage.
  arXiv  Detail & Related papers  (2026-09-09T16:59:59Z)
• MacAgentBench: Benchmarking AI Agents on Real-World macOS Desktop [32.14321067864185]
  Computer use agents (CUAs) have advanced rapidly in desktop automation, and a growing number of users deploy CUAs for always-on automation.<n>Existing benchmarks, including those for metrics, evaluate agents without framework augmentation and rely on binary evaluation.<n>We present MacAgentBench, a comprehensive agent benchmark comprising 676 tasks across 25 applications.
  arXiv  Detail & Related papers  (2026-06-21T15:32:57Z)
• WinDeskGround: A Benchmark for Robust GUI Grounding in Complex Multi-Window Desktop Environments [49.98994180610182]
  Multimodal Large Language Models (MLLMs) have revolutionized GUI automation, yet their efficacy is largely established on idealized, single-layer interfaces.<n>This paper identifies a critical reliability gap: state-of-the-art agents face distinct challenges in real-world desktop environments.<n>We introduce WinDeskGround, a novel benchmark and synthesis framework tailored for evaluating GUI grounding robustness.
  arXiv  Detail & Related papers  (2026-05-13T02:48:52Z)
• OS-Oracle: A Comprehensive Framework for Cross-Platform GUI Critic Models [54.44308299945632]
  We introduce OS-Oracle that makes three core contributions: a scalable data pipeline for cross-platform GUI critic data; a two-stage training paradigm combining supervised fine-tuning and consistency-preserving group relative policy optimization; and OS-Critic Bench, a holistic benchmark for evaluating critic model performance across Mobile, Web, and Desktop platforms.<n>The resulting critic model, OS-Oracle-7B, achieves state-of-the-art performance among open-source VLMs on OS-Critic Bench, and surpasses proprietary models on the mobile domain.
  arXiv  Detail & Related papers  (2025-12-18T08:29:50Z)
• Production-Grade Local LLM Inference on Apple Silicon: A Comparative Study of MLX, MLC-LLM, Ollama, llama.cpp, and PyTorch MPS [0.08030359871216612]
  We present a systematic, empirical evaluation of five local large language models (LLM)s on Apple Silicon.<n>MLX, MLC-LLM, llama, Ollama, and PyTorch MPS were tested.
  arXiv  Detail & Related papers  (2025-10-09T23:53:38Z)
• Screen2AX: Vision-Based Approach for Automatic macOS Accessibility Generation [0.0]
  We introduce Screen2AX, the first framework to automatically create real-time, tree-structured accessibility metadata from a single screenshot.<n>Our method uses vision-language and object detection models to detect, describe, and organize UI elements hierarchically, mirroring Desktop's system-level accessibility structure.<n>We demonstrate that Screen2AX delivers a 2.2x performance improvement over native accessibility representations and surpasses the state-of-the-art Omni V2 system on the ScreenSpot benchmark.
  arXiv  Detail & Related papers  (2025-07-22T15:38:12Z)
• macOSWorld: A Multilingual Interactive Benchmark for GUI Agents [56.25808761731417]
  Graphical User Interface (GUI) agents show promising capabilities for automating computer-use tasks.<n>Existing interactive benchmarks are mostly English-only, covering web-use or Linux, and Android environments.<n>We present GUIWorld, the first comprehensive benchmark for evaluating GUI agents on Windows.
  arXiv  Detail & Related papers  (2025-06-04T16:26:56Z)
• CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents [49.68117560675367]
  Crab is the first benchmark framework designed to support cross-environment tasks.<n>Our framework supports multiple devices and can be easily extended to any environment with a Python interface.<n>The experimental results demonstrate that the single agent with GPT-4o achieves the best completion ratio of 38.01%.
  arXiv  Detail & Related papers  (2024-07-01T17:55:04Z)
• OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments [87.41051677852231]
  We introduce OSWorld, the first-of-its-kind scalable, real computer environment for multimodal agents. OSWorld can serve as a unified, integrated computer environment for assessing open-ended computer tasks. We create a benchmark of 369 computer tasks involving real web and desktop apps in open domains, OS file I/O, and spanning multiple applications.
  arXiv  Detail & Related papers  (2024-04-11T17:56:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.