macOSWorld: A Multilingual Interactive Benchmark for GUI Agents

• URL: http://arxiv.org/abs/2506.04135v4
• Date: Sat, 18 Oct 2025 12:11:31 GMT
• Title: macOSWorld: A Multilingual Interactive Benchmark for GUI Agents
• Abstract summary: Graphical User Interface (GUI) agents show promising capabilities for automating computer-use tasks.<n>Existing interactive benchmarks are mostly English-only, covering web-use or Linux, and Android environments.<n>We present GUIWorld, the first comprehensive benchmark for evaluating GUI agents on Windows.
• Score: 56.25808761731417
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Graphical User Interface (GUI) agents show promising capabilities for automating computer-use tasks and facilitating accessibility, but existing interactive benchmarks are mostly English-only, covering web-use or Windows, Linux, and Android environments, but not macOS. macOS is a major OS with distinctive GUI patterns and exclusive applications. To bridge the gaps, we present macOSWorld, the first comprehensive benchmark for evaluating GUI agents on macOS. macOSWorld features 202 multilingual interactive tasks across 30 applications (28 macOS-exclusive), with task instructions and OS interfaces offered in 5 languages (English, Chinese, Arabic, Japanese, and Russian). As GUI agents are shown to be vulnerable to deception attacks, macOSWorld also includes a dedicated safety benchmarking subset. Our evaluation on six GUI agents reveals a dramatic gap: proprietary computer-use agents lead at above 30% success rate, while open-source lightweight research models lag at below 5\%, highlighting the need for macOS domain adaptation. Multilingual benchmarks also expose common weaknesses, especially in Arabic, with a 28.8% average degradation compared to English. Results from safety benchmarking also highlight that deception attacks are more general and demand immediate attention. Project page: https://macos-world.github.io.

Related papers

• MacAgentBench: Benchmarking AI Agents on Real-World macOS Desktop [32.14321067864185]
  Computer use agents (CUAs) have advanced rapidly in desktop automation, and a growing number of users deploy CUAs for always-on automation.<n>Existing benchmarks, including those for metrics, evaluate agents without framework augmentation and rely on binary evaluation.<n>We present MacAgentBench, a comprehensive agent benchmark comprising 676 tasks across 25 applications.
  arXiv  Detail & Related papers  (2026-06-21T15:32:57Z)
• MacArena: Benchmarking Computer Use Agents on an Online macOS Environment [0.0]
  Computer-use agents (CUAs) operate graphical user interfaces (GUIs) through vision and control primitives.<n>Only existing benchmark, OSWorld, covers a narrow slice of first-party applications with simpler tasks.<n>We introduce MacArena, a benchmark of 421 manually verified tasks spanning 50 applications.
  arXiv  Detail & Related papers  (2026-06-04T14:01:32Z)
• GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agents [19.27396264271709]
  GUI-CEval is the first comprehensive benchmark for Chinese mobile GUI agents built entirely on physical device environments.<n>It spans 201 mainstream apps across four device types and adopts a two-level structure that evaluates both atomic abilities and realistic application-level performance along five dimensions: perception, planning, reflection, execution, and evaluation.
  arXiv  Detail & Related papers  (2026-03-16T09:45:33Z)
• MPR-GUI: Benchmarking and Enhancing Multilingual Perception and Reasoning in GUI Agents [42.81572211701814]
  Large Vision-Language Models (LVLMs) exhibit impressive Perception and Reasoning (P&R) performance on Graphical User Interface (GUI) tasks.<n>However, their performance in multilingual settings has received little attention, which limits their global applications.<n>We propose MPR-GUI-Bench, a fine-grained Perception and Reasoning GUI Benchmark to evaluate GUI agents' P&R capabilities.
  arXiv  Detail & Related papers  (2025-11-30T06:47:33Z)
• GUIrilla: A Scalable Framework for Automated Desktop UI Exploration [0.0]
  GUIrilla is an automated framework that explores applications via native accessibility APIs to address the critical data collection challenge in GUI automation.<n>We construct and release GUIrilla-Task, a large-scale dataset of 27,171 functionally grounded tasks across 1,108 applications.<n> tuning LLM-based agents on GUIrilla-Task significantly improves performance on downstream UI tasks, outperforming synthetic baselines on the ScreenSpot Pro benchmark while using 97% less data.
  arXiv  Detail & Related papers  (2025-10-16T19:03:45Z)
• Mobile-Agent-v3: Fundamental Agents for GUI Automation [59.775510710011325]
  This paper introduces a foundational GUI agent model that achieves state-of-the-art performance among open-source end-to-end models.<n>We propose Mobile-Agent-v3, a general-purpose GUI agent framework that further improves performance to 73.3 on AndroidWorld and 37.7 on OSWorld.
  arXiv  Detail & Related papers  (2025-08-21T00:39:12Z)
• MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents [88.35544552383581]
  We introduce MMBench-GUI, a hierarchical benchmark for evaluating GUI automation agents across Windows, Linux, iOS, Android, and Web platforms.<n>It comprises four levels: GUI Content Understanding, Element Grounding, Task Automation, and Task Collaboration, covering essential skills for GUI agents.
  arXiv  Detail & Related papers  (2025-07-25T17:59:26Z)
• Screen2AX: Vision-Based Approach for Automatic macOS Accessibility Generation [0.0]
  We introduce Screen2AX, the first framework to automatically create real-time, tree-structured accessibility metadata from a single screenshot.<n>Our method uses vision-language and object detection models to detect, describe, and organize UI elements hierarchically, mirroring Desktop's system-level accessibility structure.<n>We demonstrate that Screen2AX delivers a 2.2x performance improvement over native accessibility representations and surpasses the state-of-the-art Omni V2 system on the ScreenSpot benchmark.
  arXiv  Detail & Related papers  (2025-07-22T15:38:12Z)
• AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuning [82.42421823672954]
  AgentCPM-GUI is built for robust and efficient on-device GUI interaction.<n>Our training pipeline includes grounding-aware pre-training to enhance perception.<n>AgentCPM-GUI achieves state-of-the-art performance on five public benchmarks.
  arXiv  Detail & Related papers  (2025-06-02T07:30:29Z)
• Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis [59.83524388782554]
  Graphical user interface (GUI) grounding remains a critical bottleneck in computer use agent development.<n>We introduce OSWorld-G, a comprehensive benchmark comprising 564 finely annotated samples across diverse task types.<n>We synthesize and release the largest computer use grounding dataset Jedi, which contains 4 million examples.
  arXiv  Detail & Related papers  (2025-05-19T15:09:23Z)
• Ponder & Press: Advancing Visual GUI Agent towards General Computer Control [13.39115823642937]
  Ponder & Press is a divide-and-conquer framework for general computer control using only visual input.<n>Our agent offers a versatile, human-like interaction paradigm applicable to a wide range of applications.
  arXiv  Detail & Related papers  (2024-12-02T08:35:31Z)
• OS-ATLAS: A Foundation Action Model for Generalist GUI Agents [55.37173845836839]
  OS-Atlas is a foundational GUI action model that excels at GUI grounding and OOD agentic tasks. We are releasing the largest open-source cross-platform GUI grounding corpus to date, which contains over 13 million GUI elements.
  arXiv  Detail & Related papers  (2024-10-30T17:10:19Z)
• Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale [22.493676199881794]
  Large language models (LLMs) show remarkable potential to act as computer agents. measuring agent performance in realistic environments remains a challenge. We introduce the Windows Agent Arena: a reproducible, general environment focusing exclusively on the Windows operating system (OS) Our agent achieves a success rate of 19.5% in the Windows domain, compared to 74.5% performance of an unassisted human.
  arXiv  Detail & Related papers  (2024-09-12T17:56:43Z)
• CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents [49.68117560675367]
  Crab is the first benchmark framework designed to support cross-environment tasks. Our framework supports multiple devices and can be easily extended to any environment with a Python interface. The experimental results demonstrate that the single agent with GPT-4o achieves the best completion ratio of 38.01%.
  arXiv  Detail & Related papers  (2024-07-01T17:55:04Z)
• OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments [87.41051677852231]
  We introduce OSWorld, the first-of-its-kind scalable, real computer environment for multimodal agents. OSWorld can serve as a unified, integrated computer environment for assessing open-ended computer tasks. We create a benchmark of 369 computer tasks involving real web and desktop apps in open domains, OS file I/O, and spanning multiple applications.
  arXiv  Detail & Related papers  (2024-04-11T17:56:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.