Beyond Sequential Interaction: Benchmarking Parallel Execution and Coordination for GUI Agents

• URL: http://arxiv.org/abs/2607.22689v1
• Date: Fri, 17 Jul 2026 12:48:03 GMT
• Title: Beyond Sequential Interaction: Benchmarking Parallel Execution and Coordination for GUI Agents
• Abstract summary: We introduce ParaGUIBench, the first benchmark dedicated to parallel execution and coordination of multiple Graphical User Interface (GUI) agents on separate desktop instances.<n>On ParaGUIBench, ParaGUI reaches a 46.4% success rate, outperforming the strongest serial baseline (Claude Sonnet 4.6) by 12.9 points while using roughly half the steps and less than half the tokens.<n>These results show that parallel execution can improve both success rate and efficiency on decomposable, long-horizon GUI tasks, pointing to a direction worth further study.
• Score: 25.221815264081
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Graphical user interface (GUI) agents are systems powered by large multimodal models (LMMs). They perceive screen state and execute user instructions through GUI actions such as clicking, typing, and scrolling on desktops and mobile devices. However, current agents scale poorly to long-horizon tasks: actions incur costly LMM inferences, and performance degrades as context grows. Humans divide such workloads among collaborators who complete sub-tasks in parallel. Yet parallel coordination among GUI agents has received little attention. To close this gap, we introduce ParaGUIBench, to our knowledge, the first benchmark dedicated to parallel execution and coordination of multiple GUI agents on separate desktop instances. It consists of three components: a multi-device Docker infrastructure with a shared file system; a dataset of 233 tasks spanning six task categories; and an evaluation system with efficiency metrics, including step reduction ratio and token cost. We further introduce ParaGUI, a planner-worker agent that decomposes GUI tasks and dispatches sub-tasks to concurrent workers on separate desktop instances. On ParaGUIBench, ParaGUI reaches a 46.4% success rate, outperforming the strongest serial baseline (Claude Sonnet 4.6) by 12.9 points while using roughly half the steps and less than half the tokens. These results show that parallel execution can improve both success rate and efficiency on decomposable, long-horizon GUI tasks, pointing to a direction worth further study.

Related papers

• JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition [51.15720718264743]
  We introduce JarvisGUI, a dynamic benchmark that evaluates GUI agents on cross-device requiring coordinated interaction across platforms.<n>JarvisGUI formulates GUI tasks as input-output transformations under a lightweight type system, which allows us to automatically compose multi-step, cross-device transformations.<n>We show that state-of-the-art open-source GUI agents struggle with the state-transfer awareness, cross-platform contextual reasoning, and long-horizon dependency management required for real-world usage.
  arXiv  Detail & Related papers  (2026-09-09T16:59:59Z)
• UIPro: Unleashing Superior Interaction Capability For GUI Agents [33.77980648230746]
  Building autonomous agents that perceive and operate graphical user interfaces (GUIs) like humans has long been a vision in the field of artificial intelligence.<n>Existing methods have tried developing GUI agents based on the multi-modal comprehension ability of vision-language models (VLMs)<n>This paper proposes textUIPro, a novel generalist GUI agent trained with extensive multi-platform and multi-task GUI interaction data.
  arXiv  Detail & Related papers  (2025-09-22T03:04:53Z)
• OmniActor: A Generalist GUI and Embodied Agent for 2D&3D Worlds [21.902626737678286]
  Multimodal large language models are evolving toward multimodal agents capable of proactively executing tasks.<n>Most agent research focuses on GUI or embodied scenarios, which correspond to agents interacting with 2D virtual worlds or 3D real worlds, respectively.<n>We propose a high-performance generalist agent OmniActor, designed from both structural and data perspectives.
  arXiv  Detail & Related papers  (2025-09-02T13:47:54Z)
• CoAct-1: Computer-using Agents with Coding as Actions [94.99657662893338]
  CoAct-1 is a novel multi-agent system that combines GUI-based control with direct programmatic execution.<n>We evaluate our system on the challenging OSWorld benchmark, where CoAct-1 achieves a new state-of-the-art success rate of 60.76%.
  arXiv  Detail & Related papers  (2025-08-05T21:33:36Z)
• MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents [88.35544552383581]
  We introduce MMBench-GUI, a hierarchical benchmark for evaluating GUI automation agents across Windows, Linux, iOS, Android, and Web platforms.<n>It comprises four levels: GUI Content Understanding, Element Grounding, Task Automation, and Task Collaboration, covering essential skills for GUI agents.
  arXiv  Detail & Related papers  (2025-07-25T17:59:26Z)
• MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environment [63.62778707277929]
  MobileGUI-RL is a scalable framework that trains GUI agent in online environment.<n>It synthesizes a curriculum of learnable tasks through self-exploration and filtering.<n>It adapts GRPO to GUI navigation with trajectory-aware advantages and composite rewards.
  arXiv  Detail & Related papers  (2025-07-08T07:07:53Z)
• ScaleTrack: Scaling and back-tracking Automated GUI Agents [11.046190201201348]
  We propose ScaleTrack, a training framework by scaling grounding and backtracking planning for automated GUI agents.<n>We collect GUI samples of different synthesis criterions from a wide range of sources, and unified them into the same template for training GUI grounding models.<n>We design a novel training strategy that predicts the next action from the current GUI image, while also backtracking the historical actions that led to the GUI image.
  arXiv  Detail & Related papers  (2025-05-01T09:27:13Z)
• UI-TARS: Pioneering Automated GUI Interaction with Native Agents [58.18100825673032]
  This paper introduces UI-TARS, a native GUI agent model that solely perceives the screenshots as input and performs human-like interactions.<n>In the OSWorld benchmark, UI-TARS achieves scores of 24.6 with 50 steps and 22.7 with 15 steps, outperforming Claude (22.0 and 14.9 respectively)
  arXiv  Detail & Related papers  (2025-01-21T17:48:10Z)
• InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection [38.833925781308665]
  We introduce textitInfiGUIAgent, an MLLM-based GUI Agent trained with a two-stage supervised fine-tuning pipeline.<n> Stage 1 enhances fundamental skills such as GUI understanding and grounding, while Stage 2 integrates hierarchical reasoning and expectation-reflection reasoning skills.<n>textitInfiGUIAgent achieves competitive performance on several GUI benchmarks.
  arXiv  Detail & Related papers  (2025-01-08T15:45:21Z)
• VideoGUI: A Benchmark for GUI Automation from Instructional Videos [78.97292966276706]
  VideoGUI is a novel multi-modal benchmark designed to evaluate GUI assistants on visual-centric GUI tasks. Sourced from high-quality web instructional videos, our benchmark focuses on tasks involving professional and novel software. Our evaluation reveals that even the SoTA large multimodal model GPT4o performs poorly on visual-centric GUI tasks.
  arXiv  Detail & Related papers  (2024-06-14T17:59:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.