GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents

• URL: http://arxiv.org/abs/2606.24551v1
• Date: Mon, 22 Jun 2026 17:05:07 GMT
• Title: GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents
• Abstract summary: We introduce a matched execution-layer benchmark of 440 desktop tasks across 18 applications and 12 workflow categories.<n>We show that the strongest GUI agent reaches a 59.1% full pass rate, outperforming the strongest original-skill CLI agent at 48.2%.
• Score: 57.03233891304263
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Computer-use agents can execute software tasks through either graphical interfaces or programmatic command interfaces, but existing evaluations confound interaction modality with differences in tasks, initial states, verifiers, and permitted actions. We introduce a matched execution-layer benchmark of 440 desktop tasks across 18 applications and 12 workflow categories, where screen-only GUI agents and skill-mediated CLI agents receive identical goals, states, and final-state verifiers while being restricted to modality-native actions. In this controlled setting, the strongest GUI agent reaches a 59.1% full pass rate, outperforming the strongest original-skill CLI agent at 48.2%; however, verifier-guided skill augmentation raises CLI success to 69.3%, showing that much of the CLI deficit comes from incomplete skill coverage rather than model capability alone. These results suggest that GUI and CLI expose different execution bottlenecks: GUI agents are limited by reliable grounded interaction over long-horizon workflows, whereas CLI agents are limited by the coverage and scalability of their skill interfaces.

Related papers

• CUA-Universe: A Scalable and Dynamic Environment for Hybrid GUI+CLI Agents [57.098070497654874]
  Computer-use agents have advanced on benchmarks like OSWorld and AndroidWorld, but still act mostly through the GUI.<n>We introduce CUA-Universe, a scalable environment-to-hybrid pipeline.
  arXiv  Detail & Related papers  (2026-09-04T17:26:18Z)
• EmbodiedSkills: A Unified Framework for Orchestrating, Training, and Deploying VLA Agents [56.25120535601186]
  Vision-language-action (VLA) models map visual observations and language instructions directly to robot actions.<n>We propose EmbodiedSkills, a framework that treats each skill decision as an execution proposal.<n>A shared executable-skill interface connects high-level skill selection, bounded low-level VLA execution, and post-action verification.
  arXiv  Detail & Related papers  (2026-09-01T14:14:47Z)
• Beyond Sequential Interaction: Benchmarking Parallel Execution and Coordination for GUI Agents [25.221815264081]
  We introduce ParaGUIBench, the first benchmark dedicated to parallel execution and coordination of multiple Graphical User Interface (GUI) agents on separate desktop instances.<n>On ParaGUIBench, ParaGUI reaches a 46.4% success rate, outperforming the strongest serial baseline (Claude Sonnet 4.6) by 12.9 points while using roughly half the steps and less than half the tokens.<n>These results show that parallel execution can improve both success rate and efficiency on decomposable, long-horizon GUI tasks, pointing to a direction worth further study.
  arXiv  Detail & Related papers  (2026-07-17T12:48:03Z)
• WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces [27.11445886768225]
  We introduce WeaveBench, a long-horizon hybrid-interface benchmark with 114 tasks across 8 real-world work domains.<n>We evaluate these tasks on a real Ubuntu desktop inside deployed CLI-agent runtimes, augmented with a minimal desktop-control plugin.<n>Across frontier model-runtime pairings, the best PassRate reaches only 41.2%, showing the benchmark remains far from saturated.
  arXiv  Detail & Related papers  (2026-06-08T12:39:23Z)
• Syll: Open-Source Personal Automation with Cross-Surface Execution [73.77463150026068]
  We present Syll, an open-source, self-hosted multimodal agent harness.<n>Syll unifies MCP/API tools, CLI execution, and visual control in a modular runtime.<n>Our implementation has been validated on production desktop applications including Adobe Photoshop, Adobe Audition, Stardew Valley, Finder.
  arXiv  Detail & Related papers  (2026-05-28T17:59:31Z)
• AppAgent-Claw: CLI Is All You Need for GUI Automation [9.116481317329493]
  AppAgent-Claw is a demonstration-driven system that converts GUI into reliable, reusable skills without runtime inference.<n>It employs a layered localization strategy to handle visual shifts and a validation-coupled execution model to ensure intended on-screen effects.
  arXiv  Detail & Related papers  (2026-04-15T17:02:09Z)
• CoAct-1: Computer-using Agents with Coding as Actions [94.99657662893338]
  CoAct-1 is a novel multi-agent system that combines GUI-based control with direct programmatic execution.<n>We evaluate our system on the challenging OSWorld benchmark, where CoAct-1 achieves a new state-of-the-art success rate of 60.76%.
  arXiv  Detail & Related papers  (2025-08-05T21:33:36Z)
• MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environment [63.62778707277929]
  MobileGUI-RL is a scalable framework that trains GUI agent in online environment.<n>It synthesizes a curriculum of learnable tasks through self-exploration and filtering.<n>It adapts GRPO to GUI navigation with trajectory-aware advantages and composite rewards.
  arXiv  Detail & Related papers  (2025-07-08T07:07:53Z)
• GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents [93.49577107524176]
  We propose GUI-Actor, a VLM-based method for coordinate-free GUI grounding.<n>At its core, GUI-Actor introduces an attention-based action head that learns to align a dedicated ACTOR> token with all relevant visual patch tokens.<n>Experiments show that GUI-Actor outperforms prior state-of-the-art methods on multiple GUI action grounding benchmarks.
  arXiv  Detail & Related papers  (2025-06-03T17:59:08Z)
• You Only Look at Screens: Multimodal Chain-of-Action Agents [37.118034745972956]
  Auto-GUI is a multimodal solution that directly interacts with the interface. We propose a chain-of-action technique to help the agent decide what action to execute. We evaluate our approach on a new device-control benchmark AITW with 30$K$ unique instructions.
  arXiv  Detail & Related papers  (2023-09-20T16:12:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.