CUA-Universe: A Scalable and Dynamic Environment for Hybrid GUI+CLI Agents

• URL: http://arxiv.org/abs/2609.05374v1
• Date: Fri, 04 Sep 2026 17:26:18 GMT
• Title: CUA-Universe: A Scalable and Dynamic Environment for Hybrid GUI+CLI Agents
• Abstract summary: Computer-use agents have advanced on benchmarks like OSWorld and AndroidWorld, but still act mostly through the GUI.<n>We introduce CUA-Universe, a scalable environment-to-hybrid pipeline.
• Score: 57.098070497654874
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Computer-use agents have advanced on benchmarks like OSWorld and AndroidWorld, but still act mostly through the GUI, often producing inefficient trajectories. Real-world computer work is hybrid, combining visual-state inspection with precise, high-throughput command-line operations, so capable agents must coordinate both modalities over shared application state. Yet scalable hybrid environments remain scarce because supporting both GUI and CLI over real applications typically requires substantial manual engineering for each application. Existing agents also struggle to use the two interfaces complementarily: CLI-native agents lack visual perception for tasks involving interface state or layout, while GUI-native agents are inefficient for operations better executed through commands. We introduce CUA-Universe, a scalable environment-to-data pipeline that turns real desktop software into hybrid GUI+CLI environments. App-Forge adapts applications into reproducible VMs and command-line surfaces it discovers, wraps, or generates, scaling to 16 applications; Task-Weave synthesizes diverse hybrid tasks of controllable difficulty from reusable operations over seed files; and Path-Steer steers rollouts along efficient hybrid paths and harvests verified trajectories for post-training. Training on this data shifts behavior from inefficient GUI interaction and brittle CLI scripting toward effective GUI+CLI orchestration. Our 9B model improves both success and efficiency on CUA-Verse (Score +39.3 pts; -37% steps, -60% tokens), OSWorld (SR +16.8 pts; -57% steps, -44% tokens), and OSWorld-MCP (Score +7.84 pts; -27% steps, -30% tokens). CUA-Universe provides a scalable path toward more capable and efficient computer-use agents.

Related papers

• EnvCraft: Synthesizing Executable Environments in Agentic RL for Claw-like Agent [21.572853994120806]
  EnvCraft is an automated framework for synthesizing executable environments and scalable training data.<n>We synthesize 139 interactive environments comprising approximately 20K complex tasks for Agentic RL training.
  arXiv  Detail & Related papers  (2026-09-04T10:13:41Z)
• GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents [57.03233891304263]
  We introduce a matched execution-layer benchmark of 440 desktop tasks across 18 applications and 12 workflow categories.<n>We show that the strongest GUI agent reaches a 59.1% full pass rate, outperforming the strongest original-skill CLI agent at 48.2%.
  arXiv  Detail & Related papers  (2026-06-22T17:05:07Z)
• WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces [27.11445886768225]
  We introduce WeaveBench, a long-horizon hybrid-interface benchmark with 114 tasks across 8 real-world work domains.<n>We evaluate these tasks on a real Ubuntu desktop inside deployed CLI-agent runtimes, augmented with a minimal desktop-control plugin.<n>Across frontier model-runtime pairings, the best PassRate reaches only 41.2%, showing the benchmark remains far from saturated.
  arXiv  Detail & Related papers  (2026-06-08T12:39:23Z)
• AppAgent-Claw: CLI Is All You Need for GUI Automation [9.116481317329493]
  AppAgent-Claw is a demonstration-driven system that converts GUI into reliable, reusable skills without runtime inference.<n>It employs a layered localization strategy to handle visual shifts and a validation-coupled execution model to ensure intended on-screen effects.
  arXiv  Detail & Related papers  (2026-04-15T17:02:09Z)
• ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents [54.04035382782041]
  textbfClawGUI-RL provides the first open-source GUI agent RL infrastructure with validated support for both parallel virtual environments and real physical devices.<n>textbfClawGUI-Eval enforces a fully standardized evaluation pipeline across 6 benchmarks and 11+ models.<n>textbfClawGUI-Agent brings trained agents to Android, HarmonyOS, and iOS through 12+ chat platforms with hybrid CLI-GUI control and persistent personalized memory.
  arXiv  Detail & Related papers  (2026-04-13T17:52:04Z)
• GUI-GENESIS: Automated Synthesis of Efficient Environments with Verifiable Rewards for GUI Agent Post-Training [30.675339113121225]
  We present GUI-GENESIS, a framework to automatically synthesize efficient GUI training environments with verifiable rewards.<n>Experiments show that GUI-GENESIS reduces environment latency by 10 times and costs by over $28,000 per epoch compared to training on real applications.
  arXiv  Detail & Related papers  (2026-02-15T10:58:01Z)
• UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Action [77.63125913907771]
  We present UltraCUA, a foundation model that bridges the gap between GUI primitives and high-level programmatic tool calls.<n>Experiments with our 7B and 32B models demonstrate substantial improvements over state-of-the-art agents.
  arXiv  Detail & Related papers  (2025-10-20T17:48:26Z)
• CoAct-1: Computer-using Agents with Coding as Actions [94.99657662893338]
  CoAct-1 is a novel multi-agent system that combines GUI-based control with direct programmatic execution.<n>We evaluate our system on the challenging OSWorld benchmark, where CoAct-1 achieves a new state-of-the-art success rate of 60.76%.
  arXiv  Detail & Related papers  (2025-08-05T21:33:36Z)
• MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environment [63.62778707277929]
  MobileGUI-RL is a scalable framework that trains GUI agent in online environment.<n>It synthesizes a curriculum of learnable tasks through self-exploration and filtering.<n>It adapts GRPO to GUI navigation with trajectory-aware advantages and composite rewards.
  arXiv  Detail & Related papers  (2025-07-08T07:07:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.