UI-Copilot: Advancing Long-Horizon GUI Automation via Tool-Integrated Policy Optimization

• URL: http://arxiv.org/abs/2604.13822v1
• Date: Wed, 15 Apr 2026 12:55:46 GMT
• Title: UI-Copilot: Advancing Long-Horizon GUI Automation via Tool-Integrated Policy Optimization
• Abstract summary: Experimental results show that UI-Copilot-7B achieves state-of-the-art performance on challenging MemGUI-Bench.<n> UI-Copilot-7B delivers a 17.1% absolute improvement on AndroidWorld over the base Qwen model.
• Score: 70.85564601986263
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: MLLM-based GUI agents have demonstrated strong capabilities in complex user interface interaction tasks. However, long-horizon scenarios remain challenging, as these agents are burdened with tasks beyond their intrinsic capabilities, suffering from memory degradation, progress confusion, and math hallucination. To address these challenges, we present UI-Copilot, a collaborative framework where the GUI agent focuses on task execution while a lightweight copilot provides on-demand assistance for memory retrieval and numerical computation. We introduce memory decoupling to separate persistent observations from transient execution context, and train the policy agent to selectively invoke the copilot as Retriever or Calculator based on task demands. To enable effective tool invocation learning, we propose Tool-Integrated Policy Optimization (TIPO), which separately optimizes tool selection through single-turn prediction and task execution through on-policy multi-turn rollouts. Experimental results show that UI-Copilot-7B achieves state-of-the-art performance on challenging MemGUI-Bench, outperforming strong 7B-scale GUI agents such as GUI-Owl-7B and UI-TARS-1.5-7B. Moreover, UI-Copilot-7B delivers a 17.1% absolute improvement on AndroidWorld over the base Qwen model, highlighting UI-Copilot's strong generalization to real-world GUI tasks.

Related papers

• ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agents [87.63969994744133]
  Computer Use Agents (CUAs) can act through both atomic GUI actions, such as click and type, and high-level tool calls, such as API-based file operations.<n>This difficulty stems from the scarcity of high-quality interleaved GUI-Tool trajectories, the cost and brittleness of collecting real tool trajectories, and the lack of trajectory-level supervision for GUI-Tool path selection.<n>We propose ToolCUA, an end-to-end agent designed to learn optimal GUI-Tool path selection through a staged training paradigm.
  arXiv  Detail & Related papers  (2026-05-12T17:57:04Z)
• AutoGUI-v2: A Comprehensive Multi-Modal GUI Functionality Understanding Benchmark [32.66632642377623]
  AutoGUI-v2 is a benchmark designed to evaluate deep GUI functionality understanding and interaction outcome prediction.<n>We construct the benchmark using a novel VLM-human collaborative pipeline that parses screenshots into hierarchical functional regions.<n>AutoGUI-v2 rigorously tests agents on region and element-level semantics, grounding, and dynamic state prediction.
  arXiv  Detail & Related papers  (2026-04-27T13:06:27Z)
• UIPro: Unleashing Superior Interaction Capability For GUI Agents [33.77980648230746]
  Building autonomous agents that perceive and operate graphical user interfaces (GUIs) like humans has long been a vision in the field of artificial intelligence.<n>Existing methods have tried developing GUI agents based on the multi-modal comprehension ability of vision-language models (VLMs)<n>This paper proposes textUIPro, a novel generalist GUI agent trained with extensive multi-platform and multi-task GUI interaction data.
  arXiv  Detail & Related papers  (2025-09-22T03:04:53Z)
• UItron: Foundational GUI Agent with Advanced Perception and Planning [13.67797194012135]
  We introduce open-source model for automatic GUI agents, featuring advanced GUI perception, grounding, and planning capabilities.<n> UItron highlights the necessity of systemic data engineering and interactive infrastructure as foundational components for advancing GUI agent development.<n>We manually collect over one million steps of operation trajectories across the top 100 most popular apps, and build the offline and online agent evaluation environments.
  arXiv  Detail & Related papers  (2025-08-29T16:40:57Z)
• CoAct-1: Computer-using Agents with Coding as Actions [94.99657662893338]
  CoAct-1 is a novel multi-agent system that combines GUI-based control with direct programmatic execution.<n>We evaluate our system on the challenging OSWorld benchmark, where CoAct-1 achieves a new state-of-the-art success rate of 60.76%.
  arXiv  Detail & Related papers  (2025-08-05T21:33:36Z)
• MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents [88.35544552383581]
  We introduce MMBench-GUI, a hierarchical benchmark for evaluating GUI automation agents across Windows, Linux, iOS, Android, and Web platforms.<n>It comprises four levels: GUI Content Understanding, Element Grounding, Task Automation, and Task Collaboration, covering essential skills for GUI agents.
  arXiv  Detail & Related papers  (2025-07-25T17:59:26Z)
• MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environment [63.62778707277929]
  MobileGUI-RL is a scalable framework that trains GUI agent in online environment.<n>It synthesizes a curriculum of learnable tasks through self-exploration and filtering.<n>It adapts GRPO to GUI navigation with trajectory-aware advantages and composite rewards.
  arXiv  Detail & Related papers  (2025-07-08T07:07:53Z)
• AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuning [82.42421823672954]
  AgentCPM-GUI is built for robust and efficient on-device GUI interaction.<n>Our training pipeline includes grounding-aware pre-training to enhance perception.<n>AgentCPM-GUI achieves state-of-the-art performance on five public benchmarks.
  arXiv  Detail & Related papers  (2025-06-02T07:30:29Z)
• Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction [69.57190742976091]
  Aguvis is a vision-based framework for autonomous GUI agents.<n>It standardizes cross-platform interactions and incorporates structured reasoning via inner monologue.<n>It achieves state-of-the-art performance across offline and real-world online benchmarks.
  arXiv  Detail & Related papers  (2024-12-05T18:58:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.