OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Execution

• URL: http://arxiv.org/abs/2601.20380v1
• Date: Wed, 28 Jan 2026 08:45:17 GMT
• Title: OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Execution
• Authors: Le Zhang, Yixiong Xiao, Xinjiang Lu, Jingjia Cao, Yusai Zhao, Jingbo Zhou, Lang An, Zikan Feng, Wanxiang Sha, Yu Shi, Congxi Xiao, Jian Xiong, Yankai Zhang, Hua Wu, Haifeng Wang,
• Abstract summary: OmegaUse is a general-purpose GUI agent model for autonomous task execution on both mobile and desktop platforms.<n>It is highly competitive across established GUI benchmarks, achieving a state-of-the-art (SOTA) score of 96.3% on ScreenSpot-V2.<n>It also performs strongly on OS-Nav, reaching 74.24% step success on ChiM-Nav and 55.9% average success on Ubu-Nav.
• Score: 32.992104943415995
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Graphical User Interface (GUI) agents show great potential for enabling foundation models to complete real-world tasks, revolutionizing human-computer interaction and improving human productivity. In this report, we present OmegaUse, a general-purpose GUI agent model for autonomous task execution on both mobile and desktop platforms, supporting computer-use and phone-use scenarios. Building an effective GUI agent model relies on two factors: (1) high-quality data and (2) effective training methods. To address these, we introduce a carefully engineered data-construction pipeline and a decoupled training paradigm. For data construction, we leverage rigorously curated open-source datasets and introduce a novel automated synthesis framework that integrates bottom-up autonomous exploration with top-down taxonomy-guided generation to create high-fidelity synthetic data. For training, to better leverage these data, we adopt a two-stage strategy: Supervised Fine-Tuning (SFT) to establish fundamental interaction syntax, followed by Group Relative Policy Optimization (GRPO) to improve spatial grounding and sequential planning. To balance computational efficiency with agentic reasoning capacity, OmegaUse is built on a Mixture-of-Experts (MoE) backbone. To evaluate cross-terminal capabilities in an offline setting, we introduce OS-Nav, a benchmark suite spanning multiple operating systems: ChiM-Nav, targeting Chinese Android mobile environments, and Ubu-Nav, focusing on routine desktop interactions on Ubuntu. Extensive experiments show that OmegaUse is highly competitive across established GUI benchmarks, achieving a state-of-the-art (SOTA) score of 96.3% on ScreenSpot-V2 and a leading 79.1% step success rate on AndroidControl. OmegaUse also performs strongly on OS-Nav, reaching 74.24% step success on ChiM-Nav and 55.9% average success on Ubu-Nav.

Related papers

• Building Autonomous GUI Navigation via Agentic-Q Estimation and Step-Wise Policy Optimization [38.863014369090074]
  We introduce a novel MLLM-centered framework for GUI agents, which consists of two components: agentic-Q estimation and step-wise policy optimization.<n>Our framework endows Ovis2.5-9B with powerful GUI interaction capabilities, achieving remarkable performances on GUI navigation and grounding benchmarks.
  arXiv  Detail & Related papers  (2026-02-14T07:44:47Z)
• UI-Venus-1.5 Technical Report [64.4832043785725]
  We present UI-Venus-1.5, a unified, end-to-end GUI Agent.<n>The proposed model family comprises two dense variants (2B and 8B) and one mixture-of-experts variant (30B-A3B)<n>In addition, UI-Venus-1.5 demonstrates robust navigation capabilities across a variety of Chinese mobile apps.
  arXiv  Detail & Related papers  (2026-02-09T18:43:40Z)
• Step-GUI Technical Report [84.83795946544292]
  We introduce a self-evolving training pipeline powered by the Calibrated Step Reward System.<n>We also introduce Step-GUI, a family of models that achieves state-of-the-art GUI performance.<n>To assess whether agents can handle authentic everyday usage, we introduce AndroidDaily.
  arXiv  Detail & Related papers  (2025-12-17T13:26:30Z)
• UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Action [77.63125913907771]
  We present UltraCUA, a foundation model that bridges the gap between GUI primitives and high-level programmatic tool calls.<n>Experiments with our 7B and 32B models demonstrate substantial improvements over state-of-the-art agents.
  arXiv  Detail & Related papers  (2025-10-20T17:48:26Z)
• UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning [155.51875080423883]
  The development of autonomous agents for graphical user interfaces presents major challenges in artificial intelligence.<n>We present UI-TARS-2, a native GUI-centered agent model that addresses these challenges through a systematic training methodology.<n> Empirical evaluation demonstrates that UI-TARS-2 achieves significant improvements over its predecessor UI-TARS-1.5.
  arXiv  Detail & Related papers  (2025-09-02T17:44:45Z)
• UItron: Foundational GUI Agent with Advanced Perception and Planning [13.67797194012135]
  We introduce open-source model for automatic GUI agents, featuring advanced GUI perception, grounding, and planning capabilities.<n> UItron highlights the necessity of systemic data engineering and interactive infrastructure as foundational components for advancing GUI agent development.<n>We manually collect over one million steps of operation trajectories across the top 100 most popular apps, and build the offline and online agent evaluation environments.
  arXiv  Detail & Related papers  (2025-08-29T16:40:57Z)
• OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis [55.390060529534644]
  We propose OS-Genesis, a novel data synthesis pipeline for Graphical User Interface (GUI) agents.<n>Instead of relying on pre-defined tasks, OS-Genesis enables agents first to perceive environments and perform step-wise interactions.<n>We demonstrate that training GUI agents with OS-Genesis significantly improves their performance on highly challenging online benchmarks.
  arXiv  Detail & Related papers  (2024-12-27T16:21:58Z)
• AutoGLM: Autonomous Foundation Agents for GUIs [51.276965515952]
  We present AutoGLM, a new series in the ChatGLM family, designed to serve as foundation agents for autonomous control of digital devices through Graphical User Interfaces (GUIs) We have developed AutoGLM as a practical foundation agent system for real-world GUI interactions. Our evaluations demonstrate AutoGLM's effectiveness across multiple domains.
  arXiv  Detail & Related papers  (2024-10-28T17:05:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.