AndroidControl-Curated: Revealing the True Potential of GUI Agents through Benchmark Purification

• URL: http://arxiv.org/abs/2510.18488v1
• Date: Tue, 21 Oct 2025 10:11:33 GMT
• Title: AndroidControl-Curated: Revealing the True Potential of GUI Agents through Benchmark Purification
• Abstract summary: On-device virtual assistants like Siri and Google Assistant are increasingly pivotal, yet their capabilities are hamstrung by a reliance on rigid, developer-dependent APIs.<n>Gui agents offer a powerful, API-independent alternative, but their adoption is hindered by the perception of poor performance.<n>Our research reveals that issue lies not only with the models but with the benchmarks themselves.
• Score: 0.764671395172401
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: On-device virtual assistants like Siri and Google Assistant are increasingly pivotal, yet their capabilities are hamstrung by a reliance on rigid, developer-dependent APIs. GUI agents offer a powerful, API-independent alternative, but their adoption is hindered by the perception of poor performance, as even the best models (e.g. Qwen3-VL-235B) scores are capped at around 60% on benchmarks like AndroidControl, far from viability for real-world use. Our research reveals that issue lies not only with the models but with the benchmarks themselves. We identified notable shortcomings in AndroidControl, including ambiguities and factual errors, which systematically underrates agent capabilities. To address this critical oversight, we enhanced AndroidControl into AndroidControl-Curated, a refined version of the benchmark improved through a rigorous purification pipeline. On this enhanced benchmark, state-of-the-art models achieve success rates nearing 75% on complex tasks (15% improvement), reflecting that on-device GUI agents are actually closer to practical deployment than previously thought. We introduce our new SOTA model, Magma-R1- 3B, post-trained on just 2.4k curated samples using 60 hours of an H20 GPU (approximately $60). Despite being 200 times smaller in parameters, this model delivers performance comparable to Qwen3- VL-235B. We release both AndroidControl-Curated benchmark and Magma-R1 model to the research community, encouraging adoption of this enhanced benchmark to better reflect model capabilities and accelerate the development of robust, on-device virtual assistants.

Related papers

• OpenMobile: Building Open Mobile Agents with Task and Trajectory Synthesis [98.43366988856592]
  We present OpenMobile, an open-source framework that synthesizes high-quality task instructions and agent trajectories.<n>Agents trained on our data achieve competitive results across three dynamic mobile agent benchmarks.
  arXiv  Detail & Related papers  (2026-04-16T14:53:08Z)
• Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents [56.72789202127874]
  The paper introduces GUI-Owl-1.5, the latest native GUI agent model.<n>It supports a range of platforms (desktop, mobile, browser, and more) to enable cloud-edge collaboration and real-time interaction.<n>It achieves state-of-the-art results on more than 20+ GUI benchmarks on open-source models.
  arXiv  Detail & Related papers  (2026-02-15T01:52:19Z)
• UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoning [51.54456545661045]
  We introduce the Instruction-as-Reasoning paradigm, treating instructions as dynamic analytical pathways that offer distinct perspectives.<n>To achieve this, we propose a two-stage training framework: supervised fine-tuning and reinforcement learning.<n>Our resulting models, UI-Ins-7B and UI-Ins-32B, achieve state-of-the-art results on five challenging grounding benchmarks.
  arXiv  Detail & Related papers  (2025-10-23T07:18:32Z)
• Hi-Agent: Hierarchical Vision-Language Agents for Mobile Device Control [72.43808515668947]
  We introduce Hi-Agent, a trainable hierarchical vision-language agent for mobile control.<n>Hi-Agent features a high-level reasoning model and a low-level action model that are jointly optimized.<n>Hi-Agent achieves a new State-Of-The-Art (SOTA) 87.9% task success rate on the Android-in-the-Wild (AitW) benchmark.
  arXiv  Detail & Related papers  (2025-10-16T07:38:21Z)
• Mobile-Agent-v3: Fundamental Agents for GUI Automation [59.775510710011325]
  This paper introduces a foundational GUI agent model that achieves state-of-the-art performance among open-source end-to-end models.<n>We propose Mobile-Agent-v3, a general-purpose GUI agent framework that further improves performance to 73.3 on AndroidWorld and 37.7 on OSWorld.
  arXiv  Detail & Related papers  (2025-08-21T00:39:12Z)
• Phi-Ground Tech Report: Advancing Perception in GUI Grounding [55.44475003199647]
  textbfPhi-Ground model family achieves state-of-the-art performance across all five grounding benchmarks.<n>textittextbf43.2 on ScreenSpot-pro and textittextbf27.2 on UI-Vision.
  arXiv  Detail & Related papers  (2025-07-31T17:59:09Z)
• OSUniverse: Benchmark for Multimodal GUI-navigation AI Agents [0.0]
  OSUniverse is a benchmark of complex, multimodal desktop-oriented tasks for advanced GUI-navigation AI agents.<n>We divide the tasks in increasing levels of complexity, from basic precision clicking to multistep, multiapplication tests requiring dexterity, precision, and clear thinking from the agent.<n>The benchmark can be scored manually, but we also introduce an automated validation mechanism that has an average error rate less than 2%.
  arXiv  Detail & Related papers  (2025-05-06T14:29:47Z)
• Does Chain-of-Thought Reasoning Help Mobile GUI Agent? An Empirical Study [4.18969040567543]
  This paper presents the first empirical study on the effectiveness of reasoning-enabled vision-language models (VLMs) in mobile GUI agents.<n>We evaluate two pairs of commercial models--Gemini 2.0 Flash and Claude 3.7 Sonnet--comparing their base and reasoning-enhanced versions across two benchmarks.<n>We surprisingly find the Claude 3.7 Sonnet reasoning model achieves state-of-the-art performance on AndroidWorld.
  arXiv  Detail & Related papers  (2025-03-21T01:52:43Z)
• MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devices [73.46317110474064]
  MobileVLM is a competent multimodal vision language model (MMVLM) targeted to run on mobile devices. It comprises a set of language models at the scale of 1.4B and 2.7B parameters, trained from scratch, a multimodal vision model that is pre-trained in the CLIP fashion.
  arXiv  Detail & Related papers  (2023-12-28T08:21:24Z)
• Autonomous Large Language Model Agents Enabling Intent-Driven Mobile GUI Testing [17.24045904273874]
  We propose DroidAgent, an autonomous GUI testing agent for Android. It is based on Large Language Models and support mechanisms such as long- and short-term memory. DroidAgent achieved 61% activity coverage, compared to 51% for current state-of-the-art GUI testing techniques.
  arXiv  Detail & Related papers  (2023-11-15T01:59:40Z)
• AutoDroid: LLM-powered Task Automation in Android [32.241570727243534]
  We introduce AutoDroid, a mobile task automation system capable of handling arbitrary tasks on any Android application without manual efforts. The main components include a functionality-aware UI representation method that bridges the UI with the LLM. We evaluate its performance on a new benchmark for memory-augmented Android task automation with 158 common tasks.
  arXiv  Detail & Related papers  (2023-08-29T13:02:30Z)
• Multi-Objective Improvement of Android Applications [10.660480034605243]
  We write tests for 21 versions of 7 Android apps, creating a new benchmark for performance improvements. We use Genetic improvement, a search-based technique that navigates the space of software variants to find improved software. We were able to improve execution time by up to 35%, and memory use by up to 33% in these apps.
  arXiv  Detail & Related papers  (2023-08-22T12:26:43Z)
• PP-PicoDet: A Better Real-Time Object Detector on Mobile Devices [13.62426382827205]
  PP-PicoDet family of real-time object detectors achieves superior performance on object detection for mobile devices. Models achieve better trade-offs between accuracy and latency compared to other popular models.
  arXiv  Detail & Related papers  (2021-11-01T12:53:17Z)
• Real-Time Execution of Large-scale Language Models on Mobile [49.32610509282623]
  We find the best model structure of BERT for a given computation size to match specific devices. Our framework can guarantee the identified model to meet both resource and real-time specifications of mobile devices. Specifically, our model is 5.2x faster on CPU and 4.1x faster on GPU with 0.5-2% accuracy loss compared with BERT-base.
  arXiv  Detail & Related papers  (2020-09-15T01:59:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.