PhoneWorld: Scaling Phone-Use Agent Environments

• URL: http://arxiv.org/abs/2605.29486v1
• Date: Thu, 28 May 2026 07:14:15 GMT
• Title: PhoneWorld: Scaling Phone-Use Agent Environments
• Abstract summary: PhoneWorld is a reusable pipeline that converts real GUI trajectories and screenshots into controllable phone-use environments.<n>It builds runnable mock Android apps backed by read-only app content and mutable state.<n>It covers 34 apps across 16 domains, spanning common consumer mobile behaviors such as search, browsing, shopping, booking, media, and social interaction.
• Score: 101.288508963127
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A central bottleneck for phone-use agents is that controllable, reproducible environments covering real mobile behavior are hard to build at scale. Existing mobile-agent benchmarks have made important progress on evaluation, but they do not by themselves provide a scalable way to construct many new phone-use environments. We present PhoneWorld, a reusable pipeline that converts real GUI trajectories and screenshots into controllable phone-use environments, executable tasks, automatic verifiers, and training rollouts. Rather than hand-building one mobile benchmark at a time, PhoneWorld uses real trajectories to recover which screens matter, how screens connect, which interactions must change environment state, and which user goals admit automatic verification. From these signals, it builds runnable mock Android apps backed by read-only app content and mutable state, then derives executable tasks, rule-based verifiers, and training rollouts from the same environments. In its current instantiation, PhoneWorld covers 34 apps across 16 domains, spanning common consumer mobile behaviors such as search, browsing, shopping, booking, media, and social interaction. Under a fixed training budget, replacing 10K steps from an auxiliary AndroidWorld corpus in an AndroidWorld-based baseline with broad PhoneWorld supervision improves all four evaluation benchmarks at once, raising HYMobileBench by 17.7 points, AndroidControl by 6.0 points, AndroidWorld by 14.7 points, and PhoneWorld by 52.5 points. We then study two additional scaling questions: increasing the amount of PhoneWorld supervision strongly improves PhoneWorld performance, and under a fixed PhoneWorld budget, expanding app coverage yields even larger gains. Overall, PhoneWorld shifts the focus from building one mobile benchmark at a time to scaling the supply of phone-use environments themselves.

Related papers

• Learning Panorama-Aware VLA for Mobile Manipulation with Whole-Body Teleoperation [55.724481205135675]
  Mobile manipulation poses two key challenges for vision-language-action policies.<n>We develop a whole-body teleoperation system and a panoramic-aware VLA policy.<n>We propose PanoVLA, a panorama-aware vision-language-action policy for mobile bimanual manipulation.
  arXiv  Detail & Related papers  (2026-08-03T14:03:26Z)
• Xiaomi-GUI-0 Technical Report [39.05385476532886]
  Xiaomi-GUI-0 is a native multimodal GUI agent for real mobile environments.<n>It is trained through a three-stage pipeline of supervised fine-tuning, step-level reinforcement learning, and agentic reinforcement learning.<n>It achieves 72.0% success on RealMobile and 78.9% on AndroidWorld, while substantially improving execution stability and abnormal-state recognition in real-world tasks.
  arXiv  Detail & Related papers  (2026-06-30T09:36:35Z)
• PhoneBuddy: Training Open Models for Agentic Phone Use [104.23243994660936]
  PhoneBuddy is a training recipe and open-model line for agentic phone use.<n>It combines a real-app environment with a mock-app environment, PhoneWorld, which reconstructs run mock apps from real GUI usage structure.<n>PhoneBuddy first builds a shared supervised fine-tuning stage from trajectories collected in both environments, then compares real-app RL against mixed RL.
  arXiv  Detail & Related papers  (2026-06-22T08:57:54Z)
• PhoneHarness: Harnessing Phone-Use Agents through Mixed GUI, CLI, and Tool Actions [73.69976712292683]
  PhoneHarness is a mixed-action benchmark and execution harness for studying phone-use agents on verifiable mobile.<n>Its benchmark, PhoneHarness Bench, evaluates whether agents complete tasks with observable side effects, not only whether they produce plausible final answers.<n>Our findings suggest that reliable phone automation depends on action-surface routing and verifiable execution, not only visual GUI control.
  arXiv  Detail & Related papers  (2026-06-12T15:01:32Z)
• How Mobile World Model Guides GUI Agents? [51.29718003718467]
  We train world models across four modalities: delta text, full text, diffusion-based images, and renderable code.<n>These models achieve SoTA performance on both MobileWorldBench and Code2WorldBench.<n>For overconfident mobile agents with low action entropy, posterior self-reflection provides limited gains.
  arXiv  Detail & Related papers  (2026-05-11T10:49:31Z)
• MobileDev-Bench: A Comprehensive Benchmark for Evaluating Language Models on Mobile Application Development [1.4302803674538154]
  We introduce MobileDev-Bench, a benchmark comprising 384 real-world issue-resolution tasks collected from 18 production mobile applications spanning Android Native (Java/Kotlin), React Native (TypeScript), and Flutter (Dart)<n>Each task pairs an authentic developer-reported issue with executable test patches, enabling fully automated validation of model-generated fixes within mobile build environments.<n> Evaluation of four state-of-the-art code-capable LLMs, GPT- 5.2, Claude Sonnet 4.5, Gemini Flash 2.5, and Qwen3-Coder, yields low end-to-end resolution rates of 3.39%
  arXiv  Detail & Related papers  (2026-03-26T02:31:03Z)
• MobileManiBench: Simplifying Model Verification for Mobile Manipulation [70.30578259859512]
  MobileManiBench is a large-scale benchmark for mobile-based robotic manipulation.<n>MobileManiBench features 2 mobile platforms (parallel-gripper and dexterous-hand robots), 2 synchronized cameras (head and right wrist), 630 objects in 20 categories, 5 skills (open, close, pull, push, pick) with over 100 tasks performed in 100 realistic scenes, yielding 300K trajectories.
  arXiv  Detail & Related papers  (2026-02-05T02:49:52Z)
• MobileBench-OL: A Comprehensive Chinese Benchmark for Evaluating Mobile GUI Agents in Real-World Environment [17.207878975582556]
  MobileBench-OL is an online benchmark with 1080 tasks from 80 Chinese apps.<n>It measures task execution, complex reasoning, and noise robustness of agents.<n>MobileBench-OL shows significant room for improvement to meet real-world requirements.
  arXiv  Detail & Related papers  (2026-01-28T07:49:48Z)
• MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments [19.665566262516275]
  AndroidWorld has emerged as the dominant benchmark due to its reproducible environment and deterministic evaluation.<n>MobileWorld is a substantially more challenging benchmark designed to reflect real-world usage through 201 tasks.
  arXiv  Detail & Related papers  (2025-12-22T14:31:28Z)
• Mobile-Agent-v3: Fundamental Agents for GUI Automation [59.775510710011325]
  This paper introduces a foundational GUI agent model that achieves state-of-the-art performance among open-source end-to-end models.<n>We propose Mobile-Agent-v3, a general-purpose GUI agent framework that further improves performance to 73.3 on AndroidWorld and 37.7 on OSWorld.
  arXiv  Detail & Related papers  (2025-08-21T00:39:12Z)
• AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents [5.044046039265116]
  We present AndroidWorld, a fully functional Android environment that provides reward signals for 116 programmatic tasks across 20 real-world Android apps.<n>Unlike existing interactive environments, which provide a static test set, AndroidWorld dynamically constructs tasks that are parameterized and expressed in natural language.<n>Our best agent can complete 30.6% of AndroidWorld's tasks, leaving ample room for future work.
  arXiv  Detail & Related papers  (2024-05-23T13:48:54Z)
• Global Tracking Transformers [76.58184022651596]
  We present a novel transformer-based architecture for global multi-object tracking. The core component is a global tracking transformer that operates on objects from all frames in the sequence. Our framework seamlessly integrates into state-of-the-art large-vocabulary detectors to track any objects.
  arXiv  Detail & Related papers  (2022-03-24T17:58:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.