PhoneBuddy: Training Open Models for Agentic Phone Use

• URL: http://arxiv.org/abs/2606.23049v2
• Date: Wed, 24 Jun 2026 02:20:35 GMT
• Title: PhoneBuddy: Training Open Models for Agentic Phone Use
• Abstract summary: PhoneBuddy is a training recipe and open-model line for agentic phone use.<n>It combines a real-app environment with a mock-app environment, PhoneWorld, which reconstructs run mock apps from real GUI usage structure.<n>PhoneBuddy first builds a shared supervised fine-tuning stage from trajectories collected in both environments, then compares real-app RL against mixed RL.
• Score: 104.23243994660936
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Phones are becoming an important execution surface for general-purpose agents, but training open models for reliable phone use remains difficult because the environment that matters at deployment, real devices running real apps, is slow, stateful, side-effectful, and hard to reset or verify, while scalable mock environments only approximate real behavior. We present PhoneBuddy, a training recipe and open-model line for agentic phone use that combines a real-app environment with a mock-app environment, PhoneWorld, which reconstructs runnable mock apps from real GUI usage structure. PhoneBuddy first builds a shared supervised fine-tuning stage from trajectories collected in both environments, then compares real-app RL against mixed RL across both environments. Across a 150-task human evaluation on real phones spanning apps, mini-apps, and cross-app workflows, task success rate improves from 36.67\% after supervised fine-tuning to 40.67\% after real-app RL and 45.33\% after mixed RL. On AndroidWorld, the same progression rises from 60.3\% to 77.2\% to 83.2\%. These results show that mock-app training is not a replacement for real-app RL, but a complementary source of scalable, resettable, and automatically checked interaction. The gains are strongest on app and mini-app tasks, while long-horizontal cross-app workflows remain an important open challenge.

Related papers

• Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents [27.09949595397669]
  Qwen-UI-Agent is a real-world centric foundation GUI agent spanning mobile, computer-use, web, and DeepSearch environments.<n>Its unified action space interleaves GUI operations with CLI execution and generates batched actions in a single model turn.<n>Online RL supports training on trajectories exceeding 100 turns, with over 10,000 environments accelerating rollout.
  arXiv  Detail & Related papers  (2026-07-30T13:58:41Z)
• Xiaomi-GUI-0 Technical Report [39.05385476532886]
  Xiaomi-GUI-0 is a native multimodal GUI agent for real mobile environments.<n>It is trained through a three-stage pipeline of supervised fine-tuning, step-level reinforcement learning, and agentic reinforcement learning.<n>It achieves 72.0% success on RealMobile and 78.9% on AndroidWorld, while substantially improving execution stability and abnormal-state recognition in real-world tasks.
  arXiv  Detail & Related papers  (2026-06-30T09:36:35Z)
• PhoneHarness: Harnessing Phone-Use Agents through Mixed GUI, CLI, and Tool Actions [73.69976712292683]
  PhoneHarness is a mixed-action benchmark and execution harness for studying phone-use agents on verifiable mobile.<n>Its benchmark, PhoneHarness Bench, evaluates whether agents complete tasks with observable side effects, not only whether they produce plausible final answers.<n>Our findings suggest that reliable phone automation depends on action-surface routing and verifiable execution, not only visual GUI control.
  arXiv  Detail & Related papers  (2026-06-12T15:01:32Z)
• PhoneWorld: Scaling Phone-Use Agent Environments [101.288508963127]
  PhoneWorld is a reusable pipeline that converts real GUI trajectories and screenshots into controllable phone-use environments.<n>It builds runnable mock Android apps backed by read-only app content and mutable state.<n>It covers 34 apps across 16 domains, spanning common consumer mobile behaviors such as search, browsing, shopping, booking, media, and social interaction.
  arXiv  Detail & Related papers  (2026-05-28T07:14:15Z)
• AndroidDaily: A Verifiable Benchmark for Mobile GUI Agents on Real-World Closed-Source Applications [36.71396790403699]
  We introduce AndroidDaily, a large-scale benchmark comprising 350 realistic daily-use tasks across 94 high-frequency Android applications.<n>We propose Guideline-grounded Reviewer for Automatic Diagnostic Evaluation (GRADE), a process-aware evaluator built on a three-tiered system of observable external guidelines.<n>GRADE produces step-level diagnostic judgments, turning long-horizon, open-ended mobile interactions into verifiable evaluation without relying on hidden internal states.
  arXiv  Detail & Related papers  (2026-05-26T23:19:42Z)
• ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents [54.04035382782041]
  textbfClawGUI-RL provides the first open-source GUI agent RL infrastructure with validated support for both parallel virtual environments and real physical devices.<n>textbfClawGUI-Eval enforces a fully standardized evaluation pipeline across 6 benchmarks and 11+ models.<n>textbfClawGUI-Agent brings trained agents to Android, HarmonyOS, and iOS through 12+ chat platforms with hybrid CLI-GUI control and persistent personalized memory.
  arXiv  Detail & Related papers  (2026-04-13T17:52:04Z)
• Generalization in Online Reinforcement Learning for Mobile Agents [32.98335803990582]
  We introduce textbfAndroidWorld-Generalization, a benchmark for evaluating zero-shot generalization to unseen task instances, templates, and applications.<n>Experiments on AndroidWorld-Generalization show that RL enables a 7B- parameter VLM agent to surpass supervised fine-tuning baselines.<n>To support and fair comparison, we open-source the full RL training system, including the environment, task suite, models, prompt configurations, and the underlying infrastructure.
  arXiv  Detail & Related papers  (2026-03-08T03:08:11Z)
• Trajectory2Task: Training Robust Tool-Calling Agents with Synthesized Yet Verifiable Data for Complex User Intents [52.30603055218294]
  Trajectory2Task is a verifiable data generation pipeline for studying tool use at scale under three realistic user scenarios.<n>It converts valid tool-call trajectories into user-facing tasks with controlled intent adaptations.<n>We benchmark seven state-of-the-art LLMs on the generated complex user scenario tasks and observe frequent failures.
  arXiv  Detail & Related papers  (2026-01-28T00:36:13Z)
• MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments [19.665566262516275]
  AndroidWorld has emerged as the dominant benchmark due to its reproducible environment and deterministic evaluation.<n>MobileWorld is a substantially more challenging benchmark designed to reflect real-world usage through 201 tasks.
  arXiv  Detail & Related papers  (2025-12-22T14:31:28Z)
• AppVLM: A Lightweight Vision Language Model for Online App Control [39.91330570886891]
  We introduce AppVLM, a lightweight Vision-Language Model (VLM)<n>First, we fine-tune it offline on the AndroidControl dataset.<n>Then, we refine its policy by collecting data from the AndroidWorld environment.
  arXiv  Detail & Related papers  (2025-02-10T12:32:21Z)
• Reinforcement Learning for Long-Horizon Interactive LLM Agents [56.9860859585028]
  Interactive digital agents (IDAs) leverage APIs of stateful digital environments to perform tasks in response to user requests.<n>We present a reinforcement learning (RL) approach that trains IDAs directly in their target environments.<n>We derive LOOP, a data- and memory-efficient variant of proximal policy optimization.
  arXiv  Detail & Related papers  (2025-02-03T18:35:42Z)
• FLaRe: Achieving Masterful and Adaptive Robot Policies with Large-Scale Reinforcement Learning Fine-Tuning [74.25049012472502]
  FLaRe is a large-scale Reinforcement Learning framework that integrates robust pre-trained representations, large-scale training, and gradient stabilization techniques. Our method aligns pre-trained policies towards task completion, achieving state-of-the-art (SoTA) performance on previously demonstrated and on entirely novel tasks and embodiments.
  arXiv  Detail & Related papers  (2024-09-25T03:15:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.