AndroidReality: How Far Are Mobile Agents from the Real World?

• URL: http://arxiv.org/abs/2608.07775v1
• Date: Fri, 07 Aug 2026 21:47:43 GMT
• Title: AndroidReality: How Far Are Mobile Agents from the Real World?
• Abstract summary: We introduce AndroidReality, a perturbation-based framework for evaluating and improving the robustness of mobile agents.<n>We build a perturbed mobile benchmark on top of AndroidWorld with realistic and controllable perturbation injections.<n>Our evaluation reveals substantial robustness gaps and four recurring error categories.
• Score: 7.828168033702986
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Mobile agents have achieved promising results on clean online benchmarks such as AndroidWorld, yet their performance often degrades sharply in real-world deployment due to environmental variations and imperfect interface conditions. In this work, we introduce AndroidReality, a perturbation-based framework for evaluating and improving the robustness of mobile agents. Through a Markov Decision Process (MDP) perspective, we organize real-world interface variability into a principled taxonomy of perturbations along three axes: state, transition, and action. Guided by this taxonomy, we build a perturbed mobile benchmark on top of AndroidWorld with realistic and controllable perturbation injections, enabling systematic robustness evaluation of mobile agents. Our evaluation reveals substantial robustness gaps and four recurring error categories, motivating a simple training-free Test-Time Introspective Recovery (TTIR) mechanism that mitigates these failures on both perturbed and clean settings. Together, these results position robustness as a missing dimension in mobile agent evaluation and establish benchmark perturbation as an effective tool for both stress testing and surfacing latent weaknesses of mobile agents.

Related papers

• Are Android GUI Agents Robust Against Runtime Anomalies? AnTrap: Evaluating Agents in Dynamic Adversarial Environments [19.009627117174947]
  We introduce AnTrap, a comprehensive benchmark that injects dynamic perturbations into agent execution trajectories.<n>We propose a taxonomy organizing real-world anomalies into four layers (State, Thinking, Action and Round) with ten fine-grained subcategories.<n>We also develop a construction pipeline that preserves task solvability while introducing realistic adversarial conditions.
  arXiv  Detail & Related papers  (2026-08-25T05:57:25Z)
• WorldSimProbe: Diagnosing Simulator Faithfulness in Action-Conditioned World Models for Embodied Manipulation [85.15806404962193]
  Action-conditioned world models (ACWMs) promise to provide embodied AI with scalable predictive simulators for planning, policy evaluation, and data generation.<n>We evaluate ACWMs through the observable capabilities expected of physical simulators.<n>WorldSimProbe reveals systematic action-realization degradation across control variation, structured failures in interaction grounding and dynamics.
  arXiv  Detail & Related papers  (2026-08-10T08:48:06Z)
• Xiaomi-GUI-0 Technical Report [39.05385476532886]
  Xiaomi-GUI-0 is a native multimodal GUI agent for real mobile environments.<n>It is trained through a three-stage pipeline of supervised fine-tuning, step-level reinforcement learning, and agentic reinforcement learning.<n>It achieves 72.0% success on RealMobile and 78.9% on AndroidWorld, while substantially improving execution stability and abnormal-state recognition in real-world tasks.
  arXiv  Detail & Related papers  (2026-06-30T09:36:35Z)
• Toward Visually Realistic Simulation: A Benchmark for Evaluating Robot Manipulation in Simulation [62.51953630639423]
  Existing benchmarks lack visual realism, creating a large domain gap between simulation and reality.<n>We propose VISER, a visually realistic benchmark for evaluating robot manipulation in simulation.<n>VISER features a high-fidelity dataset of over 1,000 3D assets with physically-based rendering (PBR) materials, along with 3D scenes created from these assets through curated layouts or generation.
  arXiv  Detail & Related papers  (2026-05-07T14:13:05Z)
• Beyond State Consistency: Behavior Consistency in Text-Based World Models [58.216587360435305]
  We introduce a new behavior-aligned training paradigm aimed at improving the functional consistency between the world model and the real environment.<n> Experiments on WebShop and TextWorld show that BehR-based training improves long-term alignment in several settings.
  arXiv  Detail & Related papers  (2026-04-15T12:56:45Z)
• ManipArena: Comprehensive Real-world Evaluation of Reasoning-Oriented Generalist Robot Manipulation [66.02142169323521]
  Vision-Language-Action models and world models have recently emerged as promising paradigms for general-purpose robotic intelligence.<n>Existing benchmarks are largely simulator-centric, which provide controllability but fail to capture the reality gap caused by perception noise.<n>We introduce ManipArena, a standardized evaluation framework designed to bridge simulation and real-world execution.
  arXiv  Detail & Related papers  (2026-03-30T15:06:41Z)
• VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability Diagnostics [23.71786084060511]
  We introduce VenusBench-Mobile, a challenging online benchmark for evaluating general-purpose mobile GUI agents.<n>VenusBench-Mobile builds two core evaluation pillars: defining what to evaluate via user-intent-driven task design that reflects real mobile usage, and how to evaluate through a capability-oriented annotation scheme for fine-grained agent behavior analysis.
  arXiv  Detail & Related papers  (2026-02-06T18:54:42Z)
• MobileBench-OL: A Comprehensive Chinese Benchmark for Evaluating Mobile GUI Agents in Real-World Environment [17.207878975582556]
  MobileBench-OL is an online benchmark with 1080 tasks from 80 Chinese apps.<n>It measures task execution, complex reasoning, and noise robustness of agents.<n>MobileBench-OL shows significant room for improvement to meet real-world requirements.
  arXiv  Detail & Related papers  (2026-01-28T07:49:48Z)
• MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments [19.665566262516275]
  AndroidWorld has emerged as the dominant benchmark due to its reproducible environment and deterministic evaluation.<n>MobileWorld is a substantially more challenging benchmark designed to reflect real-world usage through 201 tasks.
  arXiv  Detail & Related papers  (2025-12-22T14:31:28Z)
• OS-Sentinel: Towards Safety-Enhanced Mobile GUI Agents via Hybrid Validation in Realistic Workflows [77.95511352806261]
  Computer-using agents powered by Vision-Language Models (VLMs) have demonstrated human-like capabilities in operating digital environments like mobile platforms.<n>We propose OS-Sentinel, a novel hybrid safety detection framework that combines a Formal Verifier for detecting explicit system-level violations with a Contextual Judge for assessing contextual risks and agent actions.
  arXiv  Detail & Related papers  (2025-10-28T13:22:39Z)
• RoHOI: Robustness Benchmark for Human-Object Interaction Detection [84.78366452133514]
  Human-Object Interaction (HOI) detection is crucial for robot-human assistance, enabling context-aware support.<n>We introduce the first benchmark for HOI detection, evaluating model resilience under diverse challenges.<n>Our benchmark, RoHOI, includes 20 corruption types based on the HICO-DET and V-COCO datasets and a new robustness-focused metric.
  arXiv  Detail & Related papers  (2025-07-12T01:58:04Z)
• The BrowserGym Ecosystem for Web Agent Research [151.90034093362343]
  BrowserGym ecosystem addresses the growing need for efficient evaluation and benchmarking of web agents.<n>We propose an extended BrowserGym-based ecosystem for web agent research, which unifies existing benchmarks from the literature.<n>We conduct the first large-scale, multi-benchmark web agent experiment and compare the performance of 6 state-of-the-art LLMs across 6 popular web agent benchmarks.
  arXiv  Detail & Related papers  (2024-12-06T23:43:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.