MobiFlow: Real-World Mobile Agent Benchmarking through Trajectory Fusion

• URL: http://arxiv.org/abs/2604.09587v1
• Date: Sat, 28 Feb 2026 14:30:33 GMT
• Title: MobiFlow: Real-World Mobile Agent Benchmarking through Trajectory Fusion
• Abstract summary: MobiFlow is an evaluation framework built on tasks drawn from arbitrary third-party applications.<n>It can effectively compress the state space, support dynamic interaction, and better align with real-world third-party application scenarios.<n>Compared with AndroidWorld, MobiFlow's evaluation results show higher alignment with human assessments.
• Score: 7.022084181221544
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Mobile agents can autonomously complete user-assigned tasks through GUI interactions. However, existing mainstream evaluation benchmarks, such as AndroidWorld, operate by connecting to a system-level Android emulator and provide evaluation signals based on the state of system resources. In real-world mobile-agent scenarios, however, many third-party applications do not expose system-level APIs to determine whether a task has succeeded, leading to a mismatch between benchmarks and real-world usage and making it difficult to evaluate model performance accurately. To address these issues, we propose MobiFlow, an evaluation framework built on tasks drawn from arbitrary third-party applications. Using an efficient graph-construction algorithm based on multi-trajectory fusion, MobiFlow can effectively compress the state space, support dynamic interaction, and better align with real-world third-party application scenarios. MobiFlow covers 20 widely used third-party applications and comprises 240 diverse real-world tasks, with enriched evaluation metrics. Compared with AndroidWorld, MobiFlow's evaluation results show higher alignment with human assessments and can guide the training of future GUI-based models under real workloads.

Related papers

• DuMateBench: Evaluating Autonomous Agents in Complex Real-World Workflows [69.12339622053588]
  We introduce DuMateBench, a real-session benchmark reconstructed from anonymized and privacy-screened user sessions.<n>The resulting benchmark comprises 200 tasks spanning 8 broad scenarios and 17 fine-grained capability categories.<n>We execute these tasks in isolated Docker containers injected with three forms of real-world environmental complexity: Insufficient, Unstable, and Noisy.
  arXiv  Detail & Related papers  (2026-08-27T02:36:23Z)
• Flower Hub: A Reproducible Benchmarking Platform for Federated Learning in Simulation and Deployment [51.83961778298564]
  We present Flower Hub, a platform for publishing, discovering, and executing decentralized and federated applications.<n>We show how it enables reproducible benchmarking by packaging benchmarks as executable, versioned applications with standardized metadata, pinned dependencies, and explicit evaluation.
  arXiv  Detail & Related papers  (2026-08-25T20:09:44Z)
• Xiaomi-GUI-0 Technical Report [39.05385476532886]
  Xiaomi-GUI-0 is a native multimodal GUI agent for real mobile environments.<n>It is trained through a three-stage pipeline of supervised fine-tuning, step-level reinforcement learning, and agentic reinforcement learning.<n>It achieves 72.0% success on RealMobile and 78.9% on AndroidWorld, while substantially improving execution stability and abnormal-state recognition in real-world tasks.
  arXiv  Detail & Related papers  (2026-06-30T09:36:35Z)
• AISE-Bench: A Full-Cycle Curated Benchmark for Information Seeking on Academic Knowledge Graphs [55.8213573429699]
  AISE-Bench is a real-world, full-cycle annotated benchmark for information seeking on academic knowledge graphs.<n>AISE-Bench release contains 1,133 QA pairs, including query, full API execution, validated parameters, and source-grounded answers with reference links.<n>We develop a comprehensive evaluation protocol measuring answer quality, reference grounding, API-planning correctness, and execution success.
  arXiv  Detail & Related papers  (2026-06-16T07:59:46Z)
• AndroidDaily: A Verifiable Benchmark for Mobile GUI Agents on Real-World Closed-Source Applications [36.71396790403699]
  We introduce AndroidDaily, a large-scale benchmark comprising 350 realistic daily-use tasks across 94 high-frequency Android applications.<n>We propose Guideline-grounded Reviewer for Automatic Diagnostic Evaluation (GRADE), a process-aware evaluator built on a three-tiered system of observable external guidelines.<n>GRADE produces step-level diagnostic judgments, turning long-horizon, open-ended mobile interactions into verifiable evaluation without relying on hidden internal states.
  arXiv  Detail & Related papers  (2026-05-26T23:19:42Z)
• How Mobile World Model Guides GUI Agents? [51.29718003718467]
  We train world models across four modalities: delta text, full text, diffusion-based images, and renderable code.<n>These models achieve SoTA performance on both MobileWorldBench and Code2WorldBench.<n>For overconfident mobile agents with low action entropy, posterior self-reflection provides limited gains.
  arXiv  Detail & Related papers  (2026-05-11T10:49:31Z)
• MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments [19.665566262516275]
  AndroidWorld has emerged as the dominant benchmark due to its reproducible environment and deterministic evaluation.<n>MobileWorld is a substantially more challenging benchmark designed to reflect real-world usage through 201 tasks.
  arXiv  Detail & Related papers  (2025-12-22T14:31:28Z)
• OS-Oracle: A Comprehensive Framework for Cross-Platform GUI Critic Models [54.44308299945632]
  We introduce OS-Oracle that makes three core contributions: a scalable data pipeline for cross-platform GUI critic data; a two-stage training paradigm combining supervised fine-tuning and consistency-preserving group relative policy optimization; and OS-Critic Bench, a holistic benchmark for evaluating critic model performance across Mobile, Web, and Desktop platforms.<n>The resulting critic model, OS-Oracle-7B, achieves state-of-the-art performance among open-source VLMs on OS-Critic Bench, and surpasses proprietary models on the mobile domain.
  arXiv  Detail & Related papers  (2025-12-18T08:29:50Z)
• What Limits Virtual Agent Application? OmniBench: A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities [56.646832992178105]
  We introduce OmniBench, a cross-platform, graph-based benchmark with an automated pipeline for synthesizing tasks of controllable complexity.<n>We present OmniEval, a multidimensional evaluation framework that includes subtask-level evaluation, graph-based metrics, and comprehensive tests across 10 capabilities.<n>Our dataset contains 36k graph-structured tasks across 20 scenarios, achieving a 91% human acceptance rate.
  arXiv  Detail & Related papers  (2025-06-10T15:59:38Z)
• Mobile-Bench-v2: A More Realistic and Comprehensive Benchmark for VLM-based Mobile Agents [33.899782380901314]
  VLM-based mobile agents are increasingly popular due to their capabilities to interact with smartphone GUIs and XML-structured texts.<n>Existing online benchmarks struggle with obtaining stable reward signals due to dynamic environmental changes.<n>Mobile-Bench-v2 includes a common task split, with offline multi-path evaluation to assess the agent's ability to obtain step rewards.
  arXiv  Detail & Related papers  (2025-05-17T07:58:34Z)
• Beyond Pass or Fail: Multi-Dimensional Benchmarking of Foundation Models for Goal-based Mobile UI Navigation [15.80796682874844]
  We propose Sphinx, a benchmark for evaluation of foundation models (FMs) in industrial settings of user interface ( UI) navigation.<n>We evaluate 8 FMs with 20 different configurations using both Google Play applications and WeChat's internal UI test cases.<n>Our results show that existing FMs universally struggle with goal-based testing tasks, primarily due to insufficient UI-specific capabilities.
  arXiv  Detail & Related papers  (2025-01-06T09:10:11Z)
• SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation [89.24729958546168]
  Smartphone agents are increasingly important for helping users control devices efficiently.<n>We present SPA-Bench, a comprehensive SmartPhone Agent Benchmark designed to evaluate (M)LLM-based agents.
  arXiv  Detail & Related papers  (2024-10-19T17:28:48Z)
• NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking [65.24988062003096]
  We present NAVSIM, a framework for benchmarking vision-based driving policies. Our simulation is non-reactive, i.e., the evaluated policy and environment do not influence each other. NAVSIM enabled a new competition held at CVPR 2024, where 143 teams submitted 463 entries, resulting in several new insights.
  arXiv  Detail & Related papers  (2024-06-21T17:59:02Z)
• Mobile-Env: Building Qualified Evaluation Benchmarks for LLM-GUI Interaction [28.53259866617677]
  We introduce Mobile-Env, a comprehensive toolkit tailored for creating GUI benchmarks in the Android mobile environment. We collect an open-world task set across various real-world apps and a fixed world set, WikiHow, which captures a significant amount of dynamic online contents. Our findings reveal that even advanced models struggle with tasks that are relatively simple for humans.
  arXiv  Detail & Related papers  (2023-05-14T12:31:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.