OmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environments

• URL: http://arxiv.org/abs/2605.18758v1
• Date: Fri, 03 Apr 2026 08:57:06 GMT
• Title: OmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environments
• Abstract summary: We introduce OmniGUI, the first step-level benchmark designed to evaluate GUI agents in omni-modal smartphone environments.<n>The dataset encompasses 709 expert-demonstrated episodes (2,579 action steps) across 29 applications.<n>Our empirical evaluation reveals that while current models exhibit competency on visually static tasks, their action prediction performance degrades significantly in environments requiring synchronous temporal and auditory signals.
• Score: 5.613789567241434
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Current benchmarks for graphical user interface (GUI) agents predominantly rely on static screenshots. However, real-world smartphone interaction routinely requires agents to process transient audio cues and temporal video dynamics that are tightly coupled with the moment of action. To bridge this gap, we introduce OmniGUI, the first step-level benchmark designed to evaluate GUI agents in omni-modal smartphone environments. OmniGUI provides continuous, interleaved multimodal inputs comprising static images, synchronous audio, and video clips at every action step. The dataset encompasses 709 expert-demonstrated episodes (2,579 action steps) across 29 applications, systematically annotated with objective multimodal dependency levels. Because dedicated omni-modal GUI agent frameworks are currently in their nascent stage, we select foundational omni-modal models capable of natively processing interleaved inputs to serve as agent proxies for our initial baselines. Our empirical evaluation reveals that while current models exhibit competency on visually static tasks, their action prediction performance degrades significantly in environments requiring synchronous temporal and auditory signals. Furthermore, ablation studies isolate specific operational bottlenecks, notably cross-modal interference when processing task-irrelevant environmental noise. The complete dataset, evaluation pipeline, and baseline prompts are provided in the supplementary material. Project page: https://omni-gui.github.io.

Related papers

• JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition [51.15720718264743]
  We introduce JarvisGUI, a dynamic benchmark that evaluates GUI agents on cross-device requiring coordinated interaction across platforms.<n>JarvisGUI formulates GUI tasks as input-output transformations under a lightweight type system, which allows us to automatically compose multi-step, cross-device transformations.<n>We show that state-of-the-art open-source GUI agents struggle with the state-transfer awareness, cross-platform contextual reasoning, and long-horizon dependency management required for real-world usage.
  arXiv  Detail & Related papers  (2026-09-09T16:59:59Z)
• GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments [20.28422479068168]
  We introduce GUI-CC, a benchmark that evaluates contextual consistency of GUI world models as agent environments.<n>We construct 500 offline trajectory tasks from GUIOdyssey and 200 emulator-verified online tasks across 30 mobile apps.<n>Experiments show that plausible single-step generation does not guarantee reliable environment simulation.
  arXiv  Detail & Related papers  (2026-08-30T05:19:43Z)
• OmniAssistBench: Assistant-style Interaction Benchmark for Omni-LLMs [25.570460595046058]
  Recent omni-modal large language models ( Omni-LLMs) show great potential as real-time video assistants.<n>Unlike traditional passive video understanding, interactive assistants should actively combine visual states, user goals, and prior knowledge to provide effective help.
  arXiv  Detail & Related papers  (2026-08-21T17:59:52Z)
• Sandboxed Coding Agents are Competitive Omni-modal Task Solvers [22.269016330825128]
  We show that coding agents with only text+image access and a sandboxed tool-use interface can match SOTA native omnimodal models.<n>Our trajectory analysis suggests that their strength comes from writing code and orchestrating tools to extract relevant evidence.<n>We introduce Code-X, a training recipe with the OmniCoding trajectory dataset and verifiable reward, and provide baselines on Qwen-3.5-9B and Qwen-3.6-27B.
  arXiv  Detail & Related papers  (2026-05-30T07:04:27Z)
• MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction [76.4461698685681]
  Mini-o 4.5 is our latest effort towards human-level realtime streaming interaction.<n>It is based on Omni-CPM, a unified streaming framework that aligns omni-modal perception and outputs along a shared temporal axis.<n>With a total of 9B parameters, Mini-o 4.5 approaches Gemini 2.5 Flash in vision computation capabilities, delivering stateofthe-art open computation performance scale.
  arXiv  Detail & Related papers  (2026-04-30T04:05:43Z)
• AutoGUI-v2: A Comprehensive Multi-Modal GUI Functionality Understanding Benchmark [32.66632642377623]
  AutoGUI-v2 is a benchmark designed to evaluate deep GUI functionality understanding and interaction outcome prediction.<n>We construct the benchmark using a novel VLM-human collaborative pipeline that parses screenshots into hierarchical functional regions.<n>AutoGUI-v2 rigorously tests agents on region and element-level semantics, grounding, and dynamic state prediction.
  arXiv  Detail & Related papers  (2026-04-27T13:06:27Z)
• ANCHOR: Branch-Point Data Generation for GUI Agents [52.22377425487]
  End-to-end GUI agents for real desktop environments require large amounts of high-quality interaction data.<n>We present a trajectory expansion framework Anchor that bootstraps scalable desktop supervision from a small set of verified seed demonstrations.<n>Experiments on standard desktop benchmarks, OSWorld and WindowsAgentArena, show that models fine-tuned on our expanded corpus achieve consistent improvements.
  arXiv  Detail & Related papers  (2026-02-06T19:55:26Z)
• What Limits Virtual Agent Application? OmniBench: A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities [56.646832992178105]
  We introduce OmniBench, a cross-platform, graph-based benchmark with an automated pipeline for synthesizing tasks of controllable complexity.<n>We present OmniEval, a multidimensional evaluation framework that includes subtask-level evaluation, graph-based metrics, and comprehensive tests across 10 capabilities.<n>Our dataset contains 36k graph-structured tasks across 20 scenarios, achieving a 91% human acceptance rate.
  arXiv  Detail & Related papers  (2025-06-10T15:59:38Z)
• Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction [69.57190742976091]
  Aguvis is a vision-based framework for autonomous GUI agents.<n>It standardizes cross-platform interactions and incorporates structured reasoning via inner monologue.<n>It achieves state-of-the-art performance across offline and real-world online benchmarks.
  arXiv  Detail & Related papers  (2024-12-05T18:58:26Z)
• GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding [73.9254861755974]
  This paper introduces a new dataset, termed GUI-World, which features meticulously crafted Human-MLLM annotations.<n>We evaluate the capabilities of current state-of-the-art MLLMs, including Image LLMs and Video LLMs, in understanding various types of GUI content.
  arXiv  Detail & Related papers  (2024-06-16T06:56:53Z)
• You Only Look at Screens: Multimodal Chain-of-Action Agents [37.118034745972956]
  Auto-GUI is a multimodal solution that directly interacts with the interface. We propose a chain-of-action technique to help the agent decide what action to execute. We evaluate our approach on a new device-control benchmark AITW with 30$K$ unique instructions.
  arXiv  Detail & Related papers  (2023-09-20T16:12:32Z)
• Dynamic Graph Representation Learning for Video Dialog via Multi-Modal Shuffled Transformers [89.00926092864368]
  We present a semantics-controlled multi-modal shuffled Transformer reasoning framework for the audio-visual scene aware dialog task. We also present a novel dynamic scene graph representation learning pipeline that consists of an intra-frame reasoning layer producing-semantic graph representations for every frame. Our results demonstrate state-of-the-art performances on all evaluation metrics.
  arXiv  Detail & Related papers  (2020-07-08T02:00:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.