Covering Human Action Space for Computer Use: Data Synthesis and Benchmark

• URL: http://arxiv.org/abs/2605.12501v1
• Date: Tue, 12 May 2026 17:59:58 GMT
• Title: Covering Human Action Space for Computer Use: Data Synthesis and Benchmark
• Abstract summary: Computer-use agents (CUAs) automate on-screen work, as illustrated by GPT-5.4 and Claude.<n>Yet their reliability on complex, low-frequency interactions is still poor, limiting user trust.<n>We propose a new benchmark CUActSpot for evaluating models' capabilities on complex interactions.
• Score: 59.01879944842542
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Computer-use agents (CUAs) automate on-screen work, as illustrated by GPT-5.4 and Claude. Yet their reliability on complex, low-frequency interactions is still poor, limiting user trust. Our analysis of failure cases from advanced models suggests a long-tail pattern in GUI operations, where a relatively small fraction of complex and diverse interactions accounts for a disproportionate share of task failures. We hypothesize that this issue largely stems from the scarcity of data for complex interactions. To address this problem, we propose a new benchmark CUActSpot for evaluating models' capabilities on complex interactions across five modalities: GUI, text, table, canvas, and natural image, as well as a variety of actions (click, drag, draw, etc.), covering a broader range of interaction types than prior click-centric benchmarks that focus mainly on GUI widgets. We also design a renderer-based data-synthesis pipeline: scenes are automatically generated for each modality, screenshots and element coordinates are recorded, and an LLM produces matching instructions and action traces. After training on this corpus, our Phi-Ground-Any-4B outperforms open-source models with fewer than 32B parameters. We will release our benchmark, data, code, and models at https://github.com/microsoft/Phi-Ground.git

Related papers

• GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments [20.28422479068168]
  We introduce GUI-CC, a benchmark that evaluates contextual consistency of GUI world models as agent environments.<n>We construct 500 offline trajectory tasks from GUIOdyssey and 200 emulator-verified online tasks across 30 mobile apps.<n>Experiments show that plausible single-step generation does not guarantee reliable environment simulation.
  arXiv  Detail & Related papers  (2026-08-30T05:19:43Z)
• WinDeskGround: A Benchmark for Robust GUI Grounding in Complex Multi-Window Desktop Environments [49.98994180610182]
  Multimodal Large Language Models (MLLMs) have revolutionized GUI automation, yet their efficacy is largely established on idealized, single-layer interfaces.<n>This paper identifies a critical reliability gap: state-of-the-art agents face distinct challenges in real-world desktop environments.<n>We introduce WinDeskGround, a novel benchmark and synthesis framework tailored for evaluating GUI grounding robustness.
  arXiv  Detail & Related papers  (2026-05-13T02:48:52Z)
• PlotCraft: Pushing the Limits of LLMs for Complex and Interactive Data Visualization [82.96200364977737]
  We introduce PlotCraft, a new benchmark featuring 1k challenging visualization tasks.<n>PlotCraft is structured around seven high-level visualization tasks and encompasses 48 distinct chart types.<n>It is the first to systematically evaluate both single-turn generation and multi-turn refinement across a diverse spectrum of task complexities.
  arXiv  Detail & Related papers  (2025-10-15T10:14:39Z)
• Phi-Ground Tech Report: Advancing Perception in GUI Grounding [55.44475003199647]
  textbfPhi-Ground model family achieves state-of-the-art performance across all five grounding benchmarks.<n>textittextbf43.2 on ScreenSpot-pro and textittextbf27.2 on UI-Vision.
  arXiv  Detail & Related papers  (2025-07-31T17:59:09Z)
• Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis [57.371814877372515]
  Graphical user interface (GUI) grounding remains a critical bottleneck in computer use agent development.<n>We introduce OSWorld-G, a comprehensive benchmark comprising 564 finely annotated samples across diverse task types.<n>We synthesize and release the largest computer use grounding dataset Jedi, which contains 4 million examples.
  arXiv  Detail & Related papers  (2025-05-19T15:09:23Z)
• ToolACE: Winning the Points of LLM Function Calling [139.07157814653638]
  ToolACE is an automatic agentic pipeline designed to generate accurate, complex, and diverse tool-learning data.<n>We demonstrate that models trained on our synthesized data, even with only 8B parameters, achieve state-of-the-art performance on the Berkeley Function-Calling Leaderboard.
  arXiv  Detail & Related papers  (2024-09-02T03:19:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.