ComponentBench: Diagnosing Component-Level Failures in Computer-Use Agents

• URL: http://arxiv.org/abs/2608.18307v1
• Date: Tue, 18 Aug 2026 20:38:26 GMT
• Title: ComponentBench: Diagnosing Component-Level Failures in Computer-Use Agents
• Abstract summary: We present a benchmark and diagnostic pipeline-level evaluation of computeruse agents on modern web UIs.<n> ComponentBench is organized around a libraryagnostic of 97 canonical UI components instantiated as 2,910.<n>Changing only the observation and action space shifts task success by more than 30% for the same model.
• Score: 9.98905103274257
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Current evaluation of computer-use agents is split between long-horizon workflow benchmarks and atomic GUI-grounding tests. This leaves an under-instrumented middle layer: realistic component-centered interactions (e.g., toggle a button set) that are short enough to diagnose and rich enough to capture the burdens of modern interfaces. We present ComponentBench, a benchmark and diagnostic pipeline for component-level evaluation of computer-use agents on modern web UIs. ComponentBench is organized around a library-agnostic ontology of 97 canonical UI components instantiated as 2,910 programmatically verified tasks across widely used component libraries, paired with cleaned human reference trajectories that enable evaluation of both task success and interaction efficiency. Beyond task collection, we introduce a scalable pipeline for auditing realized structural difficulty after implementation and synthesizing structured failure analyses across tasks and component families. Evaluating seven models -- GPT-5.4, Gemini 3 Flash, GPT-5.4 mini, GPT-5 mini, Gemini 3.1 Flash-Lite, Qwen3-VL-235B, and UI-TARS-1.5-7B -- across four observation and action spaces, we show that these design choices critically impact performance. Within a single shared harness, changing only the observation and action space shifts task success by more than 30% for the same model: GPT-5 mini falls from 83.1% with accessibility-tree observations to 48.9% with coordinate-only Pixel control. Moreover, even the fastest configuration takes 3.7x as long as the matched human reference, and spatial manipulations that are trivial for humans continue to challenge current agents.

Related papers

• Long-Range Indirect Control-Flow Prediction in Stripped Binaries via Dual Virtual Hubs and Multi-Task Graph Learning [44.79384874347874]
  We present ICFlowNet, a unified framework for long-range ICF prediction in stripped binaries.<n>ICFlowNet introduces candidate-aware Dual Virtual Hubs, a Global Code Hub and a Global Data Hub.<n>We also develop a leakage-aware, noise-controlled pipeline with package-level splits, function-level mnemonic-hash deduplication, and a clean test protocol.
  arXiv  Detail & Related papers  (2026-09-03T02:17:23Z)
• Desktop-Delta Bench: Do Computer-Use Models Understand Desktop GUI Transitions? [3.9561795512466413]
  Current benchmarks primarily measure end-task success or single-frame grounding.<n>This is difficult because inference, remote input, app rendering, and screenshot capture are asynchronous.<n>We introduce Desktop-Delta Bench (DDB), an offline step-level benchmark with 2,013 human-verified instances from novel, multi-app Linux trajectories.
  arXiv  Detail & Related papers  (2026-07-28T17:49:51Z)
• An Intelligent-Cloud Edge Multimodal Interaction System for Robots [4.1212219465063304]
  This paper presents a cloud-edge multimodal interaction framework that integrates an enhanced YOLO-based gesture detector with coordinated large language model (LLM) and vision-language model (VLM) agents.<n>The cloud layer performs gesture detection, scene understanding, multimodal fusion, and action planning, whereas the TonyPi robot locally handles data acquisition, communication, action execution, and feedback.
  arXiv  Detail & Related papers  (2026-07-16T07:39:18Z)
• OSWorld2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks [98.07146747892239]
  OSWorld 2.0 is a benchmark of 108 long-horizon computer-use across everyday and professional tasks.<n>Each task represents a realistic end-to-end workflow that takes human users a median of about 1.6 hours to complete.
  arXiv  Detail & Related papers  (2026-06-28T17:59:17Z)
• Covering Human Action Space for Computer Use: Data Synthesis and Benchmark [59.01879944842542]
  Computer-use agents (CUAs) automate on-screen work, as illustrated by GPT-5.4 and Claude.<n>Yet their reliability on complex, low-frequency interactions is still poor, limiting user trust.<n>We propose a new benchmark CUActSpot for evaluating models' capabilities on complex interactions.
  arXiv  Detail & Related papers  (2026-05-12T17:59:58Z)
• UniToolCall: Unifying Tool-Use Representation, Data, and Evaluation for LLM Agents [22.52508596251479]
  We present UniToolCall, a unified framework for tool learning that standardizes the entire pipeline from construction and dataset generation to evaluation.<n>We convert 7 public benchmarks into a unified Query-Action--Observation-Answer (QAOA) representation with fine-grained evaluation at the function-call, turn, and conversation levels.<n>Under the Anchoror-heavy Hybrid-20 setting, achieves 93.0% single-turn Strict Precision, outperforming commercial models including GPT, Gemini, and Claude.
  arXiv  Detail & Related papers  (2026-04-13T14:43:47Z)
• SciAgentGym: Benchmarking Multi-Step Scientific Tool-use in LLM Agents [100.12367115920121]
  We introduce SciGymAgent, a scalable interactive environment featuring 1,780 domain-specific tools across four natural science disciplines.<n>We also present SciAgentBench, a tiered evaluation suite designed to stress-test agentic capabilities.
  arXiv  Detail & Related papers  (2026-02-13T14:58:18Z)
• GEBench: Benchmarking Image Generation Models as GUI Environments [49.513441724802135]
  We introduce GEBench, a benchmark for evaluating dynamic interaction and temporal coherence in GUI generation.<n>GE-Score is a novel five-dimensional metric that assesses Goal Achievement, Interaction Logic, Content Consistency, UI Plausibility, and Visual Quality.<n>Our findings identify icon interpretation, text rendering, and localization precision as critical bottlenecks.
  arXiv  Detail & Related papers  (2026-02-09T18:52:02Z)
• PinPoint3D: Fine-Grained 3D Part Segmentation from a Few Clicks [37.718136287542556]
  PinPoint3D is a novel interactive framework for fine-grained, multi-granularity 3D segmentation.<n>It generates precise part-level masks from only a few user point clicks.<n>Our work represents a significant step towards more nuanced and precise machine perception and interaction.
  arXiv  Detail & Related papers  (2025-09-30T09:05:29Z)
• Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis [57.371814877372515]
  Graphical user interface (GUI) grounding remains a critical bottleneck in computer use agent development.<n>We introduce OSWorld-G, a comprehensive benchmark comprising 564 finely annotated samples across diverse task types.<n>We synthesize and release the largest computer use grounding dataset Jedi, which contains 4 million examples.
  arXiv  Detail & Related papers  (2025-05-19T15:09:23Z)
• Disentangle Your Dense Object Detector [82.22771433419727]
  Deep learning-based dense object detectors have achieved great success in the past few years and have been applied to numerous multimedia applications such as video understanding. However, the current training pipeline for dense detectors is compromised to lots of conjunctions that may not hold. We propose Disentangled Dense Object Detector (DDOD), in which simple and effective disentanglement mechanisms are designed and integrated into the current state-of-the-art detectors.
  arXiv  Detail & Related papers  (2021-07-07T00:52:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.