CUA-Suite: Massive Human-annotated Video Demonstrations for Computer-Use Agents

• URL: http://arxiv.org/abs/2603.24440v1
• Date: Wed, 25 Mar 2026 15:52:56 GMT
• Title: CUA-Suite: Massive Human-annotated Video Demonstrations for Computer-Use Agents
• Abstract summary: We introduce CUA-Suite, a large-scale ecosystem of expert video demonstrations and dense annotations for professional desktop computer-use agents.<n>At its core is VideoCUA, which provides approximately 10,000 human-demonstrated tasks across 87 diverse applications.<n> CUA-Suite further provides two complementary resources: UI-Vision, a rigorous benchmark for evaluating grounding and planning capabilities in CUAs, and GroundCUA, a large-scale grounding dataset with 56K annotated screenshots.
• Score: 17.596111848044476
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Computer-use agents (CUAs) hold great promise for automating complex desktop workflows, yet progress toward general-purpose agents is bottlenecked by the scarcity of continuous, high-quality human demonstration videos. Recent work emphasizes that continuous video, not sparse screenshots, is the critical missing ingredient for scaling these agents. However, the largest existing open dataset, ScaleCUA, contains only 2 million screenshots, equating to less than 20 hours of video. To address this bottleneck, we introduce CUA-Suite, a large-scale ecosystem of expert video demonstrations and dense annotations for professional desktop computer-use agents. At its core is VideoCUA, which provides approximately 10,000 human-demonstrated tasks across 87 diverse applications with continuous 30 fps screen recordings, kinematic cursor traces, and multi-layerfed reasoning annotations, totaling approximately 55 hours and 6 million frames of expert video. Unlike sparse datasets that capture only final click coordinates, these continuous video streams preserve the full temporal dynamics of human interaction, forming a superset of information that can be losslessly transformed into the formats required by existing agent frameworks. CUA-Suite further provides two complementary resources: UI-Vision, a rigorous benchmark for evaluating grounding and planning capabilities in CUAs, and GroundCUA, a large-scale grounding dataset with 56K annotated screenshots and over 3.6 million UI element annotations. Preliminary evaluation reveals that current foundation action models struggle substantially with professional desktop applications (~60% task failure rate). Beyond evaluation, CUA-Suite's rich multimodal corpus supports emerging research directions including generalist screen parsing, continuous spatial control, video-based reward modeling, and visual world models. All data and models are publicly released.

Related papers

• Qwen-CUA: Native Computer Use for (almost) Everything [104.78368489343713]
  We introduce Qwen-CUA, a native computer-use agent with a mixture-of-experts backbone.<n>It observes only screenshots and acts through keyboard and mouse events, without DOM trees, accessibility metadata, or task-specific APIs.<n>Its scaffold maintains up to 20 active screenshots and folds older visual history in fixed-size blocks to retain recent evidence.
  arXiv  Detail & Related papers  (2026-08-03T15:04:20Z)
• 3rd Place at CVPR 2026 CASTLE Challenge: Agentic Multi-View Long-Context Video Understanding via Hierarchical Knowledge Graph Retrieval [0.8594140167290097]
  This paper presents our winning methodology for the CASTLE 2026 Challenge at the CVPR 2026 Workshop.<n>Our team secured third place globally.<n>The underlying dataset consists of over 600 hours synchronized footage captured by 15 ego and exo camera sources.
  arXiv  Detail & Related papers  (2026-06-01T09:01:32Z)
• Video-Based Reward Modeling for Computer-Use Agents [40.27314571412647]
  We study reward modeling from execution video: a sequence of sequences from an agent trajectory independent of the agent's internal reasoning or actions.<n>We introduce Execution Video Reward 53k (ExeVR-53k), a dataset of 53k high-quality video--task--reward triplets.<n>Building on these components, we fine-tune an Execution Video Model (ExeVRM) that takes only a user instruction and a video-execution sequence to predict task success.
  arXiv  Detail & Related papers  (2026-03-10T19:17:22Z)
• VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos [62.29924199978745]
  VideoAgentTrek is a scalable pipeline that automatically mines training data from publicly available screen-recorded videos at web scale.<n>Our approach addresses a key challenge: raw videos contain implicit demonstrations but lack explicit action labels.<n>applied to 39,000 YouTube tutorial videos, our pipeline generates 1.52 million interaction steps automatically.
  arXiv  Detail & Related papers  (2025-10-22T11:25:48Z)
• ARC-Hunyuan-Video-7B: Structured Video Comprehension of Real-World Shorts [56.75723197779384]
  ARC-Hunyuan-Video is a multimodal model that processes visual, audio, and textual signals end-to-end for structured comprehension.<n>Our model is capable of multi-granularity timestamped video captioning and summarization, open-ended video question answering, temporal video grounding, and video reasoning.
  arXiv  Detail & Related papers  (2025-07-28T15:52:36Z)
• Scalable Video-to-Dataset Generation for Cross-Platform Mobile Agents [57.59830804627066]
  We introduce MONDAY, a large-scale dataset of 313K annotated frames from 20K instructional videos capturing real-world mobile OS navigation.<n>Models that include MONDAY in their pre-training phases demonstrate robust cross-platform generalization capabilities.<n>We present an automated framework that leverages publicly available video content to create comprehensive task datasets.
  arXiv  Detail & Related papers  (2025-05-19T02:39:03Z)
• OmAgent: A Multi-modal Agent Framework for Complex Video Understanding with Task Divide-and-Conquer [14.503628667535425]
  processing extensive videos presents significant challenges due to the vast data and processing demands. We develop OmAgent, efficiently stores and retrieves relevant video frames for specific queries. It features an Divide-and-Conquer Loop capable of autonomous reasoning. We have endowed it with greater autonomy and a robust tool-calling system, enabling it to accomplish even more intricate tasks.
  arXiv  Detail & Related papers  (2024-06-24T13:05:39Z)
• VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding [28.316828641898375]
  VideoAgent: 1) constructs a structured memory to store both the generic temporal event descriptions and object-centric tracking states of the video. 2) given an input task query, it employs tools including video segment localization and object memory querying along with other visual foundation models to interactively solve the task.
  arXiv  Detail & Related papers  (2024-03-18T05:07:59Z)
• Spatio-temporal Prompting Network for Robust Video Feature Extraction [74.54597668310707]
  Frametemporal is one of the main challenges in the field of video understanding. Recent approaches exploit transformer-based integration modules to obtain quality-of-temporal information. We present a neat and unified framework called N-Temporal Prompting Network (NNSTP) It can efficiently extract video features by adjusting the input features in the network backbone.
  arXiv  Detail & Related papers  (2024-02-04T17:52:04Z)
• Dense Video Object Captioning from Disjoint Supervision [77.47084982558101]
  We propose a new task and model for dense video object captioning. This task unifies spatial and temporal localization in video. We show how our model improves upon a number of strong baselines for this new task.
  arXiv  Detail & Related papers  (2023-06-20T17:57:23Z)
• Dense-Caption Matching and Frame-Selection Gating for Temporal Localization in VideoQA [96.10612095576333]
  We propose a video question answering model which effectively integrates multi-modal input sources and finds the temporally relevant information to answer questions. Our model is also comprised of dual-level attention (word/object and frame level), multi-head self-cross-integration for different sources (video and dense captions), and which pass more relevant information to gates. We evaluate our model on the challenging TVQA dataset, where each of our model components provides significant gains, and our overall model outperforms the state-of-the-art by a large margin.
  arXiv  Detail & Related papers  (2020-05-13T16:35:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.