"Are We Done Yet?": A Vision-Based Judge for Autonomous Task Completion of Computer Use Agents

• URL: http://arxiv.org/abs/2511.20067v1
• Date: Tue, 25 Nov 2025 08:40:33 GMT
• Title: "Are We Done Yet?": A Vision-Based Judge for Autonomous Task Completion of Computer Use Agents
• Abstract summary: We present an autonomous evaluation and feedback framework that uses vision-language models to assess task completion directly from screenshots and task descriptions.<n>Our framework achieves up to 73 percent accuracy in task success detection and yields an average relative improvement of 27 percent in overall task success when evaluator feedback is applied.
• Score: 15.119045051735633
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Computer Use Agents (CUAs) are designed to autonomously operate digital interfaces, yet they often fail to reliably determine whether a given task has been completed. We present an autonomous evaluation and feedback framework that uses vision-language models to assess task completion directly from screenshots and task descriptions. Our dataset covers 42 built-in macOS applications and 1,260 human-labeled tasks across a wide range of scenarios. Our framework achieves up to 73 percent accuracy in task success detection and yields an average relative improvement of 27 percent in overall task success when evaluator feedback is applied. These results show that vision-based evaluation can serve as an effective feedback mechanism that improves the reliability and self-correction of autonomous computer-use agents.

Related papers

• Efficient Test-Time Adaptation through Human-AI Interaction [112.5131603022122]
  We propose test-time adaptation through human-agent interaction (TAHI)<n>We adapt agents to 30 individuals in two high-utility domains, writing and visual creation, on a total of 600 tasks.<n>Our agents improve solo task success by 4.5-20.9% within only tens of tasks.
  arXiv  Detail & Related papers  (2026-09-03T17:33:18Z)
• Reinforcement Learning for Computer-Use Agents with Autonomous Evaluation [15.119045051735633]
  We propose an RL fine-tuning framework that uses autonomous vision-language evaluation as a scalable supervision signal for GUI agents.<n>Because autonomous evaluators are imperfect, we model their feedback as a noisy binary reward channel and derive a noise-corrected reward estimator.<n>Experiments show that corrected evaluator rewards outperform both zero-shot baselines and raw evaluator rewards, improving success rates by an average of 12.6 percentage points over zero-shot performance and 5.1 points over raw evaluator fine-tuning.
  arXiv  Detail & Related papers  (2026-06-23T12:46:29Z)
• Running the Gauntlet: Re-evaluating the Capabilities of Agents Beyond Familiar Environments [67.2234316079859]
  GauntletBench is a web-based benchmark for evaluating agent generalisation in challenging scenarios.<n>It focuses on three underexplored capabilities (temporal perception, graphical understanding, and 3D reasoning) across five less-covered professional applications.<n>Our empirical results reveal that frontier agentic systems remain far from achieving human-level performance.
  arXiv  Detail & Related papers  (2026-06-12T12:32:24Z)
• Mind the Gap: Can Frontier LLMs Pass a Standardized Office Proficiency Exam? [90.39633603556767]
  Large Language Model (LLM) agents for computer automation is accelerating, yet their ability to navigate complex, professional-grade productivity software is largely untested.<n>To quantify this capability, we introduce an evaluation based on China's National Computer Rank Examination (NCRE)<n>We benchmark 7 frontier LLMs and observe stark limitations: single-turn models score a maximum of 36.6%.<n>A stronger agentic system with execution feedback, iterative repair, and broader Office automation access reaches 68.8%, but remains below the 95.5% community-reference score used as a scoring sanity check.
  arXiv  Detail & Related papers  (2026-06-09T14:59:14Z)
• Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces [126.23612941699565]
  Terminal-Bench 2.0 is a benchmark composed of 89 tasks in computer terminal environments inspired by problems from real world.<n>We show that frontier models and agents score less than 65% on the benchmark.<n>We publish the dataset and evaluation harness to assist developers and researchers in future work at https://www.tbench.ai/.
  arXiv  Detail & Related papers  (2026-01-17T01:29:30Z)
• UI-CUBE: Enterprise-Grade Computer Use Agent Benchmarking Beyond Task Accuracy to Operational Reliability [35.18016233072556]
  We present UI-CUBE (UiPath Computer Use BEnchmark), a benchmark comprising 226 tasks across two difficulty tiers.<n>Our evaluation covers simple UI interactions (136 tasks) and complex including copy-paste tasks (50 tasks) and enterprise application scenarios (40 tasks)<n>We find that while current CUAs can manipulate individual interface elements, they cannot yet function as reliable workflow automation tools.
  arXiv  Detail & Related papers  (2025-11-21T10:47:22Z)
• Exploring Autonomous Agents: A Closer Look at Why They Fail When Completing Tasks [8.218266805768687]
  We present a benchmark of 34 representative programmable tasks designed to rigorously assess autonomous agents.<n>We evaluate three popular open-source agent frameworks combined with two LLM backbones, observing a task completion rate of approximately 50%.<n>We develop a three-tier taxonomy of failure causes aligned with task phases, highlighting planning errors, task execution issues, and incorrect response generation.
  arXiv  Detail & Related papers  (2025-08-18T17:55:22Z)
• Evaluating Uncertainty and Quality of Visual Language Action-enabled Robots [13.26825865228582]
  We propose eight uncertainty metrics and five quality metrics specifically designed for VLA models for robotic manipulation tasks.<n>We assess their effectiveness through a large-scale empirical study involving 908 successful task executions from three state-of-the-art VLA models.
  arXiv  Detail & Related papers  (2025-07-22T22:15:59Z)
• UI-Evol: Automatic Knowledge Evolving for Computer Use Agents [23.21178608410048]
  We propose UI-Evol, a plug-and-play module for autonomous GUI knowledge evolution.<n> UI-Evol consists of two stages: a Retrace Stage that extracts faithful objective action sequences from actual agent-environment interactions, and a Critique Stage that refines existing knowledge.<n>Our results demonstrate that UI-Evol not only significantly boosts task performance but also addresses a previously overlooked issue of high behavioral standard deviation in computer use agents.
  arXiv  Detail & Related papers  (2025-05-28T04:32:05Z)
• AutoEval: A Practical Framework for Autonomous Evaluation of Mobile Agents [5.995751996623217]
  We propose AutoEval, an evaluation framework which tests mobile agents without any manual effort.<n>Our approach designs a UI state change representation which can be used to automatically generate task reward signals.<n>We also evaluate state-of-the-art mobile agents using our framework, providing insights into their performance and limitations.
  arXiv  Detail & Related papers  (2025-03-04T08:44:30Z)
• Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance [95.03771007780976]
  We tackle the challenge of developing proactive agents capable of anticipating and initiating tasks without explicit human instructions.<n>First, we collect real-world human activities to generate proactive task predictions.<n>These predictions are labeled by human annotators as either accepted or rejected.<n>The labeled data is used to train a reward model that simulates human judgment.
  arXiv  Detail & Related papers  (2024-10-16T08:24:09Z)
• WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks [85.95607119635102]
  Large language models (LLMs) can mimic human-like intelligence.<n>WorkArena++ is designed to evaluate the planning, problem-solving, logical/arithmetic reasoning, retrieval, and contextual understanding abilities of web agents.
  arXiv  Detail & Related papers  (2024-07-07T07:15:49Z)
• 360$^\circ$REA: Towards A Reusable Experience Accumulation with 360° Assessment for Multi-Agent System [71.96888731208838]
  We argue that a comprehensive evaluation and accumulating experience from evaluation feedback is an effective approach to improving system performance.<n>We propose Reusable Experience Accumulation with 360$circ$ Assessment (360$circ$REA), a hierarchical multi-agent framework inspired by corporate organizational practices.
  arXiv  Detail & Related papers  (2024-04-08T14:43:13Z)
• WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks? [83.19032025950986]
  We study the use of large language model-based agents for interacting with software via web browsers. WorkArena is a benchmark of 33 tasks based on the widely-used ServiceNow platform. BrowserGym is an environment for the design and evaluation of such agents.
  arXiv  Detail & Related papers  (2024-03-12T14:58:45Z)
• Automated system to measure Tandem Gait to assess executive functions in children [0.0]
  This work focuses on assessing motor function in children by analyzing their gait movements. We have devised a computer vision-based assessment system that only requires a camera which makes it easier to employ in school or home environments. The results highlight the efficacy of proposed work for automating the assessment of children's performances by achieving 76.61% classification accuracy.
  arXiv  Detail & Related papers  (2020-12-15T23:12:13Z)
• SideInfNet: A Deep Neural Network for Semi-Automatic Semantic Segmentation with Side Information [83.03179580646324]
  This paper proposes a novel deep neural network architecture, namely SideInfNet. It integrates features learnt from images with side information extracted from user annotations. To evaluate our method, we applied the proposed network to three semantic segmentation tasks and conducted extensive experiments on benchmark datasets.
  arXiv  Detail & Related papers  (2020-02-07T06:10:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.