A History-Aware Visually Grounded Critic for Computer Use Agents

• URL: http://arxiv.org/abs/2606.11078v1
• Date: Tue, 09 Jun 2026 16:39:10 GMT
• Title: A History-Aware Visually Grounded Critic for Computer Use Agents
• Abstract summary: HiViG is a test-time framework built around a multimodal critic trained on real GUI trajectories.<n>At test time, HiViG integrates the critic into the policy decision loop to provide macro-action history.<n>Visually grounded critique verifies raw execution coordinates against the current screenshot to intercept errors before execution.
• Score: 70.50900668601969
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Various test-time interventions for Computer Use Agents (CUAs), including critic models, have been developed to improve performance through pre-execution action evaluation in complex Graphical User Interface (GUI) environments. However, existing critics suffer from two key limitations: they (1) focus primarily on short-sighted decision loops (e.g., forgetting earlier actions) and (2) lack the visual grounding needed to detect flawed actions (e.g., clicking wrong UI elements). To address these, we introduce HiViG, a History-aware Visually Grounded test-time framework, built around a multimodal critic trained on real GUI trajectories to abstract past interactions into a compact record and to evaluate actions with visual grounding. At test time, HiViG integrates the critic into the policy decision loop to provide macro-action history, which summarizes the policy's completed achievements, and visually grounded critique, which verifies raw execution coordinates against the current screenshot to intercept errors before execution. Across web, mobile, and desktop benchmarks, HiViG consistently outperforms existing scalar and verbal critics, improving average success rates over the strongest baseline by 5.8% for Qwen3-VL-32B and 9.0% for Gemini-3-Flash, and demonstrates strong cross-platform generalization. Ablations show that macro-action history mitigates short-sighted planning and visually grounded critique reduces execution errors, with both components being critical for test-time scaling in long-horizon GUI tasks.

Related papers

• VisCritic: Visual State Comparison as Process Reward for GUI Agents [1.0998907972211756]
  We introduce VisCritic, a visual process reward framework that verifies actions by directly comparing pre-action and post-action screenshots.<n>VisCritic employs a Siamese vision transformer to extract change-aware representations, coupled with an Action-Aware Critic Head.<n>Experiments and offline analyses demonstrate that VisCritic serves as a plug-and-play enhancement for diverse GUI agents.
  arXiv  Detail & Related papers  (2026-06-23T12:57:02Z)
• VisualThink-VLA: Visual Intermediate Reasoning for Effective and Low-Latency Vision-Language-Action Policies [58.65913948991329]
  We present VISUALTHINK-VLA, a visual intermediate-reasoning framework for accurate, low-latency VLA policies.<n>Our bootstrapping philosophy is to guide action with effective visual thinking.<n>It bootstraps action prediction through a compact visual-evidence interface that preserves spatial precision while avoiding decoding overhead.
  arXiv  Detail & Related papers  (2026-05-28T14:36:53Z)
• GUITestScape: Towards Open-set Evaluation on Exploratory GUI Testing [23.562168946315804]
  We present an interactive benchmark covering 61 real-world Android applications and 508 preset defects spanning interaction and display types.<n>We introduce GUIJudge, an open-set evaluator that decomposes an agent's testing trajectory into independently diagnosable capabilities.
  arXiv  Detail & Related papers  (2026-05-28T07:47:27Z)
• See, Point, Refine: Multi-Turn Approach to GUI Grounding with Visual Feedback [8.080056970922483]
  Computer Use Agents (CUAs) fundamentally rely on graphical user interface (GUI) grounding to translate language instructions into executable screen actions.<n>Existing approaches typically rely on single-shot coordinate prediction, which lacks a mechanism for error correction and often fails in high-density interfaces.<n>In this technical report, we conduct an empirical study of pixel-precise cursor localization in coding environments.<n>We evaluate our approach across GPT-5.4, Claude, and Qwen on a suite of complex coding benchmarks, demonstrating that multi-turn refinement significantly outperforms state-of-the-art single-shot models in both click precision and
  arXiv  Detail & Related papers  (2026-04-14T17:55:46Z)
• GEBench: Benchmarking Image Generation Models as GUI Environments [49.513441724802135]
  We introduce GEBench, a benchmark for evaluating dynamic interaction and temporal coherence in GUI generation.<n>GE-Score is a novel five-dimensional metric that assesses Goal Achievement, Interaction Logic, Content Consistency, UI Plausibility, and Visual Quality.<n>Our findings identify icon interpretation, text rendering, and localization precision as critical bottlenecks.
  arXiv  Detail & Related papers  (2026-02-09T18:52:02Z)
• GAIA: A Data Flywheel System for Training GUI Test-Time Scaling Critic Models [18.14416135619429]
  We propose the GUI Action Critic's Data Flywheel System (GAIA), a training framework that enables the models to have iterative critic capabilities.<n>Specifically, we train an Intuitive Critic Model (ICM) using positive and negative action examples from a base agent.<n>We conduct experiments on various datasets and demonstrate that the proposed ICM can improve the test-time performance of various closed-source and open-source models.
  arXiv  Detail & Related papers  (2026-01-26T06:29:41Z)
• OS-Oracle: A Comprehensive Framework for Cross-Platform GUI Critic Models [54.44308299945632]
  We introduce OS-Oracle that makes three core contributions: a scalable data pipeline for cross-platform GUI critic data; a two-stage training paradigm combining supervised fine-tuning and consistency-preserving group relative policy optimization; and OS-Critic Bench, a holistic benchmark for evaluating critic model performance across Mobile, Web, and Desktop platforms.<n>The resulting critic model, OS-Oracle-7B, achieves state-of-the-art performance among open-source VLMs on OS-Critic Bench, and surpasses proprietary models on the mobile domain.
  arXiv  Detail & Related papers  (2025-12-18T08:29:50Z)
• Rethinking Prompt Design for Inference-time Scaling in Text-to-Visual Generation [63.042451267669485]
  We propose Prompt Redesign for Inference-time Scaling, a framework that adaptively revises the prompt during inference in response to scaled visual generations.<n>We introduce a new verifier, element-level factual correction, which evaluates the alignment between prompt attributes and generated visuals at a fine-grained level.<n>Experiments on both text-to-image and text-to-video benchmarks demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2025-12-03T07:54:05Z)
• Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automation [83.92224427735859]
  We introduce a pre-operative critic mechanism that provides effective feedback prior to the actual execution.<n>We develop a reasoning-bootstrapping based data collection pipeline to create a GUI-Critic-Train and a GUI-Critic-Test.<n>Our model offers significant advantages in critic accuracy compared to current MLLMs.
  arXiv  Detail & Related papers  (2025-06-05T04:12:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.