GUITestScape: Towards Open-set Evaluation on Exploratory GUI Testing

• URL: http://arxiv.org/abs/2605.29532v1
• Date: Thu, 28 May 2026 07:47:27 GMT
• Title: GUITestScape: Towards Open-set Evaluation on Exploratory GUI Testing
• Abstract summary: We present an interactive benchmark covering 61 real-world Android applications and 508 preset defects spanning interaction and display types.<n>We introduce GUIJudge, an open-set evaluator that decomposes an agent's testing trajectory into independently diagnosable capabilities.
• Score: 23.562168946315804
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Exploratory GUI testing is a particularly demanding setting for MLLM agents: without predefined test scripts, an agent must autonomously navigate an application and discover defects through its own interaction. However, current evaluation falls short on two fronts. First, existing benchmarks focus almost exclusively on interaction defects, leaving display defects outside the evaluation frame. Second, evaluation protocols are bound to predefined defect annotations, collapsing the testing process into a single end-state judgment that conflates qualitatively distinct failure modes. To address these challenges, we present GUITestScape, an interactive benchmark covering 61 real-world Android applications and 508 preset defects spanning interaction and display types, and introduce GUIJudge, an open-set evaluator that decomposes an agent's testing trajectory into independently diagnosable capabilities. Experimental results demonstrate that GUIJudge achieves reliable process-aware evaluation beyond predefined annotations, substantially outperforming all baselines. Benchmarking on GUITestScape further reveals that detection remains the critical bottleneck for existing models across both defect types, and that integrating GUIJudge's verifiers into existing agents significantly boosts their detection performance without retraining.

Related papers

• From Benchmark Performance to Tool Deployment: Human-in-the-Loop Anomaly Detection [28.67239775192971]
  We evaluate 19 unsupervised anomaly detection models on the BowTie dataset.<n>We observe that model performance is less stable than typically reported on standard benchmarks.<n>We deploy a unified human-in-the-loop framework for manufactured-part inspection.
  arXiv  Detail & Related papers  (2026-08-07T21:31:20Z)
• ScrambleToolBench: Agents Search Exhaustively Even When Their Own Map Points to the Next Step [74.83535434786145]
  We introduce ScrambleToolBench, an interactive terminal benchmark designed to isolate behavioral reasoning.<n>The benchmark requires agents to uncover hidden tool behaviors entirely through trial-and-error interaction.<n>Our evaluation of state-of-the-art language models reveals that successful initial discovery does not translate into robust adaptation.
  arXiv  Detail & Related papers  (2026-08-03T15:07:46Z)
• A History-Aware Visually Grounded Critic for Computer Use Agents [70.50900668601969]
  HiViG is a test-time framework built around a multimodal critic trained on real GUI trajectories.<n>At test time, HiViG integrates the critic into the policy decision loop to provide macro-action history.<n>Visually grounded critique verifies raw execution coordinates against the current screenshot to intercept errors before execution.
  arXiv  Detail & Related papers  (2026-06-09T16:39:10Z)
• DiagEval: Trajectory-Conditioned Diagnosis for Reliable Software Evaluation with GUI Agents [16.577116170272934]
  We present DiagEval, a trajectory-conditioned diagnostic evaluation protocol for post-failure GUI-agent evaluation of interactive software.<n>DiagEval reuses the failed trajectory to choose targeted diagnostic probes and aggregates their outcomes into an internal attribution signal.<n>We evaluate DiagEval on WebDevJudge-Unit and RealDevBench across multiple GUI-agent evaluators and LLM backbones.
  arXiv  Detail & Related papers  (2026-05-17T13:22:22Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation [72.01173512175531]
  KnowU-Bench is an online benchmark for personalized mobile agents.<n>It covers 42 general GUI tasks, 86 personalized tasks, and 64 proactive tasks.<n>Our experiments reveal a striking degradation: agents that excel at explicit task execution fall below 50% under vague instructions.
  arXiv  Detail & Related papers  (2026-04-09T16:50:50Z)
• Benchmark Designers Should "Train on the Test Set" to Expose Exploitable Non-Visual Shortcuts [49.99400612296149]
  We find that models can ace many benchmarks without strong visual understanding.<n>This is especially problematic for vision-centric benchmarks that are meant to require visual inputs.<n>We adopt a diagnostic principle for benchmark design: if a benchmark can be gamed, it will be gamed.
  arXiv  Detail & Related papers  (2025-11-06T18:43:21Z)
• Demystifying deep search: a holistic evaluation with hint-free multi-hop questions and factorised metrics [89.1999907891494]
  We present WebDetective, a benchmark of hint-free multi-hop questions paired with a controlled Wikipedia sandbox.<n>Our evaluation of 25 state-of-the-art models reveals systematic weaknesses across all architectures.<n>We develop an agentic workflow, EvidenceLoop, that explicitly targets the challenges our benchmark identifies.
  arXiv  Detail & Related papers  (2025-10-01T07:59:03Z)
• Reactive Bottom-Up Testing [15.280664862119565]
  We introduce a new paradigm that we call Reactive Bottom-Up Testing.<n>Our insight is that function-level testing is necessary but not sufficient for the validation of vulnerabilities in functions.<n>We develop a three-stage bottom-up testing scheme that identifies likely-vulnerable functions and generates type- and context-aware harnesses.
  arXiv  Detail & Related papers  (2025-09-03T20:54:43Z)
• Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automation [83.92224427735859]
  We introduce a pre-operative critic mechanism that provides effective feedback prior to the actual execution.<n>We develop a reasoning-bootstrapping based data collection pipeline to create a GUI-Critic-Train and a GUI-Critic-Test.<n>Our model offers significant advantages in critic accuracy compared to current MLLMs.
  arXiv  Detail & Related papers  (2025-06-05T04:12:36Z)
• What You See Is What You Get: Attention-based Self-guided Automatic Unit Test Generation [3.8244417073114003]
  We propose Attention-based Self-guided Automatic Unit Test GenERation (AUGER) approach.<n>AUGER contains two stages: defect detection and error triggering.<n>It makes great improvements by 4.7% to 35.3% in terms of F1-score and Precision in defect detection.<n>It can trigger 23 to 84 more errors than state-of-the-art (SOTA) approaches in unit test generation.
  arXiv  Detail & Related papers  (2024-12-01T14:28:48Z)
• Gamified GUI testing with Selenium in the IntelliJ IDE: A Prototype Plugin [0.559239450391449]
  This paper presents GIPGUT: a prototype of a gamification plugin for IntelliJ IDEA. The plugin enhances testers' engagement with typically monotonous and tedious tasks through achievements, rewards, and profile customization. The results indicate high usability and positive reception of the gamification elements.
  arXiv  Detail & Related papers  (2024-03-14T20:11:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.