EvoGUI: An Evolution-Aware Benchmark for GUI State-Transition Understanding
Abstract Overview
EvoGUI is a diagnostic framework for GUI state-transition understanding that converts normalized GUI trajectories into three visual question answering probes: temporal ordering, inverse action/value prediction, and one-step successor discrimination. The benchmark instance, EvoGUI-Bench, is built from Mind2Web and WebLINX and contains 3,000 examples across 120 domains, with labels derived mechanically from trajectory order and logged actions after normalization. This design is intended to complement end-to-end GUI-agent evaluation by isolating transition-level reasoning from broader factors such as planning, grounding, and recovery. The paper evaluates 28 vision-language model configurations in a zero-shot setting and reports substantial remaining headroom on all three probes.
Novelty
The paper's main novelty is a scalable trajectory-derived framework that turns existing GUI logs into complementary transition-level VQA diagnostics without requiring new task-label annotation after normalization. It also introduces an evolution-aware benchmark that explicitly separates temporal ordering, inverse transition explanation, and immediate successor discrimination rather than relying only on end-to-end agent success.
Results
On EvoGUI-Bench, the best evaluated model reaches 60.4 EvoGain, showing that performance is well above chance but still far from saturated. The study finds that model scale and GUI specialization do not reliably predict transition-level performance, and that action-only prediction is generally easier than exact action-plus-value recovery. Control experiments further show that the benchmark depends materially on visual evidence, while one-step reachability (T3) remains challenging especially when distinguishing immediate successors from later states in the same workflow.
Key Points
- EvoGUI derives three GUI state-transition diagnostics directly from normalized trajectories, eliminating the need for additional task-label annotation after schema normalization.
- EvoGUI-Bench contains 3,000 instances across 120 domains from Mind2Web and WebLINX and is used to evaluate 28 zero-shot VLM configurations.
- Empirical results reveal substantial headroom in GUI transition understanding, showing that neither larger model size nor GUI-specific specialization consistently guarantees stronger performance.