StepReflect: Structured UI Transition Reflection for Mobile GUI Agents
- URL: http://arxiv.org/abs/2608.05587v1
- Date: Thu, 06 Aug 2026 04:17:27 GMT
- Title: StepReflect: Structured UI Transition Reflection for Mobile GUI Agents
- Abstract summary: We propose StepReflect, which formulates per-step GUI reflection as supervised structured prediction conditioned on explicit transition specifications and paired visual evidence.<n> offline, the resulting 8B model achieves 82.16% transition-level accuracy on AndroidWorld, exceeding zero-shot GPT-5.2 by 11.83 percentage points.<n>Online, across M3A, Agent-SAMA, MAI-UI-8B, and Seed-2.0-Pro, StepReflect achieves higher task success in three of four agent configurations and remains within one successful task of the GPT-5.2 Reflection Agent in the fourth.
- Score: 22.922734013100825
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Autonomous mobile GUI agents require accurate action reflection for reliable long-horizon execution. Existing approaches rely on open-ended multimodal reasoning after each action, which is costly and poorly matched to the structured nature of GUI state transitions. We propose StepReflect, which formulates per-step GUI reflection as supervised structured prediction conditioned on explicit transition specifications and paired visual evidence. StepReflect is trained through a staged pipeline combining supervised fine-tuning, teacher-student distillation, and preference- and reward-based refinement. Offline, the resulting 8B model achieves 82.16% transition-level accuracy on AndroidWorld, exceeding zero-shot GPT-5.2 by 11.83 percentage points under the same structured input. Online, across M3A, Agent-SAMA, MAI-UI-8B, and Seed-2.0-Pro, StepReflect achieves higher task success in three of four agent configurations and remains within one successful task of the GPT-5.2 Reflection Agent in the fourth. It also reduces paid API charges relative to GPT-based reflection in all four configurations. These results establish StepReflect as a practical, locally deployable alternative to repeated frontier-model reflection for long-horizon mobile GUI agents.
Related papers
- Reflection with Action-Induced Visual Differences for Desktop GUI Agents [29.424541963900598]
The Planner-Operator-Reflector (POR) framework is widely used in GUI agents to maintain objective alignment in complex tasks.<n>Existing reflectors collapse change detection and outcome verification into one step, leaving evidence implicit and yielding weakly grounded decisions.<n>We propose Evidence-First Reflection (EFR), a two-stage reflector that explicitly decouples action-induced visual differences extraction from outcome verification.
arXiv Detail & Related papers (2026-08-25T03:10:01Z) - A Task-State Representation for Long-Horizon Mobile GUI Agents [53.359524744953575]
Task-State Representation (TSR) is a training-free framework that explicitly decouples task state from sensory input.<n> Experiments across four mobile GUI benchmarks validate TSR's effectiveness, yielding up to a 12 absolute point increase in success rate.
arXiv Detail & Related papers (2026-07-01T06:37:21Z) - TIER: Trajectory-Invariant Execution Rewards for Multi-Step Tool Composition [62.56752617853322]
Outcome-based rewards provide only sparse feedback, while trajectory-supervised rewards depend on annotated reference solutions.<n>We propose TIER: Trajectory-Invariant Execution Rewards, a reward framework that derives supervision directly from function schemas and runtime execution.
arXiv Detail & Related papers (2026-05-16T03:47:26Z) - VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automation [98.38575149237442]
VLAA-GUI is a modular GUI framework built around three integrated components.<n>A mandatory Completeness Verifier enforces UI-observable success criteria and verification at every finish step.<n>A mandatory Loop Breaker provides multi-tier switching interaction mode after repeated failures.
arXiv Detail & Related papers (2026-04-23T07:42:37Z) - UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoning [51.54456545661045]
We introduce the Instruction-as-Reasoning paradigm, treating instructions as dynamic analytical pathways that offer distinct perspectives.<n>To achieve this, we propose a two-stage training framework: supervised fine-tuning and reinforcement learning.<n>Our resulting models, UI-Ins-7B and UI-Ins-32B, achieve state-of-the-art results on five challenging grounding benchmarks.
arXiv Detail & Related papers (2025-10-23T07:18:32Z) - GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Tasks [75.50160982584943]
We introduce a Process Reward Model that provides dense, step-by-step feedback to guide agents.<n>Gui-Shepherd is trained on a diverse large-scale data set of $52$k interactions.<n>We are the first to conduct a systematic study of process supervision in GUI agents.
arXiv Detail & Related papers (2025-09-28T08:35:16Z) - MobileIPL: Enhancing Mobile Agents Thinking Process via Iterative Preference Learning [45.46445208254837]
Chain of Action-Planning Thoughts (CoaT) paradigm has been shown to improve the reasoning performance of VLM-based mobile agents in GUI tasks.<n>We propose an Iterative Preference Learning (IPL) that constructs a CoaT-tree through interative sampling, scores leaf nodes using rule-based reward, and backpropagates feedback to derive Thinking-level Direct Preference Optimization (T-DPO) pairs.
arXiv Detail & Related papers (2025-05-18T08:28:05Z) - UI-TARS: Pioneering Automated GUI Interaction with Native Agents [58.18100825673032]
This paper introduces UI-TARS, a native GUI agent model that solely perceives the screenshots as input and performs human-like interactions.<n>In the OSWorld benchmark, UI-TARS achieves scores of 24.6 with 50 steps and 22.7 with 15 steps, outperforming Claude (22.0 and 14.9 respectively)
arXiv Detail & Related papers (2025-01-21T17:48:10Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.