Scaling GUI Agents with Visual State Transitions

• URL: http://arxiv.org/abs/2607.24112v1
• Date: Mon, 27 Jul 2026 07:54:08 GMT
• Title: Scaling GUI Agents with Visual State Transitions
• Abstract summary: State Transition Pretraining (STP) is a new scaling axis for GUI agents.<n>STP continually pretrains a unified multimodal model on visual state transitions.<n>Our models consistently outperform baselines trained solely via direct fine-tuning across agent benchmarks.
• Score: 83.06866209578706
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce State Transition Pretraining (STP) as a new scaling axis for GUI agents. During the STP stage, we continually pretrain a unified multimodal model on visual state transitions by jointly optimizing inverse dynamics (predicting actions from state changes) and forward dynamics (predicting next states from current states and actions). This optimization equips the model with better action-grounded visual representations and an internal world model of GUI dynamics. When subsequently fine-tuned on trajectories with task instructions, our STP-trained models consistently outperform baselines trained solely via direct trajectory fine-tuning across agent benchmarks in both desktop and mobile GUI scenarios (AgentNetBench, AndroidControl, and GUIOdyssey). Further empirical studies show that joint dynamics optimization yields stable improvements over single-objective training, and downstream performance scales steadily with the volume of transition data.

Related papers

• LeapTS: Rethinking Time Series Forecasting as Adaptive Multi-Horizon Scheduling [74.94985663101906]
  We present LeapTS, a novel framework that reformulates time series forecasting as a dynamic scheduling process over the prediction horizon.<n>We show that LeapTS improves overall forecasting performance by at least 7.4% while achieving a 2.6$times$ to 5.3$times$ inference speedup over representative Transformer-based models.
  arXiv  Detail & Related papers  (2026-05-11T09:54:02Z)
• Building Autonomous GUI Navigation via Agentic-Q Estimation and Step-Wise Policy Optimization [38.863014369090074]
  We introduce a novel MLLM-centered framework for GUI agents, which consists of two components: agentic-Q estimation and step-wise policy optimization.<n>Our framework endows Ovis2.5-9B with powerful GUI interaction capabilities, achieving remarkable performances on GUI navigation and grounding benchmarks.
  arXiv  Detail & Related papers  (2026-02-14T07:44:47Z)
• UI-Oceanus: Scaling GUI Agents with Synthetic Environmental Dynamics [25.60139960043983]
  We propose a framework that shifts the learning focus from mimicking high-level trajectories to mastering interaction physics.<n> UI-Oceanus converts low-cost autonomous exploration into high-density generative supervision to construct a robust internal world model.
  arXiv  Detail & Related papers  (2026-02-11T17:02:38Z)
• ANCHOR: Branch-Point Data Generation for GUI Agents [52.22377425487]
  End-to-end GUI agents for real desktop environments require large amounts of high-quality interaction data.<n>We present a trajectory expansion framework Anchor that bootstraps scalable desktop supervision from a small set of verified seed demonstrations.<n>Experiments on standard desktop benchmarks, OSWorld and WindowsAgentArena, show that models fine-tuned on our expanded corpus achieve consistent improvements.
  arXiv  Detail & Related papers  (2026-02-06T19:55:26Z)
• LLMs as Scalable, General-Purpose Simulators For Evolving Digital Agent Training [55.72784274656801]
  We introduce a scalable paradigm that generates structured UI states and transitions to synthesize training trajectories at scale.<n>Our paradigm integrates a digital world simulator for diverse UI states, a guided rollout process for coherent exploration, and a trajectory wrapper.<n>Experiments on WebArena and AndroidWorld show that UI-Simulator rivals or surpasses open-source agents trained on real UIs with significantly better robustness.
  arXiv  Detail & Related papers  (2025-10-16T17:59:38Z)
• Progressive Scaling Visual Object Tracking [38.28834233600855]
  We propose a progressive scaling training strategy for visual object tracking, systematically analyzing the influence of training data volume, model size, and input resolution on tracking performance.<n>Our empirical study reveals that while scaling each factor leads to significant improvements in tracking accuracy, naive training suffers from suboptimal optimization and limited iterative refinement.<n>We introduce DT-Training, a progressive scaling framework that integrates small teacher transfer and dual-branch alignment to maximize model potential.
  arXiv  Detail & Related papers  (2025-05-26T13:45:27Z)
• Collaborative Multi-Agent Reinforcement Learning for Automated Feature Transformation with Graph-Driven Path Optimization [17.588657338437812]
  We propose TCTO, a collaborative multi-agent reinforcement learning framework that automates feature engineering through graph-driven path optimization.<n>The framework's core innovation lies in an evolving interaction graph that models features as nodes and transformations as edges.<n>We conduct comprehensive experiments and case studies, which show superior performance across a range of datasets.
  arXiv  Detail & Related papers  (2025-04-24T08:16:13Z)
• Guiding VLM Agents with Process Rewards at Inference Time for GUI Navigation [101.09478572153239]
  We propose an approach that guides VLM agents with process supervision by a reward model during GUI navigation and control at inference time.<n>This guidance allows the VLM agent to optimize actions at each inference step, thereby improving performance in both static and dynamic environments.
  arXiv  Detail & Related papers  (2025-04-22T17:52:42Z)
• An Efficient Occupancy World Model via Decoupled Dynamic Flow and Image-assisted Training [50.71892161377806]
  DFIT-OccWorld is an efficient 3D occupancy world model that leverages decoupled dynamic flow and image-assisted training strategy.<n>Our model forecasts future dynamic voxels by warping existing observations using voxel flow, whereas static voxels are easily obtained through pose transformation.
  arXiv  Detail & Related papers  (2024-12-18T12:10:33Z)
• Emergent Agentic Transformer from Chain of Hindsight Experience [96.56164427726203]
  We show that a simple transformer-based model performs competitively with both temporal-difference and imitation-learning-based approaches. This is the first time that a simple transformer-based model performs competitively with both temporal-difference and imitation-learning-based approaches.
  arXiv  Detail & Related papers  (2023-05-26T00:43:02Z)
• EUCLID: Towards Efficient Unsupervised Reinforcement Learning with Multi-choice Dynamics Model [46.99510778097286]
  Unsupervised reinforcement learning (URL) poses a promising paradigm to learn useful behaviors in a task-agnostic environment. We introduce a novel model-fused paradigm to jointly pre-train the dynamics model and unsupervised exploration policy in the pre-training phase. We show that EUCLID achieves state-of-the-art performance with high sample efficiency.
  arXiv  Detail & Related papers  (2022-10-02T12:11:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.