GPA: Learning GUI Process Automation from Demonstrations

• URL: http://arxiv.org/abs/2604.01676v2
• Date: Sat, 04 Apr 2026 01:55:00 GMT
• Title: GPA: Learning GUI Process Automation from Demonstrations
• Abstract summary: GPA is a lightweight but general vision-based Robotic Process Automation (RPA)<n>It enables fast and stable process replay with only a single demo.<n>It addresses the fragility of traditional RPA and the non-deterministic risks of current vision language model-based GUI agents.
• Score: 59.82483926336919
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: GUI Process Automation (GPA) is a lightweight but general vision-based Robotic Process Automation (RPA), which enables fast and stable process replay with only a single demo. Addressing the fragility of traditional RPA and the non-deterministic risks of current vision language model-based GUI agents, GPA introduces three core benefits: (1) Robustness via Sequential Monte Carlo-based localization to handle rescaling and detection uncertainty; (2) Deterministic and Reliability safeguarded by readiness calibration; and (3) Privacy through fast, fully local execution. This approach delivers the adaptability, robustness, and security required for enterprise workflows. It can also be used as an MCP/CLI tool by other agents with coding capabilities so that the agent only reasons and orchestrates while GPA handles the GUI execution. We conducted a pilot experiment to compare GPA with Gemini 3 Pro (with CUA tools) and found that GPA achieves higher success rate with 10 times faster execution speed in finishing long-horizon GUI tasks.

Related papers

• UI-Venus-2 Technical Report [51.54957558722245]
  We present UI-Venus-2, a general-purpose foundation GUI agent designed to operate across mobile, web, and desktop environments.<n>To bridge the gap toward practical deployment, we jointly scale three critical dimensions: (1) Environments, expanding coverage to more than 170 multilingual mobile apps and native desktop operating systems; (2) Tasks, employing a deep-research pipeline for function-grounded instruction generation; and (3) Verification, adopting trace-level and sample-level evaluators with visual keypoints and multi-model voting to ensure reliable RL signals for training.
  arXiv  Detail & Related papers  (2026-08-27T17:15:33Z)
• AutoRPA: Efficient GUI Automation through LLM-Driven Code Synthesis from Interactions [27.06451670215353]
  AutoRPA is a framework that distills the decision logic of ReAct-style agents into robust RPA functions.<n> Experiments show that RPA functions generated by AutoRPA successfully solve similar tasks while reducing token usage by 82% to 96%.
  arXiv  Detail & Related papers  (2026-05-20T12:17:43Z)
• EE-MCP: Self-Evolving MCP-GUI Agents via Automated Environment Generation and Experience Learning [24.190399395358014]
  We formulate MCP-GUI as a unified hybrid policy learning problem where the agent learns when each modality provides complementary advantages.<n>We propose a self-evolving framework with a fully automatic pipeline that orchestrates automatic environment generation and validation, trajectory collection, gap-driven task synthesis, and quality-filtered training.
  arXiv  Detail & Related papers  (2026-04-10T18:46:05Z)
• Retrieval-augmented GUI Agents with Generative Guidelines [45.75975553907856]
  We propose RAG-GUI, a lightweight VLM that leverages web tutorials at inference time.<n>RAG-GUI is first warm-started via supervised finetuning (SFT) and further refined through self-guided rejection sampling finetuning (RSF)<n>It consistently outperforms baseline agents and surpasses other inference baselines by 2.6% to 13.3% across two model sizes.
  arXiv  Detail & Related papers  (2025-09-29T02:04:20Z)
• CoAct-1: Computer-using Agents with Coding as Actions [94.99657662893338]
  CoAct-1 is a novel multi-agent system that combines GUI-based control with direct programmatic execution.<n>We evaluate our system on the challenging OSWorld benchmark, where CoAct-1 achieves a new state-of-the-art success rate of 60.76%.
  arXiv  Detail & Related papers  (2025-08-05T21:33:36Z)
• MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environment [63.62778707277929]
  MobileGUI-RL is a scalable framework that trains GUI agent in online environment.<n>It synthesizes a curriculum of learnable tasks through self-exploration and filtering.<n>It adapts GRPO to GUI navigation with trajectory-aware advantages and composite rewards.
  arXiv  Detail & Related papers  (2025-07-08T07:07:53Z)
• Learning, Reasoning, Refinement: A Framework for Kahneman's Dual-System Intelligence in GUI Agents [15.303188467166752]
  We present CogniGUI, a cognitive framework developed to overcome limitations by enabling adaptive learning for GUI automation resembling human-like behavior.<n>To assess the generalization and adaptability of agent systems, we introduce ScreenSeek, a comprehensive benchmark that includes multi application navigation, dynamic state transitions, and cross interface coherence.<n> Experimental results demonstrate that CogniGUI surpasses state-of-the-art methods in both the current GUI grounding benchmarks and our newly proposed benchmark.
  arXiv  Detail & Related papers  (2025-06-22T06:30:52Z)
• ZeroGUI: Automating Online GUI Learning at Zero Human Cost [75.21128388931945]
  We propose ZeroGUI, a scalable, online learning framework for automating GUI Agent training at Zero human cost.<n>Specifically, ZeroGUI integrates (i) VLM-based automatic task generation to produce diverse training goals from the current environment state, (ii) VLM-based automatic reward estimation to assess task success without hand-crafted evaluation functions, and (iii) two-stage online reinforcement learning to continuously interact with and learn from GUI environments.
  arXiv  Detail & Related papers  (2025-05-29T17:59:51Z)
• Guiding VLM Agents with Process Rewards at Inference Time for GUI Navigation [101.09478572153239]
  We propose an approach that guides VLM agents with process supervision by a reward model during GUI navigation and control at inference time.<n>This guidance allows the VLM agent to optimize actions at each inference step, thereby improving performance in both static and dynamic environments.
  arXiv  Detail & Related papers  (2025-04-22T17:52:42Z)
• VeriSafe Agent: Safeguarding Mobile GUI Agent via Logic-based Action Verification [11.121359688304246]
  We introduce VeriSafe Agent (VSA), a formal verification system that serves as a logically grounded safeguard for Mobile GUI Agents.<n>At its core, VSA introduces a novel autoformalization technique that translates natural language user instructions into a formally verifiable specification.<n>We evaluate VSA's performance on 300 user instructions across 18 widely used mobile apps.
  arXiv  Detail & Related papers  (2025-03-24T09:46:05Z)
• You Only Look at Screens: Multimodal Chain-of-Action Agents [37.118034745972956]
  Auto-GUI is a multimodal solution that directly interacts with the interface. We propose a chain-of-action technique to help the agent decide what action to execute. We evaluate our approach on a new device-control benchmark AITW with 30$K$ unique instructions.
  arXiv  Detail & Related papers  (2023-09-20T16:12:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.