Empowering GUI Agents via Autonomous Experience Exploration and Hindsight Experience Utilization for Task Planning

• URL: http://arxiv.org/abs/2606.27330v1
• Date: Thu, 25 Jun 2026 17:44:48 GMT
• Title: Empowering GUI Agents via Autonomous Experience Exploration and Hindsight Experience Utilization for Task Planning
• Abstract summary: Small open source MLLMs suffer from weak planning and limited cross website generalization.<n>We introduce the planning experience exploration and utilization (PEEU) method, which autonomously explores environments to discover experiences.<n>Our experiments on real world benchmarks demonstrate PEEU's superior effectiveness, outperforming the much larger Qwen2.5-VL-32B model.
• Score: 39.028462985881305
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal web agents can assist humans in operating repetitive GUI tasks, where effective task planning is essential for decomposing complex tasks into executable actions. While small open source MLLMs are cost efficient and privacy preserving compared with commercial large models, they suffer from weak planning and limited cross website generalization. To address these limitations, we introduce the planning experience exploration and utilization (PEEU) method, which autonomously explores environments to discover experiences and utilizes hindsight experience to synthesize strictly aligned, high level training data. To quantitatively analyze the generalization behaviors driving this performance, we propose the task decomposition hierarchical analysis framework (TDHAF) to systematically study compositional generalization across three task granularities: low, middle and high levels. Our analysis reveals that mastering low level atomic skills does not guarantee high level planning competence, while high level task training yields stronger OOD generalization. Experiments on real world benchmarks demonstrate PEEU's superior effectiveness: our 7B model achieves 30.6% accuracy, outperforming the much larger Qwen2.5-VL-32B model. These demonstrate constructing hindsight high level tasks and leveraging experiences is crucial for OOD planning abilities of small MLLMs.

Related papers

• How Foundational Skills Influence VLM-based Embodied Agents:A Native Perspective [18.773467537970753]
  We present NativeEmbodied, a benchmark for VLM-driven embodied agents.<n>Built on diverse simulated scenes, NativeEmbodied includes three representative high-level tasks in complex scenarios to evaluate overall performance.<n>For more detailed analysis, we construct four types of low-level tasks, each targeting a fundamental embodied skill.
  arXiv  Detail & Related papers  (2026-02-24T08:42:41Z)
• Mobile-Agent-RAG: Driving Smart Multi-Agent Coordination with Contextual Knowledge Empowerment for Long-Horizon Mobile Automation [57.12284831164602]
  Mobile agents show immense potential, yet current state-of-the-art (SoTA) agents exhibit inadequate success rates on real-world, long-horizon, cross-application tasks.<n>We propose Mobile-Agent-RAG, a novel hierarchical multi-agent framework that innovatively integrates dual-level retrieval augmentation.
  arXiv  Detail & Related papers  (2025-11-15T15:22:42Z)
• A Goal Without a Plan Is Just a Wish: Efficient and Effective Global Planner Training for Long-Horizon Agent Tasks [66.86312354478478]
  Agents based on large language models (LLMs) struggle with brainless trial-and-error and generating hallucinatory actions due to a lack of global planning in long-horizon tasks.<n>We introduce a plan-and-execute framework and propose a planner training method to enhance the executor agent's planning abilities without human effort.<n>Experiments show that executor agents equipped with our planner outperform existing methods, achieving new state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-10-07T06:10:53Z)
• Training Task Reasoning LLM Agents for Multi-turn Task Planning via Single-turn Reinforcement Learning [15.393743659727926]
  Large Language Models (LLMs) have demonstrated remarkable capabilities in knowledge acquisition, reasoning, and tool use.<n>This paper introduces a novel approach that transforms multi-turn task planning into single-turn task reasoning problems.
  arXiv  Detail & Related papers  (2025-09-24T23:47:36Z)
• Anticipate & Act : Integrating LLMs and Classical Planning for Efficient Task Execution in Household Environments [16.482992646001996]
  We develop a framework for anticipating household tasks and computing an action sequence that jointly achieves these tasks.<n>We demonstrate a 31% reduction in execution time compared with a system that does not consider upcoming tasks.
  arXiv  Detail & Related papers  (2025-02-04T07:31:55Z)
• VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks [100.3234156027118]
  We present VLABench, an open-source benchmark for evaluating universal LCM task learning.<n>VLABench provides 100 carefully designed categories of tasks, with strong randomization in each category of task and a total of 2000+ objects.<n>The benchmark assesses multiple competencies including understanding of mesh&texture, spatial relationship, semantic instruction, physical laws, knowledge transfer and reasoning.
  arXiv  Detail & Related papers  (2024-12-24T06:03:42Z)
• Spatial Reasoning and Planning for Deep Embodied Agents [2.7195102129095003]
  This thesis explores the development of data-driven techniques for spatial reasoning and planning tasks. It focuses on enhancing learning efficiency, interpretability, and transferability across novel scenarios.
  arXiv  Detail & Related papers  (2024-09-28T23:05:56Z)
• Agent Planning with World Knowledge Model [88.4897773735576]
  We introduce parametric World Knowledge Model (WKM) to facilitate agent planning.<n>We develop WKM, providing prior task knowledge to guide the global planning and dynamic state knowledge to assist the local planning.<n>Our method can achieve superior performance compared to various strong baselines.
  arXiv  Detail & Related papers  (2024-05-23T06:03:19Z)
• Efficient Learning of High Level Plans from Play [57.29562823883257]
  We present Efficient Learning of High-Level Plans from Play (ELF-P), a framework for robotic learning that bridges motion planning and deep RL. We demonstrate that ELF-P has significantly better sample efficiency than relevant baselines over multiple realistic manipulation tasks.
  arXiv  Detail & Related papers  (2023-03-16T20:09:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.