SE-GA: Memory-Augmented Self-Evolution for GUI Agents

• URL: http://arxiv.org/abs/2605.16883v1
• Date: Sat, 16 May 2026 08:51:57 GMT
• Title: SE-GA: Memory-Augmented Self-Evolution for GUI Agents
• Abstract summary: Self-Evolving GUI Agent (SE-GA) is a novel framework that integrates hierarchical memory structures with an iterative self-improvement mechanism.<n>To ensure continuous learning, we introduce Memory-Augmented Self-Evolution (MASE)
• Score: 17.051355070847166
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Autonomous Graphical User Interface (GUI) agents often struggle with multi-step tasks due to constrained context windows and static policies that fail to adapt to dynamic environments. To address these limitations, this work proposes the Self-Evolving GUI Agent (SE-GA), a novel framework that integrates hierarchical memory structures with an iterative self-improvement mechanism. At the core of our approach is Test-Time Memory Extension (TTME), which facilitates long-term planning by dynamically retrieving episodic, semantic, and experiential memories to provide salient contexts during inference. To ensure continuous learning, we introduce Memory-Augmented Self-Evolution (MASE), which is a training pipeline that adopts the data collected by TTME to stabilize and enhance the agent's foundational policy. Extensive evaluations across both offline and online benchmarks demonstrate SE-GA achieves state-of-the-art performance, reaching success rates of 89.0\% on ScreenSpot and 75.8\% on the challenging AndroidControl-High dataset. Furthermore, significant improvements on the AndroidWorld benchmark highlight the superior generalization to dynamic environments. Open source code: https://github.com/jinshilong-dev/SE-GA

Related papers

• AdaMEM: Test-Time Adaptive Memory for Language Agents [27.178795834085296]
  We propose the Adaptive Memory Agent (AdaMEM), a novel framework for agent test-time adaptation.<n>Without updating model parameters online, AdaMEM adapts agent behavior via a hybrid memory architecture.<n>AdaMEM significantly outperforms static memory baselines, achieving relative gains of up to 13% on ALFWorld and 11% on WebShop.
  arXiv  Detail & Related papers  (2026-06-04T04:06:08Z)
• STAMP: Training Explicit Memory for Mobile GUI Agents in Controllable and Scalable Virtual Environments [63.39393178045112]
  Mobile agents excel at immediate reactive control but frequently fail in realistic, longhorizon tasks that require memory.<n>We present STAMP, a framework that trains explicit memory in mobile agents through controllable virtual environments.<n>The resulting StampGUI agent sets a new high watermark on our Memory-World benchmark, demonstrating exceptional memory accuracy and task resilience.
  arXiv  Detail & Related papers  (2026-05-28T04:00:13Z)
• Mem-W: Latent Memory-Native GUI Agents [50.87647372904382]
  We introduce Mem-W, a series of latent-memory-native GUI agents that treat memory as part of the agent's continuous context.<n>Across four web and mobile navigation benchmarks, Mem-W consistently improves diverse backbones and memory-enhanced baselines.
  arXiv  Detail & Related papers  (2026-05-10T04:31:23Z)
• GEBench: Benchmarking Image Generation Models as GUI Environments [49.513441724802135]
  We introduce GEBench, a benchmark for evaluating dynamic interaction and temporal coherence in GUI generation.<n>GE-Score is a novel five-dimensional metric that assesses Goal Achievement, Interaction Logic, Content Consistency, UI Plausibility, and Visual Quality.<n>Our findings identify icon interpretation, text rendering, and localization precision as critical bottlenecks.
  arXiv  Detail & Related papers  (2026-02-09T18:52:02Z)
• UI-Mem: Self-Evolving Experience Memory for Online Reinforcement Learning in Mobile GUI Agents [50.053654092780825]
  Online Reinforcement Learning (RL) offers a promising paradigm for enhancing GUI agents through direct environment interaction.<n>We propose UI-Mem, a novel framework that enhances GUI online RL with a Hierarchical Experience Memory.<n>We show that UI-Mem significantly outperforms traditional RL baselines and static reuse strategies.
  arXiv  Detail & Related papers  (2026-02-05T16:21:43Z)
• Darwinian Memory: A Training-Free Self-Regulating Memory System for GUI Agent Evolution [18.68532215387754]
  Multimodal Large Language Model (MLLM) agents facilitate Graphical User Interface (GUI) automation but struggle with long-horizon, cross-application tasks.<n>Existing paradigms struggle to adapt to dynamic GUI environments, suffering from a mismatch between high-level intent and low-level execution.<n>We propose the Darwinian Memory System (DMS), a self-evolving architecture that constructs memory as a dynamic ecosystem governed by the law of survival of the fittest.
  arXiv  Detail & Related papers  (2026-01-30T04:01:21Z)
• MAGNET: Towards Adaptive GUI Agents with Memory-Driven Knowledge Evolution [44.92687597179025]
  Mobile agents powered by large foundation models enable autonomous task execution, but frequent updates cause agents trained on historical data to fail.<n>We introduce MAGNET, a memory-driven adaptive agent framework with dual-level memory.<n>Online benchmark AndroidWorld evaluations show substantial improvements over baselines, while offline benchmarks confirm consistent gains under distribution shifts.
  arXiv  Detail & Related papers  (2026-01-27T04:58:48Z)
• SCOPE: Prompt Evolution for Enhancing Agent Effectiveness [53.75986399936395]
  Large Language Model (LLM) agents are increasingly deployed in environments that generate massive, dynamic contexts.<n>While agents have access to this context, their static prompts lack the mechanisms to manage it effectively.<n>We introduce textbfSCOPE (Self-evolving Context Optimization via Prompt Evolution)<n>We propose a Dual-Stream mechanism that balances tactical specificity (resolving immediate errors) with strategic generality (evolving long-term principles)
  arXiv  Detail & Related papers  (2025-12-17T12:25:05Z)
• Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory [89.65731902036669]
  Evo-Memory is a streaming benchmark and framework for evaluating self-evolving memory in large language model (LLM) agents.<n>We evaluate over ten representative memory modules and evaluate them across 10 diverse multi-turn goal-oriented and single-turn reasoning and QA datasets.
  arXiv  Detail & Related papers  (2025-11-25T21:08:07Z)
• MGA: Memory-Driven GUI Agent for Observation-Centric Interaction [30.45490249299358]
  We introduce the Memory-Driven GUI Agent (MGA), which reframes GUI interaction around the principle of observe first, then decide.<n>MGA achieves substantial gains in robustness, generalization, and efficiency compared to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-10-28T08:19:58Z)
• Task-Adaptive Saliency Guidance for Exemplar-free Class Incremental Learning [60.501201259732625]
  We introduce task-adaptive saliency for EFCIL and propose a new framework, which we call Task-Adaptive Saliency Supervision (TASS) Our experiments demonstrate that our method can better preserve saliency maps across tasks and achieve state-of-the-art results on the CIFAR-100, Tiny-ImageNet, and ImageNet-Subset EFCIL benchmarks.
  arXiv  Detail & Related papers  (2022-12-16T02:43:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.