Towards Scalable Lightweight GUI Agents via Multi-role Orchestration

• URL: http://arxiv.org/abs/2604.13488v1
• Date: Wed, 15 Apr 2026 05:23:04 GMT
• Title: Towards Scalable Lightweight GUI Agents via Multi-role Orchestration
• Abstract summary: We present the LAMO framework, which endows a lightweight MLLM with GUI-specific knowledge and task scalability.<n>We develop a task-scalable native GUI agent, LAMO-3B, supporting monolithic execution and MAS-style orchestration.
• Score: 26.13785805540596
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Autonomous Graphical User Interface (GUI) agents powered by Multimodal Large Language Models (MLLMs) enable digital automation on end-user devices. While scaling both parameters and data has yielded substantial gains, advanced methods still suffer from prohibitive deployment costs on resource-constrained devices. When facing complex in-the-wild scenarios, lightweight GUI agents are bottlenecked by limited capacity and poor task scalability under end-to-end episodic learning, impeding adaptation to multi-agent systems (MAS), while training multiple skill-specific experts remains costly. Can we strike an effective trade-off in this cost-scalability dilemma, enabling lightweight MLLMs to participate in realistic GUI workflows? To address these challenges, we propose the LAMO framework, which endows a lightweight MLLM with GUI-specific knowledge and task scalability, allowing multi-role orchestration to expand its capability boundary for GUI automation. LAMO combines role-oriented data synthesis with a two-stage training recipe: (i) supervised fine-tuning with Perplexity-Weighted Cross-Entropy optimization for knowledge distillation and visual perception enhancement, and (ii) reinforcement learning for role-oriented cooperative exploration. With LAMO, we develop a task-scalable native GUI agent, LAMO-3B, supporting monolithic execution and MAS-style orchestration. When paired with advanced planners as a plug-and-play policy executor, LAMO-3B can continuously benefit from planner advances, enabling a higher performance ceiling. Extensive static and online evaluations validate the effectiveness of our design.

Related papers

• LiteGUI: Distilling Compact GUI Agents with Reinforcement Learning [11.35341244051162]
  Current on-device vision-language GUI agents are constrained by limited model capacity.<n>TraditionalSupervised Fine-Tuning for small-scale models often leads to overfitting, catastrophic forgetting and policy rigidity.<n>We propose a novel SFT-free training paradigm that significantly enhances the performance of small-scale models.
  arXiv  Detail & Related papers  (2026-05-08T09:38:29Z)
• CGL: Advancing Continual GUI Learning via Reinforcement Fine-Tuning [67.78566256784404]
  Supervised Fine-Tuning (SFT) facilitates fast adaptation, it often triggers knowledge overwriting.<n>Reinforcement Learning (RL) demonstrates an inherent resilience that shields prior interaction logic from erasure.<n>We propose a textbfContinual textbfGUI textbfLearning framework that balances efficiency and skill retention.
  arXiv  Detail & Related papers  (2026-03-03T13:02:20Z)
• Grounded in Reality: Learning and Deploying Proactive LLM from Offline Logs [72.08224879435762]
  textttLearn-to-Ask is a simulator-free framework for learning and deploying proactive dialogue agents.<n>Our approach culminates in the successful deployment of LLMs into a live, large-scale online AI service.
  arXiv  Detail & Related papers  (2025-10-29T12:08:07Z)
• SWIRL: A Staged Workflow for Interleaved Reinforcement Learning in Mobile GUI Control [38.81034547191083]
  We introduce SWIRL, a staged workflow for interleaved reinforcement learning designed for multi-agent systems.<n> SWIRL reformulates MARL into a sequence of single-agent reinforcement learning tasks, updating one agent at a time while keeping the others fixed.<n>In application to mobile GUI control, SWIRL instantiates a Navigator that converts language and screen context into structured plans, and an Interactor that grounds these plans into executable atomic actions.
  arXiv  Detail & Related papers  (2025-08-27T16:27:19Z)
• Graph-Augmented Large Language Model Agents: Current Progress and Future Prospects [57.53024716739594]
  Graph-augmented LLM Agents (GLA) enhance structure, continuity, and coordination in complex agent systems.<n>This paper offers a timely and comprehensive overview of recent advances and highlights key directions for future work.<n>We hope this paper can serve as a roadmap for future research on GLA and foster a deeper understanding of the role of graphs in GLA agent systems.
  arXiv  Detail & Related papers  (2025-07-29T00:27:12Z)
• AgentFly: Extensible and Scalable Reinforcement Learning for LM Agents [25.735754822676277]
  Language model (LM) agents have gained significant attention for their ability to autonomously complete tasks.<n> reinforcement learning (RL) has been explored to enhance LM's capabilities, such as reasoning and factuality.<n>We built AgentFly, a scalable and Agent-RL framework designed to empower LM agents with a variety of RL algorithms.
  arXiv  Detail & Related papers  (2025-07-20T10:22:36Z)
• ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replay [88.74638385288773]
  Agentic Replay Policy Optimization improves performance on complex, long-horizon computer tasks.<n>We propose a task selection strategy that filters tasks based on baseline agent performance.<n>Experiments on the OSWorld benchmark demonstrate that ARPO achieves competitive results.
  arXiv  Detail & Related papers  (2025-05-22T06:24:32Z)
• API Agents vs. GUI Agents: Divergence and Convergence [37.13923771130588]
  API- and GUI-based large language models (LLMs) interact with graphical user interfaces in a human-like manner.<n>This paper systematically analyzes their divergence and potential convergence.<n>We indicate that continuing innovations in LLM-based automation are poised to blur the lines between API- and GUI-driven agents.
  arXiv  Detail & Related papers  (2025-03-14T04:26:21Z)
• Performant LLM Agentic Framework for Conversational AI [1.6114012813668932]
  We introduce the Performant Agentic Framework (PAF), a novel system that assists Large Language Models (LLMs) in selecting appropriate nodes and executing actions in order when traversing complex graphs.<n>PAF combines LLM-based reasoning with a mathematically grounded vector scoring mechanism, achieving both higher accuracy and reduced latency.<n>Experiments demonstrate that PAF significantly outperforms baseline methods, paving the way for scalable, real-time Conversational AI systems in complex business environments.
  arXiv  Detail & Related papers  (2025-03-09T02:58:34Z)
• Scaling Autonomous Agents via Automatic Reward Modeling And Planning [52.39395405893965]
  Large language models (LLMs) have demonstrated remarkable capabilities across a range of tasks.<n>However, they still struggle with problems requiring multi-step decision-making and environmental feedback.<n>We propose a framework that can automatically learn a reward model from the environment without human annotations.
  arXiv  Detail & Related papers  (2025-02-17T18:49:25Z)
• Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment [58.94611347128066]
  multimodal large language models (MLLMs) struggle with fine-grained or precise understanding of visuals.<n>Recent studies either develop tool-using or unify specific visual tasks into the autoregressive framework, often at the expense of overall multimodal performance.<n>We propose Task Preference Optimization (TPO), a novel method that utilizes differentiable task preferences derived from typical fine-grained visual tasks.
  arXiv  Detail & Related papers  (2024-12-26T18:56:05Z)
• AgentPS: Agentic Process Supervision for Content Moderation with Multimodal LLMs [9.35901507816989]
  We introduce AgentPS, a framework that integrates Agentic Process Supervision into large language models.<n>We show that AgentPS achieves substantial improvements over baseline MLLMs on public benchmarks and proprietary datasets.<n>These results establish AgentPS as a scalable and effective solution for complex multimodal classification in large-scale industrial applications.
  arXiv  Detail & Related papers  (2024-12-15T04:58:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.