SecAgent: Efficient Mobile GUI Agent with Semantic Context
- URL: http://arxiv.org/abs/2603.08533v1
- Date: Mon, 09 Mar 2026 16:04:08 GMT
- Title: SecAgent: Efficient Mobile GUI Agent with Semantic Context
- Abstract summary: SecAgent is an efficient mobile GUI agent at 3B scale.<n>Semantic context mechanism distills history screenshots and actions into concise, natural language summaries.<n>We will open-source the training dataset, benchmark, model, and code to advance research in multilingual mobile GUI automation.
- Score: 14.843356281047543
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Mobile Graphical User Interface (GUI) agents powered by multimodal large language models have demonstrated promising capabilities in automating complex smartphone tasks. However, existing approaches face two critical limitations: the scarcity of high-quality multilingual datasets, particularly for non-English ecosystems, and inefficient history representation methods. To address these challenges, we present SecAgent, an efficient mobile GUI agent at 3B scale. We first construct a human-verified Chinese mobile GUI dataset with 18k grounding samples and 121k navigation steps across 44 applications, along with a Chinese navigation benchmark featuring multi-choice action annotations. Building upon this dataset, we propose a semantic context mechanism that distills history screenshots and actions into concise, natural language summaries, significantly reducing computational costs while preserving task-relevant information. Through supervised and reinforcement fine-tuning, SecAgent outperforms similar-scale baselines and achieves performance comparable to 7B-8B models on our and public navigation benchmarks. We will open-source the training dataset, benchmark, model, and code to advance research in multilingual mobile GUI automation.
Related papers
- HyMobileAgent: Data-Environment Co-Scaling for Efficient GUI Agents [57.53434680014667]
HyMobileAgent is a vision-native foundation model built on Hy3.0-VL-A3B.<n>We develop a joint data and environment centric scaling framework to address the key bottlenecks of mobile interaction.
arXiv Detail & Related papers (2026-07-16T04:12:42Z) - GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agents [19.27396264271709]
GUI-CEval is the first comprehensive benchmark for Chinese mobile GUI agents built entirely on physical device environments.<n>It spans 201 mainstream apps across four device types and adopts a two-level structure that evaluates both atomic abilities and realistic application-level performance along five dimensions: perception, planning, reflection, execution, and evaluation.
arXiv Detail & Related papers (2026-03-16T09:45:33Z) - AUTO-Explorer: Automated Data Collection for GUI Agent [58.58097564914626]
We propose an automated data collection method with minimal annotation costs, named Auto-Explorer.<n>It incorporates a simple yet effective exploration mechanism that autonomously parses and explores GUI environments.<n>Using the data gathered, we fine-tune a multimodal large language model (MLLM) and establish a GUI element grounding testing set.
arXiv Detail & Related papers (2025-11-09T15:13:45Z) - PAL-UI: Planning with Active Look-back for Vision-Based GUI Agents [151.86841216364294]
We propose textbfPAL-UI (textbfPlanning with textbfActive textbfLook-back), a novel framework that enables GUI agents to adaptively retrieve past observations when required.<n> PAL-UI combines a dual-level summarization agent, capturing both observation-level cues and action-level outcomes, with a dedicated retrieval tool.
arXiv Detail & Related papers (2025-10-01T01:48:39Z) - AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuning [82.42421823672954]
AgentCPM-GUI is built for robust and efficient on-device GUI interaction.<n>Our training pipeline includes grounding-aware pre-training to enhance perception.<n>AgentCPM-GUI achieves state-of-the-art performance on five public benchmarks.
arXiv Detail & Related papers (2025-06-02T07:30:29Z) - InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection [38.833925781308665]
We introduce textitInfiGUIAgent, an MLLM-based GUI Agent trained with a two-stage supervised fine-tuning pipeline.<n> Stage 1 enhances fundamental skills such as GUI understanding and grounding, while Stage 2 integrates hierarchical reasoning and expectation-reflection reasoning skills.<n>textitInfiGUIAgent achieves competitive performance on several GUI benchmarks.
arXiv Detail & Related papers (2025-01-08T15:45:21Z) - AutoDroid-V2: Boosting SLM-based GUI Agents via Code Generation [27.984521240600493]
We propose to convert the UI task automation problem to a code generation problem, which can be effectively solved by an on-device SLM and efficiently executed with an on-device code interpreter.<n>We adopt a document-centered approach that automatically builds fine-grained API documentation for each app and generates diverse task samples.<n>Based on detailed comparisons with state-of-the-art mobile UI agents, our approach effectively improves the mobile task automation with significantly higher success rates and lower latency/token consumption.
arXiv Detail & Related papers (2024-12-24T02:54:56Z) - ShowUI: One Vision-Language-Action Model for GUI Visual Agent [80.50062396585004]
Building Graphical User Interface (GUI) assistants holds significant promise for enhancing human workflow productivity.
We develop a vision-language-action model in digital world, namely ShowUI, which features the following innovations.
ShowUI, a lightweight 2B model using 256K data, achieves a strong 75.1% accuracy in zero-shot screenshot grounding.
arXiv Detail & Related papers (2024-11-26T14:29:47Z) - AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents [45.644265025493276]
We introduce the Android Multi-annotation EXpo (AMEX) to advance research on AI agents in mobile scenarios.<n>AMEX comprises over 104K high-resolution screenshots from popular mobile applications, which are annotated at multiple levels.<n>We develop this dataset from a more instructive and detailed perspective, complementing the general settings of existing datasets.
arXiv Detail & Related papers (2024-07-03T17:59:58Z) - XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented
Languages [105.54207724678767]
Data scarcity is a crucial issue for the development of highly multilingual NLP systems.
We propose XTREME-UP, a benchmark defined by its focus on the scarce-data scenario rather than zero-shot.
XTREME-UP evaluates the capabilities of language models across 88 under-represented languages over 9 key user-centric technologies.
arXiv Detail & Related papers (2023-05-19T18:00:03Z) - Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning [34.24671403624908]
Mobile User Interface Summarization generates succinct language descriptions of mobile screens for conveying important contents and functionalities of the screen.
We present Screen2Words, a novel screen summarization approach that automatically encapsulates essential information of a UI screen into a coherent language phrase.
arXiv Detail & Related papers (2021-08-07T03:01:23Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.