ReachAgent: Enhancing Mobile Agent via Page Reaching and Operation

• URL: http://arxiv.org/abs/2502.02955v1
• Date: Wed, 05 Feb 2025 07:35:23 GMT
• Title: ReachAgent: Enhancing Mobile Agent via Page Reaching and Operation
• Authors: Qinzhuo Wu, Wei Liu, Jian Luan, Bin Wang,
• Abstract summary: Given a task, mobile AI agents can interact with mobile devices in multiple steps and form a GUI flow that solves the task. To address this issue, we constructed a training dataset called MobileReach, which breaks the task into page reaching and operation subtasks. We propose ReachAgent, a two-stage framework that focuses on improving its task-completion abilities.
• Score: 11.931584529573176
• License:
• Abstract: Recently, mobile AI agents have gained increasing attention. Given a task, mobile AI agents can interact with mobile devices in multiple steps and finally form a GUI flow that solves the task. However, existing agents tend to focus on most task-relevant elements at each step, leading to local optimal solutions and ignoring the overall GUI flow. To address this issue, we constructed a training dataset called MobileReach, which breaks the task into page reaching and operation subtasks. Furthermore, we propose ReachAgent, a two-stage framework that focuses on improving its task-completion abilities. It utilizes the page reaching and page operation subtasks, along with reward-based preference GUI flows, to further enhance the agent. Experimental results show that ReachAgent significantly improves the IoU Acc and Text Acc by 7.12% and 7.69% on the step-level and 4.72% and 4.63% on the task-level compared to the SOTA agent. Our data and code will be released upon acceptance.

Related papers

• PC-Agent: A Hierarchical Multi-Agent Collaboration Framework for Complex Task Automation on PC [98.82146219495792]
  In this paper, we propose a hierarchical agent framework named PC-Agent. From the perception perspective, we devise an Active Perception Module (APM) to overcome the inadequate abilities of current MLLMs in perceiving screenshot content. From the decision-making perspective, to handle complex user instructions and interdependent subtasks more effectively, we propose a hierarchical multi-agent collaboration architecture.
  arXiv  Detail & Related papers  (2025-02-20T05:41:55Z)
• CowPilot: A Framework for Autonomous and Human-Agent Collaborative Web Navigation [70.3224918173672]
  CowPilot is a framework supporting autonomous as well as human-agent collaborative web navigation. It reduces the number of steps humans need to perform by allowing agents to propose next steps, while users are able to pause, reject, or take alternative actions. CowPilot can serve as a useful tool for data collection and agent evaluation across websites.
  arXiv  Detail & Related papers  (2025-01-28T00:56:53Z)
• Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Tasks [85.48034185086169]
  Mobile-Agent-E is a hierarchical multi-agent framework capable of self-evolution through past experience. Mobile-Agent-E achieves a 22% absolute improvement over previous state-of-the-art approaches.
  arXiv  Detail & Related papers  (2025-01-20T20:35:46Z)
• Beyond Browsing: API-Based Web Agents [58.39129004543844]
  API-based agents outperform web browsing agents in experiments on WebArena. Hybrid Agents out-perform both others nearly uniformly across tasks. Results strongly suggest that when APIs are available, they present an attractive alternative to relying on web browsing alone.
  arXiv  Detail & Related papers  (2024-10-21T19:46:06Z)
• ClickAgent: Enhancing UI Location Capabilities of Autonomous Agents [0.0]
  ClickAgent is a novel framework for building autonomous agents. In ClickAgent, the MLLM handles reasoning and action planning, while a separate UI location model identifies the relevant UI elements on the screen. Our evaluation was conducted on both an Android smartphone emulator and an actual Android smartphone, using the task success rate as the key metric for measuring agent performance.
  arXiv  Detail & Related papers  (2024-10-09T14:49:02Z)
• MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents [7.4568642040547894]
  Large language model (LLM)-based mobile agents are increasingly popular due to their capability to interact directly with mobile phone Graphic User Interfaces (GUIs) Despite their promising prospects in both academic and industrial sectors, little research has focused on benchmarking the performance of existing mobile agents. We propose an efficient and user-friendly benchmark, MobileAgentBench, designed to alleviate the burden of extensive manual testing.
  arXiv  Detail & Related papers  (2024-06-12T13:14:50Z)
• Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration [52.25473993987409]
  We propose Mobile-Agent-v2, a multi-agent architecture for mobile device operation assistance. The architecture comprises three agents: planning agent, decision agent, and reflection agent. We show that Mobile-Agent-v2 achieves over a 30% improvement in task completion compared to the single-agent architecture.
  arXiv  Detail & Related papers  (2024-06-03T05:50:00Z)
• Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception [52.5831204440714]
  We introduce Mobile-Agent, an autonomous multi-modal mobile device agent. Mobile-Agent first leverages visual perception tools to accurately identify and locate both the visual and textual elements within the app's front-end interface. It then autonomously plans and decomposes the complex operation task, and navigates the mobile Apps through operations step by step.
  arXiv  Detail & Related papers  (2024-01-29T13:46:37Z)
• MobileAgent: enhancing mobile control via human-machine interaction and SOP integration [0.0]
  Large Language Models (LLMs) are now capable of automating mobile device operations for users. Privacy concerns related to personalized user data arise during mobile operations, requiring user confirmation. We have designed interactive tasks between agents and humans to identify sensitive information and align with personalized user needs. Our approach is evaluated on the new device control benchmark AitW, which encompasses 30K unique instructions across multi-step tasks.
  arXiv  Detail & Related papers  (2024-01-04T03:44:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.