AndroidEnv: A Reinforcement Learning Platform for Android

• URL: http://arxiv.org/abs/2105.13231v1
• Date: Thu, 27 May 2021 15:20:14 GMT
• Title: AndroidEnv: A Reinforcement Learning Platform for Android
• Authors: Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad and Doina Precup
• Abstract summary: AndroidEnv is an open-source platform for Reinforcement Learning (RL) research built on top of the Android ecosystem. It allows RL agents to interact with a wide variety of apps and services commonly used by humans through a universal touchscreen interface. Since agents train on a realistic simulation of an Android device, they have the potential to be deployed on real devices.
• Score: 41.572096255032946
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce AndroidEnv, an open-source platform for Reinforcement Learning (RL) research built on top of the Android ecosystem. AndroidEnv allows RL agents to interact with a wide variety of apps and services commonly used by humans through a universal touchscreen interface. Since agents train on a realistic simulation of an Android device, they have the potential to be deployed on real devices. In this report, we give an overview of the environment, highlighting the significant features it provides for research, and we present an empirical evaluation of some popular reinforcement learning agents on a set of tasks built on this platform.

Related papers

• AndroidGen: Building an Android Language Agent under Data Scarcity [32.277219971739726]
  We develop a framework called AndroidGen to enhance the capabilities of LLM-based agents under data scarcity. We leverage AndroidGen to collect trajectories given human tasks and train open-source LLMs on these trajectories to develop an open-source mobile agent without manually labeled trajectories. We extensively evaluate AndroidGen with AndroidWorld, AitW, and various popular applications, demonstrating its improvements and revealing potential areas for future improvement.
  arXiv  Detail & Related papers  (2025-04-27T16:30:10Z)
• A3: Android Agent Arena for Mobile GUI Agents [46.73085454978007]
  Mobile GUI agents are designed to autonomously perform tasks on mobile devices. Android Agent Arena (A3) is a novel evaluation platform for assessing performance on real-world, in-the-wild tasks. A3 includes 21 widely used general third-party apps and 201 tasks representative of common user scenarios.
  arXiv  Detail & Related papers  (2025-01-02T09:03:56Z)
• AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents [32.571194718225996]
  We propose AndroidLab as a systematic Android agent framework. It includes an operation environment with different modalities, action space, and a reproducible benchmark. It supports both large language models (LLMs) and multimodal models (LMMs) in the same action space.
  arXiv  Detail & Related papers  (2024-10-31T15:25:20Z)
• OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimization [66.22117723598872]
  We introduce an open-source framework designed to facilitate the development of multimodal web agent. We first train the base model with imitation learning to gain the basic abilities. We then let the agent explore the open web and collect feedback on its trajectories.
  arXiv  Detail & Related papers  (2024-10-25T15:01:27Z)
• AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents [5.044046039265116]
  We present AndroidWorld, a fully functional Android environment that provides reward signals for 116 programmatic tasks across 20 real-world Android apps. Unlike existing interactive environments, which provide a static test set, AndroidWorld dynamically constructs tasks that are parameterized and expressed in natural language. Our best agent can complete 30.6% of AndroidWorld's tasks, leaving ample room for future work.
  arXiv  Detail & Related papers  (2024-05-23T13:48:54Z)
• Aptly: Making Mobile Apps from Natural Language [0.7852714805965528]
  Aptly is an extension of the MIT App Inventor platform enabling mobile app development via natural language. The paper concludes with insights from a study of a pilot implementation involving high school students, which examines Aptly's practicality and user experience.
  arXiv  Detail & Related papers  (2024-04-30T22:33:34Z)
• HomeRobot: Open-Vocabulary Mobile Manipulation [107.05702777141178]
  Open-Vocabulary Mobile Manipulation (OVMM) is the problem of picking any object in any unseen environment, and placing it in a commanded location. HomeRobot has two components: a simulation component, which uses a large and diverse curated object set in new, high-quality multi-room home environments; and a real-world component, providing a software stack for the low-cost Hello Robot Stretch.
  arXiv  Detail & Related papers  (2023-06-20T14:30:32Z)
• RT-1: Robotics Transformer for Real-World Control at Scale [98.09428483862165]
  We present a model class, dubbed Robotics Transformer, that exhibits promising scalable model properties. We verify our conclusions in a study of different model classes and their ability to generalize as a function of the data size, model size, and data diversity based on a large-scale data collection on real robots performing real-world tasks.
  arXiv  Detail & Related papers  (2022-12-13T18:55:15Z)
• AppBuddy: Learning to Accomplish Tasks in Mobile Apps via Reinforcement Learning [19.990946219992992]
  We introduce an RL-based framework for learning to accomplish tasks in mobile apps. RL agents are provided with states derived from the underlying representation of on-screen elements. We develop a platform which addresses several engineering challenges to enable an effective RL training environment.
  arXiv  Detail & Related papers  (2021-05-31T23:02:38Z)
• Robustness of on-device Models: Adversarial Attack to Deep Learning Models on Android Apps [14.821745719407037]
  Most deep learning models within Android apps can easily be obtained via mature reverse engineering. In this study, we propose a simple but effective approach to hacking deep learning models using adversarial attacks.
  arXiv  Detail & Related papers  (2021-01-12T10:49:30Z)
• OpenBot: Turning Smartphones into Robots [95.94432031144716]
  Current robots are either expensive or make significant compromises on sensory richness, computational power, and communication capabilities. We propose to leverage smartphones to equip robots with extensive sensor suites, powerful computational abilities, state-of-the-art communication channels, and access to a thriving software ecosystem. We design a small electric vehicle that costs $50 and serves as a robot body for standard Android smartphones.
  arXiv  Detail & Related papers  (2020-08-24T18:04:50Z)
• Federated and continual learning for classification tasks in a society of devices [59.45414406974091]
  Light Federated and Continual Consensus (LFedCon2) is a new federated and continual architecture that uses light, traditional learners. Our method allows powerless devices (such as smartphones or robots) to learn in real time, locally, continuously, autonomously and from users. In order to test our proposal, we have applied it in a heterogeneous community of smartphone users to solve the problem of walking recognition.
  arXiv  Detail & Related papers  (2020-06-12T12:37:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.