Scenic4RL: Programmatic Modeling and Generation of Reinforcement Learning Environments

• URL: http://arxiv.org/abs/2106.10365v2
• Date: Tue, 28 Mar 2023 20:25:15 GMT
• Title: Scenic4RL: Programmatic Modeling and Generation of Reinforcement Learning Environments
• Authors: Abdus Salam Azad, Edward Kim, Qiancheng Wu, Kimin Lee, Ion Stoica, Pieter Abbeel, and Sanjit A. Seshia
• Abstract summary: Generation of diverse realistic scenarios is challenging for real-time strategy (RTS) environments. Most of the existing simulators rely on randomly generating the environments. We introduce the benefits of adopting an existing formal scenario specification language, SCENIC, to assist researchers.
• Score: 89.04823188871906
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The capability of a reinforcement learning (RL) agent heavily depends on the diversity of the learning scenarios generated by the environment. Generation of diverse realistic scenarios is challenging for real-time strategy (RTS) environments. The RTS environments are characterized by intelligent entities/non-RL agents cooperating and competing with the RL agents with large state and action spaces over a long period of time, resulting in an infinite space of feasible, but not necessarily realistic, scenarios involving complex interaction among different RL and non-RL agents. Yet, most of the existing simulators rely on randomly generating the environments based on predefined settings/layouts and offer limited flexibility and control over the environment dynamics for researchers to generate diverse, realistic scenarios as per their demand. To address this issue, for the first time, we formally introduce the benefits of adopting an existing formal scenario specification language, SCENIC, to assist researchers to model and generate diverse scenarios in an RTS environment in a flexible, systematic, and programmatic manner. To showcase the benefits, we interfaced SCENIC to an existing RTS environment Google Research Football(GRF) simulator and introduced a benchmark consisting of 32 realistic scenarios, encoded in SCENIC, to train RL agents and testing their generalization capabilities. We also show how researchers/RL practitioners can incorporate their domain knowledge to expedite the training process by intuitively modeling stochastic programmatic policies with SCENIC.

Related papers

• Aquatic Navigation: A Challenging Benchmark for Deep Reinforcement Learning [53.3760591018817]
  We propose a new benchmarking environment for aquatic navigation using recent advances in the integration between game engines and Deep Reinforcement Learning. Specifically, we focus on PPO, one of the most widely accepted algorithms, and we propose advanced training techniques. Our empirical evaluation shows that a well-designed combination of these ingredients can achieve promising results.
  arXiv  Detail & Related papers  (2024-05-30T23:20:23Z)
• HAZARD Challenge: Embodied Decision Making in Dynamically Changing Environments [93.94020724735199]
  HAZARD consists of three unexpected disaster scenarios, including fire, flood, and wind. This benchmark enables us to evaluate autonomous agents' decision-making capabilities across various pipelines.
  arXiv  Detail & Related papers  (2024-01-23T18:59:43Z)
• RealGen: Retrieval Augmented Generation for Controllable Traffic Scenarios [58.62407014256686]
  RealGen is a novel retrieval-based in-context learning framework for traffic scenario generation. RealGen synthesizes new scenarios by combining behaviors from multiple retrieved examples in a gradient-free way. This in-context learning framework endows versatile generative capabilities, including the ability to edit scenarios.
  arXiv  Detail & Related papers  (2023-12-19T23:11:06Z)
• Learning Curricula in Open-Ended Worlds [17.138779075998084]
  This thesis develops a class of methods called Unsupervised Environment Design (UED) Given an environment design space, UED automatically generates an infinite sequence or curriculum of training environments. The findings in this thesis show that UED autocurricula can produce RL agents exhibiting significantly improved robustness.
  arXiv  Detail & Related papers  (2023-12-03T16:44:00Z)
• Adaptive action supervision in reinforcement learning from real-world multi-agent demonstrations [10.174009792409928]
  We propose a method for adaptive action supervision in RL from real-world demonstrations in multi-agent scenarios. In the experiments, using chase-and-escape and football tasks with the different dynamics between the unknown source and target environments, we show that our approach achieved a balance between the generalization and the generalization ability compared with the baselines.
  arXiv  Detail & Related papers  (2023-05-22T13:33:37Z)
• Sim2real for Reinforcement Learning Driven Next Generation Networks [4.29590751118341]
  Reinforcement Learning (RL) models are regarded as the key to solving RAN-related multi-objective optimization problems. One of the main reasons is the modelling gap between the simulation and the real environment, which could make the RL agent trained by simulation ill-equipped for the real environment. This article brings to the fore the sim2real challenge within the context of Open RAN (O-RAN) Several use cases are presented to exemplify and demonstrate failure modes of the simulations trained RL model in real environments.
  arXiv  Detail & Related papers  (2022-06-08T12:40:24Z)
• Multitask Adaptation by Retrospective Exploration with Learned World Models [77.34726150561087]
  We propose a meta-learned addressing model called RAMa that provides training samples for the MBRL agent taken from task-agnostic storage. The model is trained to maximize the expected agent's performance by selecting promising trajectories solving prior tasks from the storage.
  arXiv  Detail & Related papers  (2021-10-25T20:02:57Z)
• CyGIL: A Cyber Gym for Training Autonomous Agents over Emulated Network Systems [3.2550963598419957]
  CyGIL is an experimental testbed of an emulated RL training environment for network cyber operations. It uses a stateless environment architecture and incorporates the MITRE ATT&CK framework to establish a high fidelity training environment. Its comprehensive action space and flexible game design allow the agent training to focus on particular advanced persistent threat (APT) profiles.
  arXiv  Detail & Related papers  (2021-09-07T20:52:44Z)
• The AI Arena: A Framework for Distributed Multi-Agent Reinforcement Learning [0.3437656066916039]
  We introduce the AI Arena: a scalable framework with flexible abstractions for distributed multi-agent reinforcement learning. We show performance gains due to a distributed multi-agent learning approach over commonly-used RL techniques in several different learning environments.
  arXiv  Detail & Related papers  (2021-03-09T22:16:19Z)
• Deep Reinforcement Learning amidst Lifelong Non-Stationarity [67.24635298387624]
  We show that an off-policy RL algorithm can reason about and tackle lifelong non-stationarity. Our method leverages latent variable models to learn a representation of the environment from current and past experiences. We also introduce several simulation environments that exhibit lifelong non-stationarity, and empirically find that our approach substantially outperforms approaches that do not reason about environment shift.
  arXiv  Detail & Related papers  (2020-06-18T17:34:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.