Search-Based Testing of Reinforcement Learning
- URL: http://arxiv.org/abs/2205.04887v1
- Date: Sat, 7 May 2022 12:40:45 GMT
- Title: Search-Based Testing of Reinforcement Learning
- Authors: Martin Tappler, Filip Cano C\'ordoba, Bernhard K. Aichernig and
Bettina K\"onighofer
- Abstract summary: We present a search-based testing framework for evaluating the safety and performance of deep RL agents.
For safety testing, our framework utilizes a search algorithm that searches for a reference trace that solves the RL task.
For robust performance testing, we create a diverse set of traces via fuzz testing.
We apply our search-based testing approach on RL for Nintendo's Super Mario Bros.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Evaluation of deep reinforcement learning (RL) is inherently challenging.
Especially the opaqueness of learned policies and the stochastic nature of both
agents and environments make testing the behavior of deep RL agents difficult.
We present a search-based testing framework that enables a wide range of novel
analysis capabilities for evaluating the safety and performance of deep RL
agents. For safety testing, our framework utilizes a search algorithm that
searches for a reference trace that solves the RL task. The backtracking states
of the search, called boundary states, pose safety-critical situations. We
create safety test-suites that evaluate how well the RL agent escapes
safety-critical situations near these boundary states. For robust performance
testing, we create a diverse set of traces via fuzz testing. These fuzz traces
are used to bring the agent into a wide variety of potentially unknown states
from which the average performance of the agent is compared to the average
performance of the fuzz traces. We apply our search-based testing approach on
RL for Nintendo's Super Mario Bros.
Related papers
- Safe Deep Reinforcement Learning by Verifying Task-Level Properties [84.64203221849648]
Cost functions are commonly employed in Safe Deep Reinforcement Learning (DRL)
The cost is typically encoded as an indicator function due to the difficulty of quantifying the risk of policy decisions in the state space.
In this paper, we investigate an alternative approach that uses domain knowledge to quantify the risk in the proximity of such states by defining a violation metric.
arXiv Detail & Related papers (2023-02-20T15:24:06Z) - Provable Safe Reinforcement Learning with Binary Feedback [62.257383728544006]
We consider the problem of provable safe RL when given access to an offline oracle providing binary feedback on the safety of state, action pairs.
We provide a novel meta algorithm, SABRE, which can be applied to any MDP setting given access to a blackbox PAC RL algorithm for that setting.
arXiv Detail & Related papers (2022-10-26T05:37:51Z) - Mastering the Unsupervised Reinforcement Learning Benchmark from Pixels [112.63440666617494]
Reinforcement learning algorithms can succeed but require large amounts of interactions between the agent and the environment.
We propose a new method to solve it, using unsupervised model-based RL, for pre-training the agent.
We show robust performance on the Real-Word RL benchmark, hinting at resiliency to environment perturbations during adaptation.
arXiv Detail & Related papers (2022-09-24T14:22:29Z) - Ablation Study of How Run Time Assurance Impacts the Training and
Performance of Reinforcement Learning Agents [5.801944210870593]
We conduct an ablation study using evaluation best practices to investigate the impact of run time assurance (RTA) on effective learning.
Our conclusions shed light on the most promising directions of Safe Reinforcement Learning.
arXiv Detail & Related papers (2022-07-08T20:15:15Z) - A Search-Based Testing Approach for Deep Reinforcement Learning Agents [1.1580916951856255]
We propose a Search-based Testing Approach of Reinforcement Learning Agents (STARLA) to test the policy of a DRL agent.
We use machine learning models and a dedicated genetic algorithm to narrow the search towards faulty episodes.
arXiv Detail & Related papers (2022-06-15T20:51:33Z) - GalilAI: Out-of-Task Distribution Detection using Causal Active
Experimentation for Safe Transfer RL [11.058960131490903]
Out-of-distribution (OOD) detection is a well-studied topic in supervised learning.
We propose a novel task: that of Out-of-Task Distribution (OOTD) detection.
We name our method GalilAI, in honor of Galileo Galilei, as it discovers, among other causal processes, that gravitational acceleration is independent of the mass of a body.
arXiv Detail & Related papers (2021-10-29T01:45:56Z) - Robust Reinforcement Learning on State Observations with Learned Optimal
Adversary [86.0846119254031]
We study the robustness of reinforcement learning with adversarially perturbed state observations.
With a fixed agent policy, we demonstrate that an optimal adversary to perturb state observations can be found.
For DRL settings, this leads to a novel empirical adversarial attack to RL agents via a learned adversary that is much stronger than previous ones.
arXiv Detail & Related papers (2021-01-21T05:38:52Z) - Robust Deep Reinforcement Learning through Adversarial Loss [74.20501663956604]
Recent studies have shown that deep reinforcement learning agents are vulnerable to small adversarial perturbations on the agent's inputs.
We propose RADIAL-RL, a principled framework to train reinforcement learning agents with improved robustness against adversarial attacks.
arXiv Detail & Related papers (2020-08-05T07:49:42Z) - Robust Deep Reinforcement Learning against Adversarial Perturbations on
State Observations [88.94162416324505]
A deep reinforcement learning (DRL) agent observes its states through observations, which may contain natural measurement errors or adversarial noises.
Since the observations deviate from the true states, they can mislead the agent into making suboptimal actions.
We show that naively applying existing techniques on improving robustness for classification tasks, like adversarial training, is ineffective for many RL tasks.
arXiv Detail & Related papers (2020-03-19T17:59:59Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.