Novelty Detection in Reinforcement Learning with World Models
- URL: http://arxiv.org/abs/2310.08731v2
- Date: Fri, 22 Mar 2024 16:30:48 GMT
- Title: Novelty Detection in Reinforcement Learning with World Models
- Authors: Geigh Zollicoffer, Kenneth Eaton, Jonathan Balloch, Julia Kim, Mark O. Riedl, Robert Wright,
- Abstract summary: Reinforcement learning (RL) using world models has found significant recent successes.
However, when a sudden change to world mechanics or properties occurs then agent performance and reliability can dramatically decline.
Implementing novelty detection within generated world model frameworks is a crucial task for protecting the agent when deployed.
- Score: 15.01731216883798
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Reinforcement learning (RL) using world models has found significant recent successes. However, when a sudden change to world mechanics or properties occurs then agent performance and reliability can dramatically decline. We refer to the sudden change in visual properties or state transitions as novelties. Implementing novelty detection within generated world model frameworks is a crucial task for protecting the agent when deployed. In this paper, we propose straightforward bounding approaches to incorporate novelty detection into world model RL agents, by utilizing the misalignment of the world model's hallucinated states and the true observed states as an anomaly score. We provide effective approaches to detecting novelties in a distribution of transitions learned by an agent in a world model. Finally, we show the advantage of our work in a novel environment compared to traditional machine learning novelty detection methods as well as currently accepted RL focused novelty detection algorithms.
Related papers
- Federated Continual Novel Class Learning [68.05835753892907]
We propose a Global Alignment Learning framework that can accurately estimate the global novel class number.
Gal achieves significant improvements in novel-class performance, increasing the accuracy by 5.1% to 10.6%.
Gal is shown to be effective in equipping a variety of different mainstream Federated Learning algorithms with novel class discovery and learning capability.
arXiv Detail & Related papers (2023-12-21T00:31:54Z) - ReCoRe: Regularized Contrastive Representation Learning of World Model [21.29132219042405]
We present a world model that learns invariant features using contrastive unsupervised learning and an intervention-invariant regularizer.
Our method outperforms current state-of-the-art model-based and model-free RL methods and significantly improves on out-of-distribution point navigation tasks evaluated on the iGibson benchmark.
arXiv Detail & Related papers (2023-12-14T15:53:07Z) - Detecting and Learning Out-of-Distribution Data in the Open world:
Algorithm and Theory [15.875140867859209]
This thesis makes contributions to the realm of machine learning, specifically in the context of open-world scenarios.
Research investigates two intertwined steps essential for open-world machine learning: Out-of-distribution (OOD) Detection and Open-world Representation Learning (ORL)
arXiv Detail & Related papers (2023-10-10T00:25:21Z) - Activate and Reject: Towards Safe Domain Generalization under Category
Shift [71.95548187205736]
We study a practical problem of Domain Generalization under Category Shift (DGCS)
It aims to simultaneously detect unknown-class samples and classify known-class samples in the target domains.
Compared to prior DG works, we face two new challenges: 1) how to learn the concept of unknown'' during training with only source known-class samples, and 2) how to adapt the source-trained model to unseen environments.
arXiv Detail & Related papers (2023-10-07T07:53:12Z) - Learning to Operate in Open Worlds by Adapting Planning Models [12.513121330508477]
Planning agents are ill-equipped to act in novel situations in which their domain model no longer accurately represents the world.
We introduce an approach for such agents operating in open worlds that detects the presence of novelties and effectively adapts their domain models.
arXiv Detail & Related papers (2023-03-24T21:04:16Z) - Predictive Experience Replay for Continual Visual Control and
Forecasting [62.06183102362871]
We present a new continual learning approach for visual dynamics modeling and explore its efficacy in visual control and forecasting.
We first propose the mixture world model that learns task-specific dynamics priors with a mixture of Gaussians, and then introduce a new training strategy to overcome catastrophic forgetting.
Our model remarkably outperforms the naive combinations of existing continual learning and visual RL algorithms on DeepMind Control and Meta-World benchmarks with continual visual control tasks.
arXiv Detail & Related papers (2023-03-12T05:08:03Z) - Investigating the role of model-based learning in exploration and
transfer [11.652741003589027]
In this paper, we investigate transfer learning in the context of model-based agents.
We find that a model-based approach outperforms controlled model-free baselines for transfer learning.
Our results show that intrinsic exploration combined with environment models present a viable direction towards agents that are self-supervised and able to generalize to novel reward functions.
arXiv Detail & Related papers (2023-02-08T11:49:58Z) - Neuro-Symbolic World Models for Adapting to Open World Novelty [9.707805250772129]
We introduce WorldCloner, an end-to-end trainable neuro-symbolic world model for rapid novelty adaptation.
WorldCloner learns an efficient symbolic representation of the pre-novelty environment transitions.
WorldCloner augments the policy learning process using imagination-based adaptation.
arXiv Detail & Related papers (2023-01-16T07:49:12Z) - Mastering the Unsupervised Reinforcement Learning Benchmark from Pixels [112.63440666617494]
Reinforcement learning algorithms can succeed but require large amounts of interactions between the agent and the environment.
We propose a new method to solve it, using unsupervised model-based RL, for pre-training the agent.
We show robust performance on the Real-Word RL benchmark, hinting at resiliency to environment perturbations during adaptation.
arXiv Detail & Related papers (2022-09-24T14:22:29Z) - Federated and Generalized Person Re-identification through Domain and
Feature Hallucinating [88.77196261300699]
We study the problem of federated domain generalization (FedDG) for person re-identification (re-ID)
We propose a novel method, called "Domain and Feature Hallucinating (DFH)", to produce diverse features for learning generalized local and global models.
Our method achieves the state-of-the-art performance for FedDG on four large-scale re-ID benchmarks.
arXiv Detail & Related papers (2022-03-05T09:15:13Z) - Online reinforcement learning with sparse rewards through an active
inference capsule [62.997667081978825]
This paper introduces an active inference agent which minimizes the novel free energy of the expected future.
Our model is capable of solving sparse-reward problems with a very high sample efficiency.
We also introduce a novel method for approximating the prior model from the reward function, which simplifies the expression of complex objectives.
arXiv Detail & Related papers (2021-06-04T10:03:36Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.