Novelty Detection in Reinforcement Learning with World Models

• URL: http://arxiv.org/abs/2310.08731v2
• Date: Fri, 22 Mar 2024 16:30:48 GMT
• Title: Novelty Detection in Reinforcement Learning with World Models
• Authors: Geigh Zollicoffer, Kenneth Eaton, Jonathan Balloch, Julia Kim, Mark O. Riedl, Robert Wright,
• Abstract summary: Reinforcement learning (RL) using world models has found significant recent successes. However, when a sudden change to world mechanics or properties occurs then agent performance and reliability can dramatically decline. Implementing novelty detection within generated world model frameworks is a crucial task for protecting the agent when deployed.
• Score: 15.01731216883798
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning (RL) using world models has found significant recent successes. However, when a sudden change to world mechanics or properties occurs then agent performance and reliability can dramatically decline. We refer to the sudden change in visual properties or state transitions as novelties. Implementing novelty detection within generated world model frameworks is a crucial task for protecting the agent when deployed. In this paper, we propose straightforward bounding approaches to incorporate novelty detection into world model RL agents, by utilizing the misalignment of the world model's hallucinated states and the true observed states as an anomaly score. We provide effective approaches to detecting novelties in a distribution of transitions learned by an agent in a world model. Finally, we show the advantage of our work in a novel environment compared to traditional machine learning novelty detection methods as well as currently accepted RL focused novelty detection algorithms.

Related papers

• Generalize or Detect? Towards Robust Semantic Segmentation Under Multiple Distribution Shifts [56.57141696245328]
  In open-world scenarios, where both novel classes and domains may exist, an ideal segmentation model should detect anomaly classes for safety. Existing methods often struggle to distinguish between domain-level and semantic-level distribution shifts.
  arXiv  Detail & Related papers  (2024-11-06T11:03:02Z)
• Learning World Models for Unconstrained Goal Navigation [4.549550797148707]
  We introduce a goal-directed exploration algorithm, MUN, for learning world models. MUN is capable of modeling state transitions between arbitrary subgoal states in the replay buffer. Results demonstrate that MUN strengthens the reliability of world models and significantly improves the policy's capacity to generalize.
  arXiv  Detail & Related papers  (2024-11-03T01:35:06Z)
• Open-World Reinforcement Learning over Long Short-Term Imagination [91.28998327423295]
  We present LS-Imagine, which extends the imagination horizon within a limited number of state transition steps. Our method demonstrates significant improvements over state-of-the-art techniques in MineDojo.
  arXiv  Detail & Related papers  (2024-10-04T17:17:30Z)
• Open-Set Deepfake Detection: A Parameter-Efficient Adaptation Method with Forgery Style Mixture [58.60915132222421]
  We introduce an approach that is both general and parameter-efficient for face forgery detection. We design a forgery-style mixture formulation that augments the diversity of forgery source domains. We show that the designed model achieves state-of-the-art generalizability with significantly reduced trainable parameters.
  arXiv  Detail & Related papers  (2024-08-23T01:53:36Z)
• Federated Continual Novel Class Learning [68.05835753892907]
  We propose a Global Alignment Learning framework that can accurately estimate the global novel class number. Gal achieves significant improvements in novel-class performance, increasing the accuracy by 5.1% to 10.6%. Gal is shown to be effective in equipping a variety of different mainstream Federated Learning algorithms with novel class discovery and learning capability.
  arXiv  Detail & Related papers  (2023-12-21T00:31:54Z)
• ReCoRe: Regularized Contrastive Representation Learning of World Model [21.29132219042405]
  We present a world model that learns invariant features using contrastive unsupervised learning and an intervention-invariant regularizer. Our method outperforms current state-of-the-art model-based and model-free RL methods and significantly improves on out-of-distribution point navigation tasks evaluated on the iGibson benchmark.
  arXiv  Detail & Related papers  (2023-12-14T15:53:07Z)
• Learning to Operate in Open Worlds by Adapting Planning Models [12.513121330508477]
  Planning agents are ill-equipped to act in novel situations in which their domain model no longer accurately represents the world. We introduce an approach for such agents operating in open worlds that detects the presence of novelties and effectively adapts their domain models.
  arXiv  Detail & Related papers  (2023-03-24T21:04:16Z)
• Predictive Experience Replay for Continual Visual Control and Forecasting [62.06183102362871]
  We present a new continual learning approach for visual dynamics modeling and explore its efficacy in visual control and forecasting. We first propose the mixture world model that learns task-specific dynamics priors with a mixture of Gaussians, and then introduce a new training strategy to overcome catastrophic forgetting. Our model remarkably outperforms the naive combinations of existing continual learning and visual RL algorithms on DeepMind Control and Meta-World benchmarks with continual visual control tasks.
  arXiv  Detail & Related papers  (2023-03-12T05:08:03Z)
• Investigating the role of model-based learning in exploration and transfer [11.652741003589027]
  In this paper, we investigate transfer learning in the context of model-based agents. We find that a model-based approach outperforms controlled model-free baselines for transfer learning. Our results show that intrinsic exploration combined with environment models present a viable direction towards agents that are self-supervised and able to generalize to novel reward functions.
  arXiv  Detail & Related papers  (2023-02-08T11:49:58Z)
• Neuro-Symbolic World Models for Adapting to Open World Novelty [9.707805250772129]
  We introduce WorldCloner, an end-to-end trainable neuro-symbolic world model for rapid novelty adaptation. WorldCloner learns an efficient symbolic representation of the pre-novelty environment transitions. WorldCloner augments the policy learning process using imagination-based adaptation.
  arXiv  Detail & Related papers  (2023-01-16T07:49:12Z)
• Mastering the Unsupervised Reinforcement Learning Benchmark from Pixels [112.63440666617494]
  Reinforcement learning algorithms can succeed but require large amounts of interactions between the agent and the environment. We propose a new method to solve it, using unsupervised model-based RL, for pre-training the agent. We show robust performance on the Real-Word RL benchmark, hinting at resiliency to environment perturbations during adaptation.
  arXiv  Detail & Related papers  (2022-09-24T14:22:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.