Learning to Operate in Open Worlds by Adapting Planning Models

• URL: http://arxiv.org/abs/2303.14272v1
• Date: Fri, 24 Mar 2023 21:04:16 GMT
• Title: Learning to Operate in Open Worlds by Adapting Planning Models
• Authors: Wiktor Piotrowski and Roni Stern and Yoni Sher and Jacob Le and Matthew Klenk and Johan deKleer and Shiwali Mohan
• Abstract summary: Planning agents are ill-equipped to act in novel situations in which their domain model no longer accurately represents the world. We introduce an approach for such agents operating in open worlds that detects the presence of novelties and effectively adapts their domain models.
• Score: 12.513121330508477
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Planning agents are ill-equipped to act in novel situations in which their domain model no longer accurately represents the world. We introduce an approach for such agents operating in open worlds that detects the presence of novelties and effectively adapts their domain models and consequent action selection. It uses observations of action execution and measures their divergence from what is expected, according to the environment model, to infer existence of a novelty. Then, it revises the model through a heuristics-guided search over model changes. We report empirical evaluations on the CartPole problem, a standard Reinforcement Learning (RL) benchmark. The results show that our approach can deal with a class of novelties very quickly and in an interpretable fashion.

Related papers

• SPARTAN: A Sparse Transformer Learning Local Causation [63.29645501232935]
  Causal structures play a central role in world models that flexibly adapt to changes in the environment. We present the SPARse TrANsformer World model (SPARTAN), a Transformer-based world model that learns local causal structures between entities in a scene. By applying sparsity regularisation on the attention pattern between object-factored tokens, SPARTAN identifies sparse local causal models that accurately predict future object states.
  arXiv  Detail & Related papers  (2024-11-11T11:42:48Z)
• Adaptive World Models: Learning Behaviors by Latent Imagination Under Non-Stationarity [16.15952351162363]
  We introduce a new formalism, Hidden. POMDP, designed for control with adaptive world models. We demonstrate that this approach enables learning robust behaviors across a variety of non-stationary RL benchmarks.
  arXiv  Detail & Related papers  (2024-11-02T19:09:56Z)
• Partial Models for Building Adaptive Model-Based Reinforcement Learning Agents [37.604622216020765]
  We show that the conceptually simple idea of partial models can allow deep model-based agents to overcome this challenge. We demonstrate this by showing that the use of partial models in agents such as deep Dyna-Q, PlaNet and Dreamer can allow for them to effectively adapt to the local changes in their environments.
  arXiv  Detail & Related papers  (2024-05-27T07:46:36Z)
• STAT: Towards Generalizable Temporal Action Localization [56.634561073746056]
  Weakly-supervised temporal action localization (WTAL) aims to recognize and localize action instances with only video-level labels. Existing methods suffer from severe performance degradation when transferring to different distributions. We propose GTAL, which focuses on improving the generalizability of action localization methods.
  arXiv  Detail & Related papers  (2024-04-20T07:56:21Z)
• ReCoRe: Regularized Contrastive Representation Learning of World Model [21.29132219042405]
  We present a world model that learns invariant features using contrastive unsupervised learning and an intervention-invariant regularizer. Our method outperforms current state-of-the-art model-based and model-free RL methods and significantly improves on out-of-distribution point navigation tasks evaluated on the iGibson benchmark.
  arXiv  Detail & Related papers  (2023-12-14T15:53:07Z)
• Novelty Detection in Reinforcement Learning with World Models [15.01731216883798]
  Reinforcement learning (RL) using world models has found significant recent successes. However, when a sudden change to world mechanics or properties occurs then agent performance and reliability can dramatically decline. Implementing novelty detection within generated world model frameworks is a crucial task for protecting the agent when deployed.
  arXiv  Detail & Related papers  (2023-10-12T21:38:07Z)
• Predictive Experience Replay for Continual Visual Control and Forecasting [62.06183102362871]
  We present a new continual learning approach for visual dynamics modeling and explore its efficacy in visual control and forecasting. We first propose the mixture world model that learns task-specific dynamics priors with a mixture of Gaussians, and then introduce a new training strategy to overcome catastrophic forgetting. Our model remarkably outperforms the naive combinations of existing continual learning and visual RL algorithms on DeepMind Control and Meta-World benchmarks with continual visual control tasks.
  arXiv  Detail & Related papers  (2023-03-12T05:08:03Z)
• Investigating the role of model-based learning in exploration and transfer [11.652741003589027]
  In this paper, we investigate transfer learning in the context of model-based agents. We find that a model-based approach outperforms controlled model-free baselines for transfer learning. Our results show that intrinsic exploration combined with environment models present a viable direction towards agents that are self-supervised and able to generalize to novel reward functions.
  arXiv  Detail & Related papers  (2023-02-08T11:49:58Z)
• Online reinforcement learning with sparse rewards through an active inference capsule [62.997667081978825]
  This paper introduces an active inference agent which minimizes the novel free energy of the expected future. Our model is capable of solving sparse-reward problems with a very high sample efficiency. We also introduce a novel method for approximating the prior model from the reward function, which simplifies the expression of complex objectives.
  arXiv  Detail & Related papers  (2021-06-04T10:03:36Z)
• Bridging Imagination and Reality for Model-Based Deep Reinforcement Learning [72.18725551199842]
  We propose a novel model-based reinforcement learning algorithm, called BrIdging Reality and Dream (BIRD) It maximizes the mutual information between imaginary and real trajectories so that the policy improvement learned from imaginary trajectories can be easily generalized to real trajectories. We demonstrate that our approach improves sample efficiency of model-based planning, and achieves state-of-the-art performance on challenging visual control benchmarks.
  arXiv  Detail & Related papers  (2020-10-23T03:22:01Z)
• Goal-Aware Prediction: Learning to Model What Matters [105.43098326577434]
  One of the fundamental challenges in using a learned forward dynamics model is the mismatch between the objective of the learned model and that of the downstream planner or policy. We propose to direct prediction towards task relevant information, enabling the model to be aware of the current task and encouraging it to only model relevant quantities of the state space. We find that our method more effectively models the relevant parts of the scene conditioned on the goal, and as a result outperforms standard task-agnostic dynamics models and model-free reinforcement learning.
  arXiv  Detail & Related papers  (2020-07-14T16:42:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.