Neuro-Symbolic World Models for Adapting to Open World Novelty

• URL: http://arxiv.org/abs/2301.06294v1
• Date: Mon, 16 Jan 2023 07:49:12 GMT
• Title: Neuro-Symbolic World Models for Adapting to Open World Novelty
• Authors: Jonathan Balloch and Zhiyu Lin and Robert Wright and Xiangyu Peng and Mustafa Hussain and Aarun Srinivas and Julia Kim and Mark O. Riedl
• Abstract summary: We introduce WorldCloner, an end-to-end trainable neuro-symbolic world model for rapid novelty adaptation. WorldCloner learns an efficient symbolic representation of the pre-novelty environment transitions. WorldCloner augments the policy learning process using imagination-based adaptation.
• Score: 9.707805250772129
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Open-world novelty--a sudden change in the mechanics or properties of an environment--is a common occurrence in the real world. Novelty adaptation is an agent's ability to improve its policy performance post-novelty. Most reinforcement learning (RL) methods assume that the world is a closed, fixed process. Consequentially, RL policies adapt inefficiently to novelties. To address this, we introduce WorldCloner, an end-to-end trainable neuro-symbolic world model for rapid novelty adaptation. WorldCloner learns an efficient symbolic representation of the pre-novelty environment transitions, and uses this transition model to detect novelty and efficiently adapt to novelty in a single-shot fashion. Additionally, WorldCloner augments the policy learning process using imagination-based adaptation, where the world model simulates transitions of the post-novelty environment to help the policy adapt. By blending ''imagined'' transitions with interactions in the post-novelty environment, performance can be recovered with fewer total environment interactions. Using environments designed for studying novelty in sequential decision-making problems, we show that the symbolic world model helps its neural policy adapt more efficiently than model-based and model-based neural-only reinforcement learning methods.

Related papers

• AdaWorld: Learning Adaptable World Models with Latent Actions [76.50869178593733]
  We propose AdaWorld, an innovative world model learning approach that enables efficient adaptation. Key idea is to incorporate action information during the pretraining of world models. We then develop an autoregressive world model that conditions on these latent actions.
  arXiv  Detail & Related papers  (2025-03-24T17:58:15Z)
• Disentangled World Models: Learning to Transfer Semantic Knowledge from Distracting Videos for Reinforcement Learning [93.58897637077001]
  This paper tries to learn and understand underlying semantic variations from distracting videos via offline-to-online latent distillation and flexible disentanglement constraints. We pretrain the action-free video prediction model offline with disentanglement regularization to extract semantic knowledge from distracting videos. For finetuning in the online environment, we exploit the knowledge from the pretrained model and introduce a disentanglement constraint to the world model.
  arXiv  Detail & Related papers  (2025-03-11T13:50:22Z)
• Learning Transformer-based World Models with Contrastive Predictive Coding [58.0159270859475]
  We show that the next state prediction objective is insufficient to fully exploit the representation capabilities of Transformers. We propose to extend world model predictions to longer time horizons by introducing TWISTER, a world model using action-conditioned Contrastive Predictive Coding. TWISTER achieves a human-normalized mean score of 162% on the Atari 100k benchmark, setting a new record among state-of-the-art methods that do not employ look-ahead search.
  arXiv  Detail & Related papers  (2025-03-06T13:18:37Z)
• Multimodal Dreaming: A Global Workspace Approach to World Model-Based Reinforcement Learning [2.5749046466046903]
  In Reinforcement Learning (RL), world models aim to capture how the environment evolves in response to the agent's actions. We show that performing the dreaming process inside the latent space allows for training with fewer environment steps. We conclude that the combination of GW with World Models holds great potential for improving decision-making in RL agents.
  arXiv  Detail & Related papers  (2025-02-28T15:24:17Z)
• Training a Generally Curious Agent [86.84089201249104]
  We present PAPRIKA, a fine-tuning approach that enables language models to develop general decision-making capabilities. Experimental results show that models fine-tuned with PAPRIKA can effectively transfer their learned decision-making capabilities to entirely unseen tasks. These results suggest a promising path towards AI systems that can autonomously solve novel sequential decision-making problems.
  arXiv  Detail & Related papers  (2025-02-24T18:56:58Z)
• Robotic World Model: A Neural Network Simulator for Robust Policy Optimization in Robotics [50.191655141020505]
  This work advances model-based reinforcement learning by addressing the challenges of long-horizon prediction, error accumulation, and sim-to-real transfer. By providing a scalable and robust framework, the introduced methods pave the way for adaptive and efficient robotic systems in real-world applications.
  arXiv  Detail & Related papers  (2025-01-17T10:39:09Z)
• Open-World Reinforcement Learning over Long Short-Term Imagination [91.28998327423295]
  We present LS-Imagine, which extends the imagination horizon within a limited number of state transition steps. Our method demonstrates significant improvements over state-of-the-art techniques in MineDojo.
  arXiv  Detail & Related papers  (2024-10-04T17:17:30Z)
• Mitigating Covariate Shift in Imitation Learning for Autonomous Vehicles Using Latent Space Generative World Models [60.87795376541144]
  A world model is a neural network capable of predicting an agent's next state given past states and actions. During end-to-end training, our policy learns how to recover from errors by aligning with states observed in human demonstrations. We present qualitative and quantitative results, demonstrating significant improvements upon prior state of the art in closed-loop testing.
  arXiv  Detail & Related papers  (2024-09-25T06:48:25Z)
• One-shot World Models Using a Transformer Trained on a Synthetic Prior [37.027893127637036]
  One-Shot World Model (OSWM) is a transformer world model that is learned in an in-context learning fashion from purely synthetic data. OSWM is able to quickly adapt to the dynamics of a simple grid world, as well as the CartPole gym and a custom control environment.
  arXiv  Detail & Related papers  (2024-09-21T09:39:32Z)
• Learning from Random Demonstrations: Offline Reinforcement Learning with Importance-Sampled Diffusion Models [19.05224410249602]
  We propose a novel approach for offline reinforcement learning with closed-loop policy evaluation and world-model adaptation. We analyzed the performance of the proposed method and provided an upper bound on the return gap between our method and the real environment under an optimal policy.
  arXiv  Detail & Related papers  (2024-05-30T09:34:31Z)
• Efficient Imitation Learning with Conservative World Models [54.52140201148341]
  We tackle the problem of policy learning from expert demonstrations without a reward function. We re-frame imitation learning as a fine-tuning problem, rather than a pure reinforcement learning one.
  arXiv  Detail & Related papers  (2024-05-21T20:53:18Z)
• Novelty Detection in Reinforcement Learning with World Models [15.01731216883798]
  Reinforcement learning (RL) using world models has found significant recent successes. However, when a sudden change to world mechanics or properties occurs then agent performance and reliability can dramatically decline. Implementing novelty detection within generated world model frameworks is a crucial task for protecting the agent when deployed.
  arXiv  Detail & Related papers  (2023-10-12T21:38:07Z)
• Novelty Accommodating Multi-Agent Planning in High Fidelity Simulated Open World [0.0]
  Novelty is an unexpected phenomenon that can alter the core characteristics, composition, and dynamics of the environment. Previous studies show that novelty has catastrophic impact on agent performance. In this work, we demonstrate that a domain-independent AI agent can be adapted to successfully perform and reason with novelty in realistic high-fidelity simulator of the military domain.
  arXiv  Detail & Related papers  (2023-06-22T03:44:04Z)
• Learning to Operate in Open Worlds by Adapting Planning Models [12.513121330508477]
  Planning agents are ill-equipped to act in novel situations in which their domain model no longer accurately represents the world. We introduce an approach for such agents operating in open worlds that detects the presence of novelties and effectively adapts their domain models.
  arXiv  Detail & Related papers  (2023-03-24T21:04:16Z)
• Predictive Experience Replay for Continual Visual Control and Forecasting [62.06183102362871]
  We present a new continual learning approach for visual dynamics modeling and explore its efficacy in visual control and forecasting. We first propose the mixture world model that learns task-specific dynamics priors with a mixture of Gaussians, and then introduce a new training strategy to overcome catastrophic forgetting. Our model remarkably outperforms the naive combinations of existing continual learning and visual RL algorithms on DeepMind Control and Meta-World benchmarks with continual visual control tasks.
  arXiv  Detail & Related papers  (2023-03-12T05:08:03Z)
• Real-World Humanoid Locomotion with Reinforcement Learning [92.85934954371099]
  We present a fully learning-based approach for real-world humanoid locomotion. Our controller can walk over various outdoor terrains, is robust to external disturbances, and can adapt in context.
  arXiv  Detail & Related papers  (2023-03-06T18:59:09Z)
• Environment Shaping in Reinforcement Learning using State Abstraction [63.444831173608605]
  We propose a novel framework of emphenvironment shaping using state abstraction. Our key idea is to compress the environment's large state space with noisy signals to an abstracted space. We show that the agent's policy learnt in the shaped environment preserves near-optimal behavior in the original environment.
  arXiv  Detail & Related papers  (2020-06-23T17:00:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.