Efficient Exploration and Discriminative World Model Learning with an Object-Centric Abstraction

• URL: http://arxiv.org/abs/2408.11816v1
• Date: Wed, 21 Aug 2024 17:59:31 GMT
• Title: Efficient Exploration and Discriminative World Model Learning with an Object-Centric Abstraction
• Authors: Anthony GX-Chen, Kenneth Marino, Rob Fergus,
• Abstract summary: We study whether giving an agent an object-centric mapping (describing a set of items and their attributes) allow for more efficient learning. We find this problem is best solved hierarchically by modelling items at a higher level of state abstraction to pixels. We make use of this to propose a fully model-based algorithm that learns a discriminative world model.
• Score: 19.59151245929067
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In the face of difficult exploration problems in reinforcement learning, we study whether giving an agent an object-centric mapping (describing a set of items and their attributes) allow for more efficient learning. We found this problem is best solved hierarchically by modelling items at a higher level of state abstraction to pixels, and attribute change at a higher level of temporal abstraction to primitive actions. This abstraction simplifies the transition dynamic by making specific future states easier to predict. We make use of this to propose a fully model-based algorithm that learns a discriminative world model, plans to explore efficiently with only a count-based intrinsic reward, and can subsequently plan to reach any discovered (abstract) states. We demonstrate the model's ability to (i) efficiently solve single tasks, (ii) transfer zero-shot and few-shot across item types and environments, and (iii) plan across long horizons. Across a suite of 2D crafting and MiniHack environments, we empirically show our model significantly out-performs state-of-the-art low-level methods (without abstraction), as well as performant model-free and model-based methods using the same abstraction. Finally, we show how to reinforce learn low level object-perturbing policies, as well as supervise learn the object mapping itself.

Related papers

• SOLD: Reinforcement Learning with Slot Object-Centric Latent Dynamics [16.020835290802548]
  Slot-Attention for Object-centric Latent Dynamics is a novel algorithm that learns object-centric dynamics models from pixel inputs. We demonstrate that the structured latent space not only improves model interpretability but also provides a valuable input space for behavior models to reason over. Our results show that SOLD outperforms DreamerV3, a state-of-the-art model-based RL algorithm, across a range of benchmark robotic environments.
  arXiv  Detail & Related papers  (2024-10-11T14:03:31Z)
• Building Minimal and Reusable Causal State Abstractions for Reinforcement Learning [63.58935783293342]
  Causal Bisimulation Modeling (CBM) is a method that learns the causal relationships in the dynamics and reward functions for each task to derive a minimal, task-specific abstraction. CBM's learned implicit dynamics models identify the underlying causal relationships and state abstractions more accurately than explicit ones.
  arXiv  Detail & Related papers  (2024-01-23T05:43:15Z)
• Forecaster: Towards Temporally Abstract Tree-Search Planning from Pixels [42.275164872809746]
  We introduce Forecaster, a deep hierarchical reinforcement learning approach which plans over high-level goals. Forecaster learns an abstract model of its environment by modelling the transitions dynamics at an abstract level. It then uses this world model to choose optimal high-level goals through a tree-search planning procedure.
  arXiv  Detail & Related papers  (2023-10-16T01:13:26Z)
• Weakly-supervised Contrastive Learning for Unsupervised Object Discovery [52.696041556640516]
  Unsupervised object discovery is promising due to its ability to discover objects in a generic manner. We design a semantic-guided self-supervised learning model to extract high-level semantic features from images. We introduce Principal Component Analysis (PCA) to localize object regions.
  arXiv  Detail & Related papers  (2023-07-07T04:03:48Z)
• Causal Dynamics Learning for Task-Independent State Abstraction [61.707048209272884]
  We introduce Causal Dynamics Learning for Task-Independent State Abstraction (CDL) CDL learns a theoretically proved causal dynamics model that removes unnecessary dependencies between state variables and the action. A state abstraction can then be derived from the learned dynamics.
  arXiv  Detail & Related papers  (2022-06-27T17:02:53Z)
• Online Learning of Reusable Abstract Models for Object Goal Navigation [18.15382773079023]
  We present a novel approach to incrementally learn an Abstract Model of an unknown environment. We show how an agent can reuse the learned model for tackling the Object Goal Navigation task.
  arXiv  Detail & Related papers  (2022-03-04T21:44:43Z)
• Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning [120.38381203153159]
  Reinforcement learning can train policies that effectively perform complex tasks. For long-horizon tasks, the performance of these methods degrades with horizon, often necessitating reasoning over and composing lower-level skills. We propose Value Function Spaces: a simple approach that produces such a representation by using the value functions corresponding to each lower-level skill.
  arXiv  Detail & Related papers  (2021-11-04T22:46:16Z)
• Dynamic probabilistic logic models for effective abstractions in RL [35.54018388244684]
  RePReL is a hierarchical framework that leverages a relational planner to provide useful state abstractions for learning. Our experiments show that RePReL not only achieves better performance and efficient learning on the task at hand but also demonstrates better generalization to unseen tasks.
  arXiv  Detail & Related papers  (2021-10-15T18:53:04Z)
• Model-free Representation Learning and Exploration in Low-rank MDPs [64.72023662543363]
  We present the first model-free representation learning algorithms for low rank MDPs. Key algorithmic contribution is a new minimax representation learning objective. Result can accommodate general function approximation to scale to complex environments.
  arXiv  Detail & Related papers  (2021-02-14T00:06:54Z)
• Goal-Aware Prediction: Learning to Model What Matters [105.43098326577434]
  One of the fundamental challenges in using a learned forward dynamics model is the mismatch between the objective of the learned model and that of the downstream planner or policy. We propose to direct prediction towards task relevant information, enabling the model to be aware of the current task and encouraging it to only model relevant quantities of the state space. We find that our method more effectively models the relevant parts of the scene conditioned on the goal, and as a result outperforms standard task-agnostic dynamics models and model-free reinforcement learning.
  arXiv  Detail & Related papers  (2020-07-14T16:42:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.