Synthesizing world models for bilevel planning

• URL: http://arxiv.org/abs/2503.20124v1
• Date: Wed, 26 Mar 2025 00:10:01 GMT
• Title: Synthesizing world models for bilevel planning
• Authors: Zergham Ahmed, Joshua B. Tenenbaum, Christopher J. Bates, Samuel J. Gershman,
• Abstract summary: Theory-based reinforcement learning (TBRL) is an algorithmic framework specifically designed to address this gap.<n>TBRL exploits hierarchical representations of theories and efficient program synthesis methods for more powerful learning and planning.<n>We demonstrate that this approach can be successfully applied to diverse and challenging grid-world games, where approaches based on directly synthesizing a policy perform poorly.
• Score: 46.21010194281677
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern reinforcement learning (RL) systems have demonstrated remarkable capabilities in complex environments, such as video games. However, they still fall short of achieving human-like sample efficiency and adaptability when learning new domains. Theory-based reinforcement learning (TBRL) is an algorithmic framework specifically designed to address this gap. Modeled on cognitive theories, TBRL leverages structured, causal world models - "theories" - as forward simulators for use in planning, generalization and exploration. Although current TBRL systems provide compelling explanations of how humans learn to play video games, they face several technical limitations: their theory languages are restrictive, and their planning algorithms are not scalable. To address these challenges, we introduce TheoryCoder, an instantiation of TBRL that exploits hierarchical representations of theories and efficient program synthesis methods for more powerful learning and planning. TheoryCoder equips agents with general-purpose abstractions (e.g., "move to"), which are then grounded in a particular environment by learning a low-level transition model (a Python program synthesized from observations by a large language model). A bilevel planning algorithm can exploit this hierarchical structure to solve large domains. We demonstrate that this approach can be successfully applied to diverse and challenging grid-world games, where approaches based on directly synthesizing a policy perform poorly. Ablation studies demonstrate the benefits of using hierarchical abstractions.

Related papers

• Training Cross-Morphology Embodied AI Agents: From Practical Challenges to Theoretical Foundations [16.735655028118817]
  This article shows that theoretical insight is essential for overcoming real-world engineering barriers.<n>We formalize the Heterogeneous Embodied Agent Training (HEAT) problem and prove it reduces to a structured Partially Observable Markov Decision Process (POMDP) that is PSPACE-complete.<n>We also explore Collective Adaptation, a distributed learning alternative inspired by biological systems.
  arXiv  Detail & Related papers  (2025-06-04T06:44:49Z)
• GenRL: Multimodal-foundation world models for generalization in embodied agents [12.263162194821787]
  Reinforcement learning (RL) is hard to scale up as it requires a complex reward design for each task. Current foundation vision-language models (VLMs) require fine-tuning or other adaptations to be adopted in embodied contexts. Lack of multimodal data in such domains represents an obstacle to developing foundation models for embodied applications.
  arXiv  Detail & Related papers  (2024-06-26T03:41:48Z)
• Improving Complex Reasoning over Knowledge Graph with Logic-Aware Curriculum Tuning [89.89857766491475]
  We propose a curriculum-based logical-aware instruction tuning framework, named LACT.<n>Specifically, we augment the arbitrary first-order logical queries via binary tree decomposition.<n> Experiments across widely used datasets demonstrate that LACT has substantial improvements(brings an average +5.5% MRR score) over advanced methods, achieving the new state-of-the-art.
  arXiv  Detail & Related papers  (2024-05-02T18:12:08Z)
• A General Framework for Learning from Weak Supervision [93.89870459388185]
  This paper introduces a general framework for learning from weak supervision (GLWS) with a novel algorithm. Central to GLWS is an Expectation-Maximization (EM) formulation, adeptly accommodating various weak supervision sources. We also present an advanced algorithm that significantly simplifies the EM computational demands.
  arXiv  Detail & Related papers  (2024-02-02T21:48:50Z)
• Neural Causal Abstractions [63.21695740637627]
  We develop a new family of causal abstractions by clustering variables and their domains. We show that such abstractions are learnable in practical settings through Neural Causal Models. Our experiments support the theory and illustrate how to scale causal inferences to high-dimensional settings involving image data.
  arXiv  Detail & Related papers  (2024-01-05T02:00:27Z)
• The RL Perceptron: Generalisation Dynamics of Policy Learning in High Dimensions [13.774600272141761]
  Reinforcement learning algorithms have proven transformative in a range of domains.<n>Much theory of RL has focused on discrete state spaces or worst-case analysis.<n>We propose a solvable high-dimensional model of RL that can capture a variety of learning protocols.
  arXiv  Detail & Related papers  (2023-06-17T18:16:51Z)
• Mesarovician Abstract Learning Systems [0.0]
  Current approaches to learning hold notions of problem domain and problem task as fundamental precepts. Mesarovician abstract systems theory is used as a super-structure for learning.
  arXiv  Detail & Related papers  (2021-11-29T18:17:32Z)
• Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning [27.593497502386143]
  Theory-Based Reinforcement Learning uses human-like intuitive theories to explore and model an environment. We instantiate the approach in a video game playing agent called EMPA. EMPA matches human learning efficiency on a suite of 90 Atari-style video games.
  arXiv  Detail & Related papers  (2021-07-27T01:38:13Z)
• World Model as a Graph: Learning Latent Landmarks for Planning [12.239590266108115]
  Planning is a hallmark of human intelligence. One prominent framework, Model-Based RL, learns a world model and plans using step-by-step virtual rollouts. We propose to learn graph-structured world models composed of sparse, multi-step transitions.
  arXiv  Detail & Related papers  (2020-11-25T02:49:21Z)
• Discovering Reinforcement Learning Algorithms [53.72358280495428]
  Reinforcement learning algorithms update an agent's parameters according to one of several possible rules. This paper introduces a new meta-learning approach that discovers an entire update rule. It includes both 'what to predict' (e.g. value functions) and 'how to learn from it' by interacting with a set of environments.
  arXiv  Detail & Related papers  (2020-07-17T07:38:39Z)
• Provably Efficient Exploration for Reinforcement Learning Using Unsupervised Learning [96.78504087416654]
  Motivated by the prevailing paradigm of using unsupervised learning for efficient exploration in reinforcement learning (RL) problems, we investigate when this paradigm is provably efficient. We present a general algorithmic framework that is built upon two components: an unsupervised learning algorithm and a noregret tabular RL algorithm.
  arXiv  Detail & Related papers  (2020-03-15T19:23:59Z)
• Information Theoretic Model Predictive Q-Learning [64.74041985237105]
  We present a novel theoretical connection between information theoretic MPC and entropy regularized RL. We develop a Q-learning algorithm that can leverage biased models.
  arXiv  Detail & Related papers  (2019-12-31T00:29:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.