Planning as Emergent Behavior in Reinforcement Learning with Relational Hidden States

• URL: http://arxiv.org/abs/2607.18589v1
• Date: Mon, 20 Jul 2026 23:49:42 GMT
• Title: Planning as Emergent Behavior in Reinforcement Learning with Relational Hidden States
• Abstract summary: Recent work has shown that planning can emerge from model-free reinforcement learning alone.<n>We present evidence that, in the observed cases, the hidden-state structure of the neural architecture is the deciding factor.<n>We argue that this explains the observed phenomenon of emergent planning in model-free reinforcement learning.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Reinforcement learning is conventionally divided into model-based and model-free methods. In this taxonomy, model-based methods perform lookahead planning over a learned world model, whereas model-free methods learn a reactive state-action mapping. Recent work, however, has shown that planning can emerge from model-free reinforcement learning alone. The conditions under which this behavior emerges from a pure reward-maximization objective have so far remained unclear. In this paper, we present evidence that, in the observed cases, the hidden-state structure of the neural architecture is the deciding factor. We find that a network of relational hidden states, each anchored to an environment state and exchanging messages along learned relations, acquires a planning mechanism. These hidden states recover the environment's transition structure in their learned relations, and improve the policy at decision time by planning over the learned graph. In a matched control agent that must additionally discover which cells represent which states, no such binding arises, and no planning follows from it. We argue that this explains the observed phenomenon of emergent planning in model-free reinforcement learning and raises the question of how common such emergent planning might be more generally. Finally, we hypothesize that the discovered mechanism could describe how planning emerges from pure reward maximization in the human brain through a neural architectural prior.

Related papers

• Towards Zero-Shot Task Transfer with Neurosymbolic World Models [17.135921836577715]
  State-of-the-art model-based reinforcement learning methods learn neural world models that allow policy improvement by planning in a latent space.<n>We present a novel world model formulation where the reward prediction only depends on a subset of structured, symbolic components of the whole latent state.<n>We discuss the main advantages and challenges of learning these neurosymbolic world models and demonstrate the strong generalisation properties of our approach over purely neural methods.
  arXiv  Detail & Related papers  (2026-08-18T16:12:40Z)
• Differentiable Learning of Lifted Action Schemas for Classical Planning [6.30853901017231]
  We develop a novel neural network architecture for learning action schemas from traces where states are fully observed but action arguments are unobserved.<n>We evaluate the architecture on various planning domains, where the learned lifted action schemas must recover the ground-truth structure.
  arXiv  Detail & Related papers  (2026-05-13T09:59:49Z)
• World Action Models: The Next Frontier in Embodied AI [123.5787299299832]
  Vision-Language-Action (VLA) models have achieved strong semantic generalization for embodied policy learning.<n>They learn reactive observation-to-action mappings without explicitly modeling how the physical world evolves under intervention.<n>A growing body of work addresses this limitation by integrating world models, predictive models of environment dynamics, into the action generation pipeline.
  arXiv  Detail & Related papers  (2026-05-12T13:10:52Z)
• Interpreting Emergent Planning in Model-Free Reinforcement Learning [13.820891288919002]
  We present the first evidence that model-free reinforcement learning agents can learn to plan.<n>This is achieved by applying a methodology based on concept-based interpretability to a model-free agent in Sokoban.
  arXiv  Detail & Related papers  (2025-04-02T16:24:23Z)
• Leveraging Hierarchical Taxonomies in Prompt-based Continual Learning [41.13568563835089]
  We find that applying human habits of organizing and connecting information can serve as an efficient strategy when training deep learning models.<n>We propose a novel regularization loss function that encourages models to focus more on challenging knowledge areas.
  arXiv  Detail & Related papers  (2024-10-06T01:30:40Z)
• Interpretable Imitation Learning with Dynamic Causal Relations [65.18456572421702]
  We propose to expose captured knowledge in the form of a directed acyclic causal graph. We also design this causal discovery process to be state-dependent, enabling it to model the dynamics in latent causal graphs. The proposed framework is composed of three parts: a dynamic causal discovery module, a causality encoding module, and a prediction module, and is trained in an end-to-end manner.
  arXiv  Detail & Related papers  (2023-09-30T20:59:42Z)
• Learning Visual Planning Models from Partially Observed Images [17.25694427734666]
  We provide a novel framework, aTypeRecplan, for learning a transition model from partially observed raw image traces. We also propose a neural-network-based approach to learn a model that estimates the distance toward a given goal observation. Our approach is more effective than a state-of-the-art approach of learning visual planning models in the environment with incomplete observations.
  arXiv  Detail & Related papers  (2022-11-25T07:00:56Z)
• Planning with Diffusion for Flexible Behavior Synthesis [125.24438991142573]
  We consider what it would look like to fold as much of the trajectory optimization pipeline as possible into the modeling problem. The core of our technical approach lies in a diffusion probabilistic model that plans by iteratively denoising trajectories.
  arXiv  Detail & Related papers  (2022-05-20T07:02:03Z)
• Behavior Priors for Efficient Reinforcement Learning [97.81587970962232]
  We consider how information and architectural constraints can be combined with ideas from the probabilistic modeling literature to learn behavior priors. We discuss how such latent variable formulations connect to related work on hierarchical reinforcement learning (HRL) and mutual information and curiosity based objectives. We demonstrate the effectiveness of our framework by applying it to a range of simulated continuous control domains.
  arXiv  Detail & Related papers  (2020-10-27T13:17:18Z)
• Developing Constrained Neural Units Over Time [81.19349325749037]
  This paper focuses on an alternative way of defining Neural Networks, that is different from the majority of existing approaches. The structure of the neural architecture is defined by means of a special class of constraints that are extended also to the interaction with data. The proposed theory is cast into the time domain, in which data are presented to the network in an ordered manner.
  arXiv  Detail & Related papers  (2020-09-01T09:07:25Z)
• Goal-Aware Prediction: Learning to Model What Matters [105.43098326577434]
  One of the fundamental challenges in using a learned forward dynamics model is the mismatch between the objective of the learned model and that of the downstream planner or policy. We propose to direct prediction towards task relevant information, enabling the model to be aware of the current task and encouraging it to only model relevant quantities of the state space. We find that our method more effectively models the relevant parts of the scene conditioned on the goal, and as a result outperforms standard task-agnostic dynamics models and model-free reinforcement learning.
  arXiv  Detail & Related papers  (2020-07-14T16:42:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.