Interpreting Emergent Planning in Model-Free Reinforcement Learning

• URL: http://arxiv.org/abs/2504.01871v1
• Date: Wed, 02 Apr 2025 16:24:23 GMT
• Title: Interpreting Emergent Planning in Model-Free Reinforcement Learning
• Authors: Thomas Bush, Stephen Chung, Usman Anwar, Adrià Garriga-Alonso, David Krueger,
• Abstract summary: We present the first evidence that model-free reinforcement learning agents can learn to plan.<n>This is achieved by applying a methodology based on concept-based interpretability to a model-free agent in Sokoban.
• Score: 13.820891288919002
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present the first mechanistic evidence that model-free reinforcement learning agents can learn to plan. This is achieved by applying a methodology based on concept-based interpretability to a model-free agent in Sokoban -- a commonly used benchmark for studying planning. Specifically, we demonstrate that DRC, a generic model-free agent introduced by Guez et al. (2019), uses learned concept representations to internally formulate plans that both predict the long-term effects of actions on the environment and influence action selection. Our methodology involves: (1) probing for planning-relevant concepts, (2) investigating plan formation within the agent's representations, and (3) verifying that discovered plans (in the agent's representations) have a causal effect on the agent's behavior through interventions. We also show that the emergence of these plans coincides with the emergence of a planning-like property: the ability to benefit from additional test-time compute. Finally, we perform a qualitative analysis of the planning algorithm learned by the agent and discover a strong resemblance to parallelized bidirectional search. Our findings advance understanding of the internal mechanisms underlying planning behavior in agents, which is important given the recent trend of emergent planning and reasoning capabilities in LLMs through RL

Related papers

• Latent Diffusion Planning for Imitation Learning [78.56207566743154]
  Latent Diffusion Planning (LDP) is a modular approach consisting of a planner and inverse dynamics model. By separating planning from action prediction, LDP can benefit from the denser supervision signals of suboptimal and action-free data. On simulated visual robotic manipulation tasks, LDP outperforms state-of-the-art imitation learning approaches.
  arXiv  Detail & Related papers  (2025-04-23T17:53:34Z)
• DHP: Discrete Hierarchical Planning for Hierarchical Reinforcement Learning Agents [2.1438108757511958]
  Our key contribution is a Discrete Hierarchical Planning (DHP) method, an alternative to traditional distance-based approaches.<n>We provide theoretical foundations for the method and demonstrate its effectiveness through extensive empirical evaluations.<n>We evaluate our method on long-horizon visual planning tasks in a 25-room environment, where it significantly outperforms previous benchmarks at success rate and average episode length.
  arXiv  Detail & Related papers  (2025-02-04T03:05:55Z)
• Model-Free RL Agents Demonstrate System 1-Like Intentionality [16.427085062620215]
  We argue that model-free reinforcement learning agents exhibit behaviours that can be analogised to System 1 processes in human cognition.<n>We propose a novel framework linking the dichotomy of System 1 and System 2 to the distinction between model-free and model-based RL.
  arXiv  Detail & Related papers  (2025-01-30T12:21:50Z)
• ACT-JEPA: Novel Joint-Embedding Predictive Architecture for Efficient Policy Representation Learning [90.41852663775086]
  ACT-JEPA is a novel architecture that integrates imitation learning and self-supervised learning. We train a policy to predict action sequences and abstract observation sequences. Our experiments show that ACT-JEPA improves the quality of representations by learning temporal environment dynamics.
  arXiv  Detail & Related papers  (2025-01-24T16:41:41Z)
• Predicting Future Actions of Reinforcement Learning Agents [27.6973598477153]
  This paper experimentally evaluates and compares the effectiveness of future action and event prediction for three types of reinforcement learning agents. We employ two approaches: the inner state approach, which involves predicting based on the inner computations of the agents, and a simulation-based approach, which involves unrolling the agent in a learned world model. Using internal plans proves more robust to model quality compared to simulation-based approaches when predicting actions, while the results for event prediction are more mixed.
  arXiv  Detail & Related papers  (2024-10-29T18:48:18Z)
• Exploring and Benchmarking the Planning Capabilities of Large Language Models [57.23454975238014]
  This work lays the foundations for improving planning capabilities of large language models (LLMs) We construct a comprehensive benchmark suite encompassing both classical planning benchmarks and natural language scenarios. We investigate the use of many-shot in-context learning to enhance LLM planning, exploring the relationship between increased context length and improved planning performance.
  arXiv  Detail & Related papers  (2024-06-18T22:57:06Z)
• On Predictive planning and counterfactual learning in active inference [0.20482269513546453]
  In this paper, we examine two decision-making schemes in active inference based on 'planning' and 'learning from experience' We introduce a mixed model that navigates the data-complexity trade-off between these strategies. We evaluate our proposed model in a challenging grid-world scenario that requires adaptability from the agent.
  arXiv  Detail & Related papers  (2024-03-19T04:02:31Z)
• Understanding the planning of LLM agents: A survey [98.82513390811148]
  This survey provides the first systematic view of LLM-based agents planning, covering recent works aiming to improve planning ability. Comprehensive analyses are conducted for each direction, and further challenges in the field of research are discussed.
  arXiv  Detail & Related papers  (2024-02-05T04:25:24Z)
• Planning as In-Painting: A Diffusion-Based Embodied Task Planning Framework for Environments under Uncertainty [56.30846158280031]
  Task planning for embodied AI has been one of the most challenging problems. We propose a task-agnostic method named 'planning as in-painting' The proposed framework achieves promising performances in various embodied AI tasks.
  arXiv  Detail & Related papers  (2023-12-02T10:07:17Z)
• Interpretable Imitation Learning with Dynamic Causal Relations [65.18456572421702]
  We propose to expose captured knowledge in the form of a directed acyclic causal graph. We also design this causal discovery process to be state-dependent, enabling it to model the dynamics in latent causal graphs. The proposed framework is composed of three parts: a dynamic causal discovery module, a causality encoding module, and a prediction module, and is trained in an end-to-end manner.
  arXiv  Detail & Related papers  (2023-09-30T20:59:42Z)
• A Look at Value-Based Decision-Time vs. Background Planning Methods Across Different Settings [41.606112019744174]
  We study how the value-based versions of decision-time and background planning methods will compare against each other across different settings. Overall, our findings suggest that even though value-based versions of the two planning methods perform on par in their simplest instantiations, the modern instantiations of value-based decision-time planning methods can perform on par or better than the modern instantiations of value-based background planning methods.
  arXiv  Detail & Related papers  (2022-06-16T20:48:19Z)
• A Consciousness-Inspired Planning Agent for Model-Based Reinforcement Learning [104.3643447579578]
  We present an end-to-end, model-based deep reinforcement learning agent which dynamically attends to relevant parts of its state. The design allows agents to learn to plan effectively, by attending to the relevant objects, leading to better out-of-distribution generalization.
  arXiv  Detail & Related papers  (2021-06-03T19:35:19Z)
• Forethought and Hindsight in Credit Assignment [62.05690959741223]
  We work to understand the gains and peculiarities of planning employed as forethought via forward models or as hindsight operating with backward models. We investigate the best use of models in planning, primarily focusing on the selection of states in which predictions should be (re)-evaluated.
  arXiv  Detail & Related papers  (2020-10-26T16:00:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.