Reinforced Planning with Latent World Models

• URL: http://arxiv.org/abs/2608.18669v1
• Date: Wed, 19 Aug 2026 08:21:28 GMT
• Title: Reinforced Planning with Latent World Models
• Abstract summary: We introduce the Reinforced Planning, a method based on the idea that search can be learned by reinforcing good search rules into a neural planner.<n>Our implementation RP1 learns both how to evaluate imagined outcomes through a critic, as well as how to improve multi-step plans through an offline training.<n>RP1 substantially outperforms hand-designed search algorithms, reaching near-perfect success in several settings.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Humans solve complex problems by constructing plans and mentally simulating their outcomes with an internal model of the world. Machine learning has produced world models that similarly predict the outcomes of action sequences, but the improvement of candidate plans still isn't fully learned. Current planners are either hand-designed, distilled from a hand-designed optimizer, or learned only to inform an amortized policy rather than to revise the plan itself. We introduce the Reinforced Planning, a method based on the idea that search can be learned by reinforcing good search rules into a neural planner. Our implementation RP1 learns both how to evaluate imagined outcomes through a critic, as well as how to improve multi-step plans through an optimizer trained fully offline from imagined world-model roll-outs. To our knowledge, RP1 is the first method to fully learn how to improve multi-step plans. Furthermore, it can be trained independently of and attached to any pretrained latent world model. Across visual navigation, arm reaching, and robotic manipulation on two world-model backbones, RP1 substantially outperforms hand-designed search algorithms, reaching near-perfect success in several settings while using $1,000 \times$ less world-model rollouts and being up to $67 \times$ faster than the strongest alternative under concurrent planner inference.

Related papers

• IMWM: Intuition Models Complement World Models for Latent Planning [21.033541866844683]
  We show that a finite-budget sample-based planner still fails on some tasks, indicating that the bottleneck can lie in search rather than in world-model accuracy.<n>Motivated by this gap, we propose IMWM, which pairs the world model with an intuition model trained from demonstrations to recognize promising actions.
  arXiv  Detail & Related papers  (2026-06-01T03:23:25Z)
• Hierarchical Planning with Latent World Models [49.82095442483551]
  hierarchical planning achieves higher success while requiring up to 4x less planning-time compute.<n>We demonstrate that this hierarchical approach enables zero-shot control on real-world non-greedy robotic tasks.
  arXiv  Detail & Related papers  (2026-04-03T17:32:36Z)
• Closing the Train-Test Gap in World Models for Gradient-Based Planning [64.36544881136405]
  We propose improved methods for training world models that enable efficient gradient-based planning.<n>At test time, our approach outperforms or matches the classical gradient-free cross-entropy method.
  arXiv  Detail & Related papers  (2025-12-10T18:59:45Z)
• From Forecasting to Planning: Policy World Model for Collaborative State-Action Prediction [57.56072009935036]
  We introduce a new driving paradigm named Policy World Model (PWM)<n>PWM integrates world modeling and trajectory planning within a unified architecture.<n>Our method matches or exceeds state-of-the-art approaches that rely on multi-view and multi-modal inputs.
  arXiv  Detail & Related papers  (2025-10-22T14:57:51Z)
• Personalized Learning Path Planning with Goal-Driven Learner State Modeling [68.96987567845824]
  We introduce Pxplore, a novel framework for personalized learning paths.<n>We design a structured learner state model and an automated reward function that transforms abstract objectives into computable signals.<n>Experiments validate Pxplore's effectiveness in producing coherent, personalized, and goal-driven learning paths.
  arXiv  Detail & Related papers  (2025-10-15T06:59:49Z)
• A Goal Without a Plan Is Just a Wish: Efficient and Effective Global Planner Training for Long-Horizon Agent Tasks [66.86312354478478]
  Agents based on large language models (LLMs) struggle with brainless trial-and-error and generating hallucinatory actions due to a lack of global planning in long-horizon tasks.<n>We introduce a plan-and-execute framework and propose a planner training method to enhance the executor agent's planning abilities without human effort.<n>Experiments show that executor agents equipped with our planner outperform existing methods, achieving new state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-10-07T06:10:53Z)
• Enter the Void - Planning to Seek Entropy When Reward is Scarce [6.208654695856247]
  We propose a novel approach that anticipates and actively seeks out high-entropy states using short-horizon latent predictions.<n>We present a hierarchical planner that dynamically decides when to replan, planning horizon length, and the weighting between reward and entropy.<n>Our method finishes the Miniworld procedurally generated mazes 50% faster than base Dreamer at convergence and the policy trained in imagination converges in only 60% of the environment steps that base Dreamer needs.
  arXiv  Detail & Related papers  (2025-05-22T15:28:50Z)
• World Modeling Makes a Better Planner: Dual Preference Optimization for Embodied Task Planning [60.100794160682646]
  We propose a new learning framework that jointly optimize state prediction and action selection through preference learning.<n>To automatically collect trajectories and stepwise preference data without human annotation, we introduce a tree search mechanism for extensive exploration via trial-and-error.<n>Our method significantly outperforms existing methods and GPT-4o when applied to Qwen2-VL (7B), LLaVA-1.6 (7B), and LLaMA-3.2 (11B)
  arXiv  Detail & Related papers  (2025-03-13T15:49:56Z)
• Thinker: Learning to Plan and Act [18.425843346728648]
  The Thinker algorithm wraps the environment with a world model and introduces new actions designed for interacting with the world model. We demonstrate the algorithm's effectiveness through experimental results in the game of Sokoban and the Atari 2600 benchmark.
  arXiv  Detail & Related papers  (2023-07-27T16:40:14Z)
• Learning to Execute: Efficient Learning of Universal Plan-Conditioned Policies in Robotics [20.148408520475655]
  We introduce Learning to Execute (L2E), which leverages information contained in approximate plans to learn universal policies that are conditioned on plans. In our robotic manipulation experiments, L2E exhibits increased performance when compared to pure RL, pure planning, or baseline methods combining learning and planning.
  arXiv  Detail & Related papers  (2021-11-15T16:58:50Z)
• Evolutionary Planning in Latent Space [7.863826008567604]
  Planning is a powerful approach to reinforcement learning with several desirable properties. We learn a world model that enables Evolutionary Planning in Latent Space. We show how to build a model of the world by bootstrapping it with rollouts from a random policy and iteratively refining it with rollouts from an increasingly accurate planning policy.
  arXiv  Detail & Related papers  (2020-11-23T09:21:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.