When to Plan: Learning to Select Between Reactive Control and Deliberative Planning

• URL: http://arxiv.org/abs/2607.16421v1
• Date: Fri, 17 Jul 2026 18:13:38 GMT
• Title: When to Plan: Learning to Select Between Reactive Control and Deliberative Planning
• Abstract summary: We introduce an RL method for training a meta-reasoning policy that allocates computation by conditioning on a reactive-policy uncertainty score.<n>We conduct an empirical study on motion planning and navigation environments, showing that this design enables the meta-reasoning policy to learn when the reactive policy provides a good-enough action versus when decision-time planning is needed.
• Score: 11.723839544878244
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: It has long been recognized that humans have the ability to switch between fast, reactive decision-making and slower, deliberative planning. In this paper, we study the question of how to learn this ability, known as meta-reasoning, in artificial agents. We model reactive decision-making as a policy that directly maps state observations to actions. Such policies can be trained with reinforcement learning (RL) or imitation learning, but may generalize poorly outside of their training distribution. Alternatively, model-based decision-time planning is more likely to produce good actions across a broader set of states but requires additional computation time, which delays acting. In this work, we introduce an RL method for training a meta-reasoning policy that allocates computation by conditioning on a reactive-policy uncertainty score. This score enables it to predict when the reactive policy is likely to perform poorly and when planning is needed. We conduct an empirical study on motion planning and navigation environments, showing that this design enables the meta-reasoning policy to learn when the reactive policy provides a good-enough action versus when decision-time planning is needed. Additionally, we show that our design enables the meta-agent to shift toward fully reactive control as the reactive policy improves.

Related papers

• CAST: Alternating State-Value Targets and Expanded Policy Gradients for Model-Based Reinforcement Learning [6.586422759470378]
  We present CAST (Critic with Alternating State-value Target), which uses planner-guided behavior to improve value learning while regularizing the value estimate with the current policy.<n>We evaluate CAST on the DeepMind Control and HumanoidBench Suites against several state-of-the-art methods, and demonstrate successful transfer to a physical Unitree Go2ruped performing a dynamic handstand.
  arXiv  Detail & Related papers  (2026-09-08T15:05:16Z)
• When in Doubt, Plan It Out: Committed Small Language Model Deliberation for Reactive Reinforcement Learning [1.2526963688768455]
  Reinforcement Learning (RL) policies often degrade in unfamiliar environments because they lack explicit deliberation.<n>We propose Plan, Align, Commit, Think (PACT), a hybrid architecture that combines a fast, reactive RL policy with a slow, deliberative Small Language Model (SLM) planner.
  arXiv  Detail & Related papers  (2026-06-15T17:31:22Z)
• PriPG-RL: Privileged Planner-Guided Reinforcement Learning for Partially Observable Systems with Anytime-Feasible MPC [1.4312387415573748]
  We formalize a Partially Observable Markov Decision Process (POMDP) in which a planner agent guides a learning agent that observes only a lossy projection of the true state.<n>We propose Planner-to-Policy Soft Actor-Critic (P2P-SAC) to distill the planner agent's privileged knowledge to mitigate partial observability.<n>We validate our approach in simulation using NVIDIA Isaac Lab and successfully deploy it on a real-world Unitree Go2 quadruped navigating complex, obstacle-rich environments.
  arXiv  Detail & Related papers  (2026-04-09T09:41:11Z)
• When Should a Robot Think? Resource-Aware Reasoning via Reinforcement Learning for Embodied Robotic Decision-Making [68.12864562049957]
  Embodied robotic systems increasingly rely on large language model (LLM)-based agents to support high-level reasoning.<n>This raises a fundamental question for embodied agents: when should the agent reason, and when should it act?<n>We propose RARRL (Resource-Aware Reasoning via Reinforcement Learning), a hierarchical framework for resource-aware orchestration of embodied agents.
  arXiv  Detail & Related papers  (2026-03-17T15:38:50Z)
• Learning Optimal Deterministic Policies with Stochastic Policy Gradients [62.81324245896716]
  Policy gradient (PG) methods are successful approaches to deal with continuous reinforcement learning (RL) problems. In common practice, convergence (hyper)policies are learned only to deploy their deterministic version. We show how to tune the exploration level used for learning to optimize the trade-off between the sample complexity and the performance of the deployed deterministic policy.
  arXiv  Detail & Related papers  (2024-05-03T16:45:15Z)
• Integration of Reinforcement Learning Based Behavior Planning With Sampling Based Motion Planning for Automated Driving [0.5801044612920815]
  We propose a method to employ a trained deep reinforcement learning policy for dedicated high-level behavior planning. To the best of our knowledge, this work is the first to apply deep reinforcement learning in this manner.
  arXiv  Detail & Related papers  (2023-04-17T13:49:55Z)
• Automatically Learning Fallback Strategies with Model-Free Reinforcement Learning in Safety-Critical Driving Scenarios [9.761912672523977]
  We present a principled approach for a model-free Reinforcement Learning (RL) agent to capture multiple modes of behaviour in an environment. We introduce an extra pseudo-reward term to the reward model, to encourage exploration to areas of state-space different from areas privileged by the optimal policy. We show that we are able to learn useful policies that would have otherwise been missed out on during training, and unavailable to use when executing the control algorithm.
  arXiv  Detail & Related papers  (2022-04-11T15:34:49Z)
• Imitating, Fast and Slow: Robust learning from demonstrations via decision-time planning [96.72185761508668]
  Planning at Test-time (IMPLANT) is a new meta-algorithm for imitation learning. We demonstrate that IMPLANT significantly outperforms benchmark imitation learning approaches on standard control environments.
  arXiv  Detail & Related papers  (2022-04-07T17:16:52Z)
• Offline Reinforcement Learning with Implicit Q-Learning [85.62618088890787]
  Current offline reinforcement learning methods need to query the value of unseen actions during training to improve the policy. We propose an offline RL method that never needs to evaluate actions outside of the dataset. This method enables the learned policy to improve substantially over the best behavior in the data through generalization.
  arXiv  Detail & Related papers  (2021-10-12T17:05:05Z)
• TempoRL: Learning When to Act [34.8990562818147]
  Reinforcement learning is a powerful approach to learn behaviour through interactions with an environment. We propose a proactive setting in which the agent selects an action in a state and also for how long to commit to that action. Our TempoRL approach introduces skip connections between states and learns a skip-policy for repeating the same action along these skips. We demonstrate the effectiveness of TempoRL on a variety of traditional and deep RL environments, showing that our approach is capable of learning successful policies up to an order of magnitude faster than vanilla Q-learning.
  arXiv  Detail & Related papers  (2021-06-09T17:58:31Z)
• Learning Adaptive Exploration Strategies in Dynamic Environments Through Informed Policy Regularization [100.72335252255989]
  We study the problem of learning exploration-exploitation strategies that effectively adapt to dynamic environments. We propose a novel algorithm that regularizes the training of an RNN-based policy using informed policies trained to maximize the reward in each task.
  arXiv  Detail & Related papers  (2020-05-06T16:14:48Z)
• Efficient Deep Reinforcement Learning via Adaptive Policy Transfer [50.51637231309424]
  Policy Transfer Framework (PTF) is proposed to accelerate Reinforcement Learning (RL) Our framework learns when and which source policy is the best to reuse for the target policy and when to terminate it. Experimental results show it significantly accelerates the learning process and surpasses state-of-the-art policy transfer methods.
  arXiv  Detail & Related papers  (2020-02-19T07:30:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.