When in Doubt, Plan It Out: Committed Small Language Model Deliberation for Reactive Reinforcement Learning

• URL: http://arxiv.org/abs/2606.16995v1
• Date: Mon, 15 Jun 2026 17:31:22 GMT
• Title: When in Doubt, Plan It Out: Committed Small Language Model Deliberation for Reactive Reinforcement Learning
• Abstract summary: Reinforcement Learning (RL) policies often degrade in unfamiliar environments because they lack explicit deliberation.<n>We propose Plan, Align, Commit, Think (PACT), a hybrid architecture that combines a fast, reactive RL policy with a slow, deliberative Small Language Model (SLM) planner.
• Score: 1.2526963688768455
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement Learning (RL) policies often degrade in unfamiliar environments because they lack explicit deliberation. We propose Plan, Align, Commit, Think (PACT), a hybrid architecture that combines a fast, reactive RL policy with a slow, deliberative Small Language Model (SLM) planner. PACT invokes the SLM asynchronously to generate and validate candidate action plans. Once a plan is verified through simulation as safe, feasible, and complete, it is executed directly, bypassing the RL policy without retraining or modifying it. Evaluated on three FrozenLake configurations of increasing difficulty, PACT outperforms all baselines while relying on a 2B-parameter SLM backbone, suggesting that deliberative planning and reactive execution are more powerful in concert than either is alone in these settings.

Related papers

• Policy-Invariant Reward Shaping from LLM Feedback: A Framework for Hybrid RL Agents [0.0]
  We formalize the hybrid LLM-planner and RL-controller architecture as a Goal-Augmented Markov Decision Process.<n>We show that when the LLM per-state progress score is used as a bounded potential function, the resulting shaping term preserves the optimal policy set even when the LLM scores are inaccurate.
  arXiv  Detail & Related papers  (2026-08-18T16:55:46Z)
• Hybrid LLM-Augmented Reinforcement Learning Agents for Complex Sequential Decision Tasks [0.0]
  Large Language Models (LLMs) have recently shown strong capabilities in reasoning, planning, and tool-use, enabling new forms of autonomous agents.<n>This paper introduces an LLM-Augmented Reinforcement Learning Agent that integrates LLM-driven planning with RL-based action optimization.
  arXiv  Detail & Related papers  (2026-08-04T11:44:07Z)
• Agentic Incident Response through Digital Twin-Enhanced Multiscale Planning [3.733830491643117]
  Incident response is currently managed by security operators using predefined playbooks.<n>Decision-theoretic approaches based on control, optimization, and reinforcement learning have been proposed to automate such planning tasks.<n>A promising approach is to use the security knowledge embedded in large language models to develop agentic response systems.
  arXiv  Detail & Related papers  (2026-08-03T16:03:16Z)
• Beyond Action Residuals: Real-World Robot Policy Steering via Bottleneck Latent Reinforcement Learning [42.74142065376427]
  We propose Z-Perturbation Reinforcement Learning (ZPRL) to steer pretrained policies through a compact bottleneck latent.<n>In the real world, ZPRL improves the average success rate on four tasks by 33.7% over imitation base policies.
  arXiv  Detail & Related papers  (2026-05-19T14:43:26Z)
• In-Context Autonomous Network Incident Response: An End-to-End Large Language Model Agent Approach [3.733830491643117]
  We propose to leverage large language models' pre-trained security knowledge and in-context learning to create an end-to-end agentic solution for incident response planning.<n>Specifically, our agent integrates four functionalities, perception, reasoning, planning, and action, into one lightweight LLM (14b model)<n>Our agent achieves recovery up to 23% faster than those of frontier LLMs.
  arXiv  Detail & Related papers  (2026-02-13T18:09:30Z)
• rSIM: Incentivizing Reasoning Capabilities of LLMs via Reinforced Strategy Injection [49.74493901036598]
  Large language models (LLMs) are post-trained through reinforcement learning (RL) to evolve into Reasoning Language Models (RLMs)<n>This paper proposes a novel reinforced strategy injection mechanism (rSIM) that enables any LLM to become an RLM by employing a small planner.<n> Experimental results show that rSIM enables Qwen2.5-0.5B to become an RLM and significantly outperform Qwen2.5-14B.
  arXiv  Detail & Related papers  (2025-12-09T06:55:39Z)
• Enhancing Long Chain-of-Thought Reasoning through Multi-Path Plan Aggregation [32.86351316550696]
  We analyze raw long CoTs and uncover a reasoning hierarchy consisting of planning and execution steps.<n>Motivated by this observation, we propose Multi-Path Plan Aggregation (MPPA), a framework that augments single-pass reasoning with plan exploration and aggregation.<n>To overcome this, we introduce online Step-DPO, a process-level preference optimization scheme that leverages Twisted Sequential Monte Carlo (TSMC) to provide scalable stepwise supervision.
  arXiv  Detail & Related papers  (2025-10-13T17:02:41Z)
• ReflecSched: Solving Dynamic Flexible Job-Shop Scheduling via LLM-Powered Hierarchical Reflection [4.101501114944147]
  ReflecSched is a framework that empowers the LLM beyond a direct scheduler.<n>It distills simulations across multiple planning horizons into a concise, natural-language summary.<n>This summary is then integrated into the prompt of a final decision-making module, guiding it to produce non-myopic actions.
  arXiv  Detail & Related papers  (2025-08-03T11:26:35Z)
• Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models? [65.18157595903124]
  This work investigates iterative approximate evaluation for arbitrary prompts.<n>It introduces Model Predictive Prompt Selection (MoPPS), a Bayesian risk-predictive framework.<n>MoPPS reliably predicts prompt difficulty and accelerates training with significantly reduced rollouts.
  arXiv  Detail & Related papers  (2025-07-07T03:20:52Z)
• Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RL [62.984693936073974]
  Large language models (LLMs) excel in tasks like question answering and dialogue.<n>Complex tasks requiring interaction, such as negotiation and persuasion, require additional long-horizon reasoning and planning.<n>We propose a novel approach that uses goal-conditioned value functions to guide the reasoning of LLM agents.
  arXiv  Detail & Related papers  (2025-05-23T16:51:54Z)
• Towards Efficient LLM Grounding for Embodied Multi-Agent Collaboration [68.29746557968107]
  We propose a novel framework for multi-agent collaboration that introduces Reinforced Advantage feedback (ReAd) for efficient self-refinement of plans.<n> Experiments on Over-AI and a difficult variant of RoCoBench show that ReAd surpasses baselines in success rate, and also significantly decreases the interaction steps of agents.
  arXiv  Detail & Related papers  (2024-05-23T08:33:19Z)
• REBEL: Reinforcement Learning via Regressing Relative Rewards [59.68420022466047]
  We propose REBEL, a minimalist RL algorithm for the era of generative models.<n>In theory, we prove that fundamental RL algorithms like Natural Policy Gradient can be seen as variants of REBEL.<n>We find that REBEL provides a unified approach to language modeling and image generation with stronger or similar performance as PPO and DPO.
  arXiv  Detail & Related papers  (2024-04-25T17:20:45Z)
• AdaPlanner: Adaptive Planning from Feedback with Language Models [56.367020818139665]
  Large language models (LLMs) have recently demonstrated the potential in acting as autonomous agents for sequential decision-making tasks. We propose a closed-loop approach, AdaPlanner, which allows the LLM agent to refine its self-generated plan adaptively in response to environmental feedback. To mitigate hallucination, we develop a code-style LLM prompt structure that facilitates plan generation across a variety of tasks, environments, and agent capabilities.
  arXiv  Detail & Related papers  (2023-05-26T05:52:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.