PriPG-RL: Privileged Planner-Guided Reinforcement Learning for Partially Observable Systems with Anytime-Feasible MPC

• URL: http://arxiv.org/abs/2604.08036v1
• Date: Thu, 09 Apr 2026 09:41:11 GMT
• Title: PriPG-RL: Privileged Planner-Guided Reinforcement Learning for Partially Observable Systems with Anytime-Feasible MPC
• Abstract summary: We formalize a Partially Observable Markov Decision Process (POMDP) in which a planner agent guides a learning agent that observes only a lossy projection of the true state.<n>We propose Planner-to-Policy Soft Actor-Critic (P2P-SAC) to distill the planner agent's privileged knowledge to mitigate partial observability.<n>We validate our approach in simulation using NVIDIA Isaac Lab and successfully deploy it on a real-world Unitree Go2 quadruped navigating complex, obstacle-rich environments.
• Score: 1.4312387415573748
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper addresses the problem of training a reinforcement learning (RL) policy under partial observability by exploiting a privileged, anytime-feasible planner agent available exclusively during training. We formalize this as a Partially Observable Markov Decision Process (POMDP) in which a planner agent with access to an approximate dynamical model and privileged state information guides a learning agent that observes only a lossy projection of the true state. To realize this framework, we introduce an anytime-feasible Model Predictive Control (MPC) algorithm that serves as the planner agent. For the learning agent, we propose Planner-to-Policy Soft Actor-Critic (P2P-SAC), a method that distills the planner agent's privileged knowledge to mitigate partial observability and thereby improve both sample efficiency and final policy performance. We support this framework with rigorous theoretical analysis. Finally, we validate our approach in simulation using NVIDIA Isaac Lab and successfully deploy it on a real-world Unitree Go2 quadruped navigating complex, obstacle-rich environments.

Related papers

• Learning Agent-based Model Predictive Control for Holistic Vehicle Performance [21.34873540426368]
  Agent-based model predictive control (AMPC) has recently been proposed as a distributed scheme that collaborates with all agents to achieve optimal holistic performance.<n>This research proposes a novel practical hybrid control scheme - learning agent-based MPC (LAMPC)<n>LAMPC combines the model-based AMPC approach and data-based learning methods to improve the holistic vehicle performance.
  arXiv  Detail & Related papers  (2026-09-10T17:43:16Z)
• When to Plan: Learning to Select Between Reactive Control and Deliberative Planning [11.723839544878244]
  We introduce an RL method for training a meta-reasoning policy that allocates computation by conditioning on a reactive-policy uncertainty score.<n>We conduct an empirical study on motion planning and navigation environments, showing that this design enables the meta-reasoning policy to learn when the reactive policy provides a good-enough action versus when decision-time planning is needed.
  arXiv  Detail & Related papers  (2026-07-17T18:13:38Z)
• Breaking the Computational Barrier: Provably Efficient Actor-Critic for Low-Rank MDPs [53.412166189410904]
  We use supervised learning as a computational proxy to establish a hierarchy of commonly adopted RL oracles under low-rank Markov Decision Processes (MDPs)<n>Motivated by this observation, we propose a novel optimistic actor-critic algorithm that relies solely on the policy evaluation oracle.<n>We prove that our algorithm outperforms the existing sample complexity guarantees for low-rank MDPs while avoiding computationally expensive planning or optimization oracles commonly assumed in prior works.
  arXiv  Detail & Related papers  (2026-05-02T04:46:54Z)
• Curriculum Design for Trajectory-Constrained Agent: Compressing Chain-of-Thought Tokens in LLMs [26.165537937650413]
  Training agents to operate under strict constraints during deployment presents significant challenges.<n>We propose a curriculum learning strategy that gradually tightens constraints during training, enabling the agent to incrementally master the deployment requirements.
  arXiv  Detail & Related papers  (2025-11-04T16:14:56Z)
• Learning When to Plan: Efficiently Allocating Test-Time Compute for LLM Agents [35.79575378215309]
  Training large language models (LLMs) to reason via reinforcement learning (RL) significantly improves their problem-solving capabilities.<n>We introduce a conceptual framework formalizing dynamic planning for LLM agents, enabling them to flexibly decide when to allocate test-time compute for planning.<n>Experiments on the Crafter environment show that dynamic planning agents trained with this approach are more sample-efficient and consistently achieve more complex objectives.
  arXiv  Detail & Related papers  (2025-09-03T18:00:13Z)
• Smart Exploration in Reinforcement Learning using Bounded Uncertainty Models [0.8602553195689513]
  Reinforcement learning (RL) is a powerful framework for decision-making in uncertain environments.<n>We address this challenge by incorporating prior model knowledge to guide exploration and accelerate the learning process.<n>We demonstrate the effectiveness of the proposed exploration strategy, which we call BUMEX, in a simulation study.
  arXiv  Detail & Related papers  (2025-04-08T12:33:38Z)
• REBEL: Reinforcement Learning via Regressing Relative Rewards [59.68420022466047]
  We propose REBEL, a minimalist RL algorithm for the era of generative models.<n>In theory, we prove that fundamental RL algorithms like Natural Policy Gradient can be seen as variants of REBEL.<n>We find that REBEL provides a unified approach to language modeling and image generation with stronger or similar performance as PPO and DPO.
  arXiv  Detail & Related papers  (2024-04-25T17:20:45Z)
• Theoretically Guaranteed Policy Improvement Distilled from Model-Based Planning [64.10794426777493]
  Model-based reinforcement learning (RL) has demonstrated remarkable successes on a range of continuous control tasks. Recent practices tend to distill optimized action sequences into an RL policy during the training phase. We develop an approach to distill from model-based planning to the policy.
  arXiv  Detail & Related papers  (2023-07-24T16:52:31Z)
• Provably Efficient Representation Learning with Tractable Planning in Low-Rank POMDP [81.00800920928621]
  We study representation learning in partially observable Markov Decision Processes (POMDPs) We first present an algorithm for decodable POMDPs that combines maximum likelihood estimation (MLE) and optimism in the face of uncertainty (OFU) We then show how to adapt this algorithm to also work in the broader class of $gamma$-observable POMDPs.
  arXiv  Detail & Related papers  (2023-06-21T16:04:03Z)
• Jump-Start Reinforcement Learning [68.82380421479675]
  We present a meta algorithm that can use offline data, demonstrations, or a pre-existing policy to initialize an RL policy. In particular, we propose Jump-Start Reinforcement Learning (JSRL), an algorithm that employs two policies to solve tasks. We show via experiments that JSRL is able to significantly outperform existing imitation and reinforcement learning algorithms.
  arXiv  Detail & Related papers  (2022-04-05T17:25:22Z)
• Learning Off-Policy with Online Planning [18.63424441772675]
  We investigate a novel instantiation of H-step lookahead with a learned model and a terminal value function. We show the flexibility of LOOP to incorporate safety constraints during deployment with a set of navigation environments.
  arXiv  Detail & Related papers  (2020-08-23T16:18:44Z)
• Model-based Reinforcement Learning for Decentralized Multiagent Rendezvous [66.6895109554163]
  Underlying the human ability to align goals with other agents is their ability to predict the intentions of others and actively update their own plans. We propose hierarchical predictive planning (HPP), a model-based reinforcement learning method for decentralized multiagent rendezvous.
  arXiv  Detail & Related papers  (2020-03-15T19:49:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.