ProxPI: Proximal Prior Injection for Sampling-Based MPC under Learned-Prior Mismatch

• URL: http://arxiv.org/abs/2609.00941v1
• Date: Tue, 01 Sep 2026 09:03:25 GMT
• Title: ProxPI: Proximal Prior Injection for Sampling-Based MPC under Learned-Prior Mismatch
• Abstract summary: In model predictive path integral control, a policy-centered warm-start approach centers the sampling distribution on the policy output.<n>We propose Proximal Prior Injection (ProxPI), which retains nominal-centered MPPI sampling and incorporates the policy through a soft proximity cost.<n>We theoretically show that re-centering on the prior discards's correction at every update, whereas nominal-centered sampling retains it and converges to a solution set by both the task cost and the prior.
• Score: 3.345437353879255
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Combining learned policies with model predictive control can leverage learned task priors while retaining online adaptation to new objectives and constraints, but performance degrades when the policy is out of distribution. In policy-guided model predictive path integral (MPPI) control, a policy-centered warm-start approach centers the sampling distribution on the policy output. When the prior is mismatched, centering the sampling distribution on the policy output restricts exploration around an unsuitable solution and prevents recovery toward the task optimum. We propose Proximal Prior Injection (ProxPI), which retains nominal-centered MPPI sampling and incorporates the policy through a soft proximity cost. This matches the in-distribution performance of existing prior-injection schemes while enabling the optimizer to escape an inaccurate policy and recover vanilla MPPI-level performance. We theoretically show that re-centering on the prior discards the optimizer's correction at every update, whereas nominal-centered sampling retains it and converges to a solution set by both the task cost and the prior, and that this failure is not removed by a larger rollout budget. Simulations and real-robot experiments demonstrate robust performance under both in-distribution and out-of-distribution tasks.

Related papers

• Ratio-Variance Regularized Policy Optimization [64.95520246570446]
  We show that explicitly constraining the policy ratio variance provides a principled local approximation to trust-region constraints.<n>We introduce $bf R2bf VPO$ (Ratio-Variance Regularized Policy Optimization), which implements this constraint via a primal-dual optimization framework.
  arXiv  Detail & Related papers  (2026-05-26T09:53:42Z)
• Stochastic MeanFlow Policies: One-Step Generative Control with Entropic Mirror Descent [53.828537014796574]
  Online off-policy reinforcement learning (RL) is shaped by two choices: the policy class and the update rule.<n>We propose MeanFlow Policies, a one-step generative policy class that maps noise to actions through a MeanFlow transformation.<n>Across seven MuJoCo benchmarks, S improves over Gaussian and generative baselines while retaining single-step inference efficiency.
  arXiv  Detail & Related papers  (2026-05-20T15:14:14Z)
• OGPO: Sample Efficient Full-Finetuning of Generative Control Policies [53.42266064673132]
  Generative control policies (GCPs) have emerged as effective parameterizations for robot learning.<n>This work introduces Off-policy Generative Policy Optimization (OGPO), a sample-efficient for finetuning GCPs.<n>OGPO achieves state-of-the-art performance on manipulation tasks spanning multi-task settings, high-precision insertion, and dexterous control.
  arXiv  Detail & Related papers  (2026-05-04T18:36:40Z)
• Robust Regularized Policy Iteration under Transition Uncertainty [6.7431287237221085]
  We formulate offline RL as robust policy optimization, treating the transition kernel as a decision variable within an uncertainty set.<n>We propose Robust Regularized Policy Iteration (RRPI), which replaces the intractable max-min bilevel objective with a tractable KL-regularized surrogate.<n>Experiments on D4RL benchmarks demonstrate that RRPI achieves strong average performance, outperforming recent baselines.
  arXiv  Detail & Related papers  (2026-03-10T08:18:27Z)
• Unbiased Dynamic Pruning for Efficient Group-Based Policy Optimization [60.87651283510059]
  Group Relative Policy Optimization (GRPO) effectively scales LLM reasoning but incurs prohibitive computational costs.<n>We propose Dynamic Pruning Policy Optimization (DPPO), a framework that enables dynamic pruning while preserving unbiased gradient estimation.<n>To mitigate the data sparsity induced by pruning, we introduce Dense Prompt Packing, a window-based greedy strategy.
  arXiv  Detail & Related papers  (2026-03-04T14:48:53Z)
• SEMDICE: Off-policy State Entropy Maximization via Stationary Distribution Correction Estimation [54.537828696303286]
  In unsupervised-training for reinforcement learning, the agent aims to learn a prior policy for downstream tasks without relying on task-specific reward functions.<n>We focus on state entropy (SEM), where the goal is to learn a policy that maximizes the entropy of the state stationary distribution.<n>We introduce SEMDICE, a principled off-policy algorithm that computes an SEM policy from an arbitrary off-policy dataset.
  arXiv  Detail & Related papers  (2025-12-10T19:50:21Z)
• OptPO: Optimal Rollout Allocation for Test-time Policy Optimization [11.375209834858135]
  Test-time policy optimization enables large language models to adapt to distribution shifts by leveraging feedback from self-generated rollouts.<n>We propose Optimal Rollout Allocation for Test-time Policy Optimization (OptPO), a principled framework that adaptively allocates inference budgets.
  arXiv  Detail & Related papers  (2025-12-02T15:38:52Z)
• A KL-regularization framework for learning to plan with adaptive priors [1.0246259631050245]
  We introduce Policy Optimization-Model Predictive Control (PO-MPC)<n>PO-MPC integrates the planner's action distribution as a prior in policy optimization.<n>Our experiments show that these extended configurations yield significant performance improvements.
  arXiv  Detail & Related papers  (2025-10-05T16:45:38Z)
• Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer [52.09480867526656]
  We identify the source of misalignment as a form of distributional shift and uncertainty in learning human preferences.<n>To mitigate overoptimization, we first propose a theoretical algorithm that chooses the best policy for an adversarially chosen reward model.<n>Using the equivalence between reward models and the corresponding optimal policy, the algorithm features a simple objective that combines a preference optimization loss and a supervised learning loss.
  arXiv  Detail & Related papers  (2024-05-26T05:38:50Z)
• Importance-Weighted Offline Learning Done Right [16.4989952150404]
  We study the problem of offline policy optimization in contextual bandit problems. The goal is to learn a near-optimal policy based on a dataset of decision data collected by a suboptimal behavior policy. We show that a simple alternative approach based on the "implicit exploration" estimator of citet2015 yields performance guarantees that are superior in nearly all possible terms to all previous results.
  arXiv  Detail & Related papers  (2023-09-27T16:42:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.