Rao-Blackwellized POMDP Planning

• URL: http://arxiv.org/abs/2409.16392v2
• Date: Mon, 03 Mar 2025 20:56:33 GMT
• Title: Rao-Blackwellized POMDP Planning
• Abstract summary: This study introduces Rao-Blackwellized POMDP approximate solvers and outlines generic methods to apply Rao-Blackwellization in both belief updates and online planning.<n>We compare the performance of SIRPF and RBPF in a simulated localization problem where an agent navigates toward a target in a GPS-denied environment using POMCPOW and RB-POMCPOW planners.<n>Our results not only confirm that RBPFs maintain accurate belief approximations over time with fewer particles, but, more surprisingly, RBPFs combined with quadrature-based integration improve planning quality significantly compared to SIRPF-based planning under
• Score: 10.762101459838048
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Partially Observable Markov Decision Processes (POMDPs) provide a structured framework for decision-making under uncertainty, but their application requires efficient belief updates. Sequential Importance Resampling Particle Filters (SIRPF), also known as Bootstrap Particle Filters, are commonly used as belief updaters in large approximate POMDP solvers, but they face challenges such as particle deprivation and high computational costs as the system's state dimension grows. To address these issues, this study introduces Rao-Blackwellized POMDP (RB-POMDP) approximate solvers and outlines generic methods to apply Rao-Blackwellization in both belief updates and online planning. We compare the performance of SIRPF and Rao-Blackwellized Particle Filters (RBPF) in a simulated localization problem where an agent navigates toward a target in a GPS-denied environment using POMCPOW and RB-POMCPOW planners. Our results not only confirm that RBPFs maintain accurate belief approximations over time with fewer particles, but, more surprisingly, RBPFs combined with quadrature-based integration improve planning quality significantly compared to SIRPF-based planning under the same computational limits.

Related papers

• Scalable Rao-Blackwellized Online Planning for High-Dimensional POMDPs [10.757434965980124]
  We extend the Rao-Blackwellized online POMDP (RB-POMDP) framework to improve its generalizability in high-dimensional settings through hybrid continuous-discrete belief representations.<n>We demonstrate the effectiveness of this framework in a robotic search-and-rescue task by integrating it with FastSLAM 2.0.
  arXiv  Detail & Related papers  (2026-09-01T14:57:10Z)
• VOiLA: Vectorized Online Planning with Learned Diffusion Model for POMDP Agents [9.270170611697141]
  Planning under uncertainty is an essential capability for autonomous robots.<n>We present a framework that learns task-agnostic POMDP models for online planning under uncertainty.
  arXiv  Detail & Related papers  (2026-06-18T02:51:54Z)
• Rethinking the Divergence Regularization in LLM RL [56.952725733148746]
  Reinforcement learning (RL) has become a key component of post-training large language models (LLMs)<n>We propose Divergence Regularized Policy Optimization (DRPO), which replaces the hard mask with a smooth advantage-weighted quadratic regularizer on policy shift.<n>DRPO preserves the same trust-region geometry as DPPO while inducing bounded, continuous gradient weights.
  arXiv  Detail & Related papers  (2026-06-08T17:58:23Z)
• Unbiased Dynamic Pruning for Efficient Group-Based Policy Optimization [60.87651283510059]
  Group Relative Policy Optimization (GRPO) effectively scales LLM reasoning but incurs prohibitive computational costs.<n>We propose Dynamic Pruning Policy Optimization (DPPO), a framework that enables dynamic pruning while preserving unbiased gradient estimation.<n>To mitigate the data sparsity induced by pruning, we introduce Dense Prompt Packing, a window-based greedy strategy.
  arXiv  Detail & Related papers  (2026-03-04T14:48:53Z)
• POMDPPlanners: Open-Source Package for POMDP Planning [9.269394037577177]
  POMDPPlanners is an open-source Python package for empirical evaluation of Partially Observable Markov Decision Process (POMDP) planning algorithms.<n>POMDPPlanners is designed to enable scalable, reproducible research on decision-making under uncertainty.
  arXiv  Detail & Related papers  (2026-02-24T11:50:04Z)
• Mitigating Premature Exploitation in Particle-based Monte Carlo for Inference-Time Scaling [15.828750560145751]
  Inference-Time Scaling (ITS) improves language models by allocating more computation at generation time.<n>PF has emerged as a strong ITS method for complex mathematical reasoning tasks.<n>It is vulnerable when guided by process reward models, which often assign overconfident scores early in the reasoning process.<n>This failure mode, known as particle impoverishment, is especially severe under constrained computational budgets.<n>We introduce Entropic Particle Filtering (ePF), an algorithm that integrates two new techniques to solve these issues.
  arXiv  Detail & Related papers  (2025-10-07T11:48:32Z)
• Rule-Based Modeling of Low-Dimensional Data with PCA and Binary Particle Swarm Optimization (BPSO) in ANFIS [0.29465623430708915]
  Fuzzy rule-based systems interpret data in low-dimensional domains, providing transparency and interpretability. Deep learning excels in complex tasks but is prone to overfitting in sparse, unstructured, or low-dimensional data. This interpretability is crucial in fields like healthcare and finance.
  arXiv  Detail & Related papers  (2025-02-06T09:13:55Z)
• Beyond the Boundaries of Proximal Policy Optimization [17.577317574595206]
  This work offers an alternative perspective of PPO, in which it is decomposed into the inner-loop estimation of update vectors. We propose outer proximal policy optimization (outer-PPO); a framework wherein these update vectors are applied using an arbitrary gradient-based gradient. Methods are evaluated against an aggressively tuned PPO baseline on Brax, Jumanji and MinAtar environments.
  arXiv  Detail & Related papers  (2024-11-01T15:29:10Z)
• DyPNIPP: Predicting Environment Dynamics for RL-based Robust Informative Path Planning [13.462524685985818]
  DyPNIPP is a robust RL-based IPP framework designed to effectively acrosstemporal environments. Our experiments in a wildfire environment demonstrate that DyPNIPP outperforms existing RL-based IPP algorithms.
  arXiv  Detail & Related papers  (2024-10-22T17:07:26Z)
• Efficient Learning of POMDPs with Known Observation Model in Average-Reward Setting [56.92178753201331]
  We propose the Observation-Aware Spectral (OAS) estimation technique, which enables the POMDP parameters to be learned from samples collected using a belief-based policy. We show the consistency of the OAS procedure, and we prove a regret guarantee of order $mathcalO(sqrtT log(T)$ for the proposed OAS-UCRL algorithm.
  arXiv  Detail & Related papers  (2024-10-02T08:46:34Z)
• Learning Logic Specifications for Policy Guidance in POMDPs: an Inductive Logic Programming Approach [57.788675205519986]
  We learn high-quality traces from POMDP executions generated by any solver. We exploit data- and time-efficient Indu Logic Programming (ILP) to generate interpretable belief-based policy specifications. We show that learneds expressed in Answer Set Programming (ASP) yield performance superior to neural networks and similar to optimal handcrafted task-specifics within lower computational time.
  arXiv  Detail & Related papers  (2024-02-29T15:36:01Z)
• Inferring Dynamic Networks from Marginals with Iterative Proportional Fitting [57.487936697747024]
  A common network inference problem, arising from real-world data constraints, is how to infer a dynamic network from its time-aggregated adjacency matrix. We introduce a principled algorithm that guarantees IPF converges under minimal changes to the network structure.
  arXiv  Detail & Related papers  (2024-02-28T20:24:56Z)
• Optimality Guarantees for Particle Belief Approximation of POMDPs [55.83001584645448]
  Partially observable Markov decision processes (POMDPs) provide a flexible representation for real-world decision and control problems. POMDPs are notoriously difficult to solve, especially when the state and observation spaces are continuous or hybrid. We propose a theory characterizing the approximation error of the particle filtering techniques that these algorithms use.
  arXiv  Detail & Related papers  (2022-10-10T21:11:55Z)
• Proximal Policy Optimization with Adaptive Threshold for Symmetric Relative Density Ratio [8.071506311915396]
  A popular method, so-called policy optimization (PPO), and its variants constrain density ratio of the latest and baseline policies when the density ratio exceeds a given threshold. This paper proposes a new PPO derived using relative Pearson (RPE) divergence, therefore so-called PPO-RPE, to design the threshold adaptively.
  arXiv  Detail & Related papers  (2022-03-18T09:13:13Z)
• Rule-based Shielding for Partially Observable Monte-Carlo Planning [78.05638156687343]
  We propose two contributions to Partially Observable Monte-Carlo Planning (POMCP) The first is a method for identifying unexpected actions selected by POMCP with respect to expert prior knowledge of the task. The second is a shielding approach that prevents POMCP from selecting unexpected actions. We evaluate our approach on Tiger, a standard benchmark for POMDPs, and a real-world problem related to velocity regulation in mobile robot navigation.
  arXiv  Detail & Related papers  (2021-04-28T14:23:38Z)
• Modular Deep Reinforcement Learning for Continuous Motion Planning with Temporal Logic [59.94347858883343]
  This paper investigates the motion planning of autonomous dynamical systems modeled by Markov decision processes (MDP) The novelty is to design an embedded product MDP (EP-MDP) between the LDGBA and the MDP. The proposed LDGBA-based reward shaping and discounting schemes for the model-free reinforcement learning (RL) only depend on the EP-MDP states.
  arXiv  Detail & Related papers  (2021-02-24T01:11:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.