Guided Policy Optimization under Partial Observability

• URL: http://arxiv.org/abs/2505.15418v1
• Date: Wed, 21 May 2025 12:01:08 GMT
• Title: Guided Policy Optimization under Partial Observability
• Authors: Yueheng Li, Guangming Xie, Zongqing Lu,
• Abstract summary: Reinforcement Learning (RL) in partially observable environments poses significant challenges due to the complexity of learning under uncertainty.<n>We introduce Guided Policy Optimization (GPO), a framework that co-trains a guider and a learner.<n>We theoretically demonstrate that this learning scheme achieves optimality comparable to direct RL, thereby overcoming key limitations inherent in existing approaches.
• Score: 36.853129816484845
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement Learning (RL) in partially observable environments poses significant challenges due to the complexity of learning under uncertainty. While additional information, such as that available in simulations, can enhance training, effectively leveraging it remains an open problem. To address this, we introduce Guided Policy Optimization (GPO), a framework that co-trains a guider and a learner. The guider takes advantage of privileged information while ensuring alignment with the learner's policy that is primarily trained via imitation learning. We theoretically demonstrate that this learning scheme achieves optimality comparable to direct RL, thereby overcoming key limitations inherent in existing approaches. Empirical evaluations show strong performance of GPO across various tasks, including continuous control with partial observability and noise, and memory-based challenges, significantly outperforming existing methods.

Related papers

• PIGDreamer: Privileged Information Guided World Models for Safe Partially Observable Reinforcement Learning [23.384621982394673]
  We propose a model-based safe reinforcement learning approach that leverages privileged information to enhance the agent's safety and performance.<n>Our empirical results demonstrate that our approach significantly outperforms existing methods in terms of safety and task-centric performance.
  arXiv  Detail & Related papers  (2025-08-04T08:01:19Z)
• Observations Meet Actions: Learning Control-Sufficient Representations for Robust Policy Generalization [6.408943565801689]
  Capturing latent variations ("contexts") is key to deploying reinforcement-learning (RL) agents beyond their training regime.<n>We recast context-based RL as a dual inference-control problem and formally characterize two properties and their hierarchy.<n>We derive a contextual evidence lower bound(ELBO)-style objective that cleanly separates representation learning from policy learning.
  arXiv  Detail & Related papers  (2025-07-25T17:08:16Z)
• Improving LLM Reasoning for Vulnerability Detection via Group Relative Policy Optimization [45.799380822683034]
  We present an extensive study aimed at advancing RL-based finetuning techniques for Large Language Models (LLMs)<n>We highlight key limitations of commonly adopted LLMs, such as their tendency to over-predict certain types of vulnerabilities while failing to detect others.<n>To address this challenge, we explore the use of Group Relative Policy Optimization (GRPO), a recent policy-gradient method, for guiding LLM behavior through structured, rule-based rewards.
  arXiv  Detail & Related papers  (2025-07-03T11:52:45Z)
• Diffusion Guidance Is a Controllable Policy Improvement Operator [98.11511661904618]
  CFGRL is trained with the simplicity of supervised learning, yet can further improve on the policies in the data.<n>On offline RL tasks, we observe a reliable trend -- increased guidance weighting leads to increased performance.
  arXiv  Detail & Related papers  (2025-05-29T14:06:50Z)
• Do We Need to Verify Step by Step? Rethinking Process Supervision from a Theoretical Perspective [59.61868506896214]
  We show that under standard data coverage assumptions, reinforcement learning is no more statistically difficult than through process supervision.<n>We prove that any policy's advantage function can serve as an optimal process reward model.
  arXiv  Detail & Related papers  (2025-02-14T22:21:56Z)
• Feasibility Consistent Representation Learning for Safe Reinforcement Learning [25.258227763316228]
  We introduce a novel framework named Feasibility Consistent Safe Reinforcement Learning (FCSRL) This framework combines representation learning with feasibility-oriented objectives to identify and extract safety-related information from the raw state for safe RL. Our method is capable of learning a better safety-aware embedding and achieving superior performance than previous representation learning baselines.
  arXiv  Detail & Related papers  (2024-05-20T01:37:21Z)
• Blending Imitation and Reinforcement Learning for Robust Policy Improvement [16.588397203235296]
  Imitation learning (IL) utilizes oracles to improve sample efficiency. RPI draws on the strengths of IL, using oracle queries to facilitate exploration. RPI is capable of learning from and improving upon a diverse set of black-box oracles.
  arXiv  Detail & Related papers  (2023-10-03T01:55:54Z)
• Variance-Covariance Regularization Improves Representation Learning [28.341622247252705]
  We adapt a self-supervised learning regularization technique to supervised learning contexts, introducing Variance-Covariance Regularization (VCReg) We demonstrate that VCReg significantly enhances transfer learning for images and videos, achieving state-of-the-art performance across numerous tasks and datasets. In summary, VCReg offers a universally applicable regularization framework that significantly advances transfer learning and highlights the connection between gradient starvation, neural collapse, and feature transferability.
  arXiv  Detail & Related papers  (2023-06-23T05:01:02Z)
• Reinforcement Learning with Stepwise Fairness Constraints [50.538878453547966]
  We introduce the study of reinforcement learning with stepwise fairness constraints. We provide learning algorithms with strong theoretical guarantees in regard to policy optimality and fairness violation.
  arXiv  Detail & Related papers  (2022-11-08T04:06:23Z)
• Jump-Start Reinforcement Learning [68.82380421479675]
  We present a meta algorithm that can use offline data, demonstrations, or a pre-existing policy to initialize an RL policy. In particular, we propose Jump-Start Reinforcement Learning (JSRL), an algorithm that employs two policies to solve tasks. We show via experiments that JSRL is able to significantly outperform existing imitation and reinforcement learning algorithms.
  arXiv  Detail & Related papers  (2022-04-05T17:25:22Z)
• Reinforcement Learning using Guided Observability [26.307025803058714]
  We propose a simple but efficient approach to make reinforcement learning cope with partial observability. Our main insight is that smoothly transitioning from full observability to partial observability during the training process yields a high performance policy. A comprehensive evaluation in discrete partially observableMarkov decision process (POMDP) benchmark problems and continuous partially observable MuJoCo and OpenAI gym tasks shows that PO-GRL improves performance.
  arXiv  Detail & Related papers  (2021-04-22T10:47:35Z)
• Off-Policy Imitation Learning from Observations [78.30794935265425]
  Learning from Observations (LfO) is a practical reinforcement learning scenario from which many applications can benefit. We propose a sample-efficient LfO approach that enables off-policy optimization in a principled manner. Our approach is comparable with state-of-the-art locomotion in terms of both sample-efficiency and performance.
  arXiv  Detail & Related papers  (2021-02-25T21:33:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.