Absolute Policy Optimization

• URL: http://arxiv.org/abs/2310.13230v5
• Date: Thu, 30 May 2024 17:13:04 GMT
• Title: Absolute Policy Optimization
• Authors: Weiye Zhao, Feihan Li, Yifan Sun, Rui Chen, Tianhao Wei, Changliu Liu,
• Abstract summary: We introduce a novel objective function, optimizing which leads to guaranteed monotonic improvement in the lower probability bound of performance with high confidence. Our experiments demonstrate the effectiveness of our approach across challenging continuous control benchmark tasks and extend its applicability to mastering Atari games.
• Score: 10.503953488401926
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In recent years, trust region on-policy reinforcement learning has achieved impressive results in addressing complex control tasks and gaming scenarios. However, contemporary state-of-the-art algorithms within this category primarily emphasize improvement in expected performance, lacking the ability to control over the worst-case performance outcomes. To address this limitation, we introduce a novel objective function, optimizing which leads to guaranteed monotonic improvement in the lower probability bound of performance with high confidence. Building upon this groundbreaking theoretical advancement, we further introduce a practical solution called Absolute Policy Optimization (APO). Our experiments demonstrate the effectiveness of our approach across challenging continuous control benchmark tasks and extend its applicability to mastering Atari games. Our findings reveal that APO as well as its efficient variation Proximal Absolute Policy Optimization (PAPO) significantly outperforms state-of-the-art policy gradient algorithms, resulting in substantial improvements in worst-case performance, as well as expected performance.

Related papers

• Action Robust Reinforcement Learning via Optimal Adversary Aware Policy Optimization [7.680706215758518]
  Reinforcement Learning (RL) has achieved remarkable success in sequential decision tasks.<n>Recent studies have revealed the vulnerability of RL policies to different perturbations.<n>We introduce a novel framework called Optimal Adversary-aware Policy Iteration (OA-PI)
  arXiv  Detail & Related papers  (2025-07-04T08:11:15Z)
• Exterior Penalty Policy Optimization with Penalty Metric Network under Constraints [52.37099916582462]
  In Constrained Reinforcement Learning (CRL), agents explore the environment to learn the optimal policy while satisfying constraints. We propose a theoretically guaranteed penalty function method, Exterior Penalty Policy Optimization (EPO), with adaptive penalties generated by a Penalty Metric Network (PMN) PMN responds appropriately to varying degrees of constraint violations, enabling efficient constraint satisfaction and safe exploration.
  arXiv  Detail & Related papers  (2024-07-22T10:57:32Z)
• Reflective Policy Optimization [20.228281670899204]
  Reflective Policy Optimization (RPO) amalgamates past and future state-action information for policy optimization. RPO empowers the agent for introspection, allowing modifications to its actions within the current state. Empirical results demonstrate RPO's feasibility and efficacy in two reinforcement learning benchmarks.
  arXiv  Detail & Related papers  (2024-06-06T01:46:49Z)
• Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimation [46.61909578101735]
  Adversarial Policy Optimization (AdvPO) is a novel solution to the pervasive issue of reward over-optimization in Reinforcement Learning from Human Feedback. In this paper, we introduce a lightweight way to quantify uncertainties in rewards, relying solely on the last layer embeddings of the reward model.
  arXiv  Detail & Related papers  (2024-03-08T09:20:12Z)
• Dropout Strategy in Reinforcement Learning: Limiting the Surrogate Objective Variance in Policy Optimization Methods [0.0]
  Policy-based reinforcement learning algorithms are widely used in various fields. These algorithms introduce importance sampling into policy iteration. This can lead to a high variance of the surrogate objective and indirectly affects the stability and convergence of the algorithm.
  arXiv  Detail & Related papers  (2023-10-31T11:38:26Z)
• Mimicking Better by Matching the Approximate Action Distribution [48.95048003354255]
  We introduce MAAD, a novel, sample-efficient on-policy algorithm for Imitation Learning from Observations. We show that it requires considerable fewer interactions to achieve expert performance, outperforming current state-of-the-art on-policy methods.
  arXiv  Detail & Related papers  (2023-06-16T12:43:47Z)
• Adversarial Policy Optimization in Deep Reinforcement Learning [16.999444076456268]
  The policy represented by the deep neural network can overfitting, which hamper a reinforcement learning agent from learning effective policy. Data augmentation can provide a performance boost to RL agents by mitigating the effect of overfitting. We propose a novel RL algorithm to mitigate the above issue and improve the efficiency of the learned policy.
  arXiv  Detail & Related papers  (2023-04-27T21:01:08Z)
• Off-policy Reinforcement Learning with Optimistic Exploration and Distribution Correction [73.77593805292194]
  We train a separate exploration policy to maximize an approximate upper confidence bound of the critics in an off-policy actor-critic framework. To mitigate the off-policy-ness, we adapt the recently introduced DICE framework to learn a distribution correction ratio for off-policy actor-critic training.
  arXiv  Detail & Related papers  (2021-10-22T22:07:51Z)
• Offline RL Without Off-Policy Evaluation [49.11859771578969]
  We show that simply doing one step of constrained/regularized policy improvement using an on-policy Q estimate of the behavior policy performs surprisingly well. This one-step algorithm beats the previously reported results of iterative algorithms on a large portion of the D4RL benchmark.
  arXiv  Detail & Related papers  (2021-06-16T16:04:26Z)
• Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO [90.90009491366273]
  We study the roots of algorithmic progress in deep policy gradient algorithms through a case study on two popular algorithms. Specifically, we investigate the consequences of "code-level optimizations:" Our results show that they (a) are responsible for most of PPO's gain in cumulative reward over TRPO, and (b) fundamentally change how RL methods function.
  arXiv  Detail & Related papers  (2020-05-25T16:24:59Z)
• Stable Policy Optimization via Off-Policy Divergence Regularization [50.98542111236381]
  Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) are among the most successful policy gradient approaches in deep reinforcement learning (RL) We propose a new algorithm which stabilizes the policy improvement through a proximity term that constrains the discounted state-action visitation distribution induced by consecutive policies to be close to one another. Our proposed method can have a beneficial effect on stability and improve final performance in benchmark high-dimensional control tasks.
  arXiv  Detail & Related papers  (2020-03-09T13:05:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.