Proximal Policy Distillation

• URL: http://arxiv.org/abs/2407.15134v1
• Date: Sun, 21 Jul 2024 12:08:54 GMT
• Title: Proximal Policy Distillation
• Authors: Giacomo Spigler,
• Abstract summary: We introduce Proximal Policy Distillation (PPD), a novel policy distillation method that integrates student-driven distillation and Proximal Policy Optimization (PPO) We compare PPD with two common alternatives, student-distill and teacher-distill, over a wide range of reinforcement learning environments. Our findings indicate that PPD improves sample efficiency and produces better student policies compared to typical policy distillation approaches.
• Score: 3.2634122554914002
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce Proximal Policy Distillation (PPD), a novel policy distillation method that integrates student-driven distillation and Proximal Policy Optimization (PPO) to increase sample efficiency and to leverage the additional rewards that the student policy collects during distillation. To assess the efficacy of our method, we compare PPD with two common alternatives, student-distill and teacher-distill, over a wide range of reinforcement learning environments that include discrete actions and continuous control (ATARI, Mujoco, and Procgen). For each environment and method, we perform distillation to a set of target student neural networks that are smaller, identical (self-distillation), or larger than the teacher network. Our findings indicate that PPD improves sample efficiency and produces better student policies compared to typical policy distillation approaches. Moreover, PPD demonstrates greater robustness than alternative methods when distilling policies from imperfect demonstrations. The code for the paper is released as part of a new Python library built on top of stable-baselines3 to facilitate policy distillation: `sb3-distill'.

Related papers

• On-Policy Context Distillation for Language Models [92.82835176360864]
  We propose On-Policy Context Distillation (OPCD), a framework that bridges on-policy distillation with context distillation.<n>We demonstrate the effectiveness of OPCD on two important applications: experiential knowledge distillation and system prompt distillation.
  arXiv  Detail & Related papers  (2026-02-12T18:58:28Z)
• Diffusion Policy Policy Optimization [37.04382170999901]
  Diffusion Policy Optimization, DPPO, is an algorithmic framework for fine-tuning diffusion-based policies. DPO achieves the strongest overall performance and efficiency for fine-tuning in common benchmarks. We show that DPPO takes advantage of unique synergies between RL fine-tuning and the diffusion parameterization.
  arXiv  Detail & Related papers  (2024-09-01T02:47:50Z)
• Online Policy Distillation with Decision-Attention [23.807761525617384]
  Policy Distillation (PD) has become an effective method to improve deep reinforcement learning tasks. We study the knowledge transfer between different policies that can learn diverse knowledge from the same environment. We propose Online Policy Distillation (OPD) with Decision-Attention (DA)
  arXiv  Detail & Related papers  (2024-06-08T14:40:53Z)
• Policy Gradient with Active Importance Sampling [55.112959067035916]
  Policy gradient (PG) methods significantly benefit from IS, enabling the effective reuse of previously collected samples. However, IS is employed in RL as a passive tool for re-weighting historical samples. We look for the best behavioral policy from which to collect samples to reduce the policy gradient variance.
  arXiv  Detail & Related papers  (2024-05-09T09:08:09Z)
• Theoretically Guaranteed Policy Improvement Distilled from Model-Based Planning [64.10794426777493]
  Model-based reinforcement learning (RL) has demonstrated remarkable successes on a range of continuous control tasks. Recent practices tend to distill optimized action sequences into an RL policy during the training phase. We develop an approach to distill from model-based planning to the policy.
  arXiv  Detail & Related papers  (2023-07-24T16:52:31Z)
• Policy Dispersion in Non-Markovian Environment [53.05904889617441]
  This paper tries to learn the diverse policies from the history of state-action pairs under a non-Markovian environment. We first adopt a transformer-based method to learn policy embeddings. Then, we stack the policy embeddings to construct a dispersion matrix to induce a set of diverse policies.
  arXiv  Detail & Related papers  (2023-02-28T11:58:39Z)
• Robust Policy Optimization in Deep Reinforcement Learning [16.999444076456268]
  In continuous action domains, parameterized distribution of action distribution allows easy control of exploration. In particular, we propose an algorithm called Robust Policy Optimization (RPO), which leverages a perturbed distribution. We evaluated our methods on various continuous control tasks from DeepMind Control, OpenAI Gym, Pybullet, and IsaacGym.
  arXiv  Detail & Related papers  (2022-12-14T22:43:56Z)
• Unbiased Knowledge Distillation for Recommendation [66.82575287129728]
  Knowledge distillation (KD) has been applied in recommender systems (RS) to reduce inference latency. Traditional solutions first train a full teacher model from the training data, and then transfer its knowledge to supervise the learning of a compact student model. We find such a standard distillation paradigm would incur serious bias issue -- popular items are more heavily recommended after the distillation.
  arXiv  Detail & Related papers  (2022-11-27T05:14:03Z)
• Off-policy Reinforcement Learning with Optimistic Exploration and Distribution Correction [73.77593805292194]
  We train a separate exploration policy to maximize an approximate upper confidence bound of the critics in an off-policy actor-critic framework. To mitigate the off-policy-ness, we adapt the recently introduced DICE framework to learn a distribution correction ratio for off-policy actor-critic training.
  arXiv  Detail & Related papers  (2021-10-22T22:07:51Z)
• Prime-Aware Adaptive Distillation [27.66963552145635]
  Knowledge distillation aims to improve the performance of a student network by mimicing the knowledge from a powerful teacher network. Previous effective hard mining methods are not appropriate for distillation. Prime-Aware Adaptive Distillation (PAD) perceives the prime samples in distillation and then emphasizes their effect adaptively.
  arXiv  Detail & Related papers  (2020-08-04T10:53:12Z)
• Evolutionary Stochastic Policy Distillation [139.54121001226451]
  We propose a new method called Evolutionary Policy Distillation (ESPD) to solve GCRS tasks. ESPD enables a target policy to learn from a series of its variants through the technique of policy distillation (PD) The experiments based on the MuJoCo control suite show the high learning efficiency of the proposed method.
  arXiv  Detail & Related papers  (2020-04-27T16:19:25Z)
• Real-time Policy Distillation in Deep Reinforcement Learning [11.026828277064293]
  Policy distillation is an effective way to transfer control policies from a larger network to a smaller untrained network. Existing approaches are computationally inefficient, resulting in a long distillation time. We propose a new distillation mechanism, called real-time policy distillation, in which training the teacher model and distilling the policy to the student model occur simultaneously.
  arXiv  Detail & Related papers  (2019-12-29T11:10:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.