Neural-to-Tree Policy Distillation with Policy Improvement Criterion

• URL: http://arxiv.org/abs/2108.06898v1
• Date: Mon, 16 Aug 2021 05:14:54 GMT
• Title: Neural-to-Tree Policy Distillation with Policy Improvement Criterion
• Abstract summary: A feasible way to gain insight into a black-box model is to distill it into an interpretable model such as a decision tree. A typical policy distillation that clones model behaviors with even a small error could bring a data distribution shift. We propose to address this issue by changing the distillation objective from behavior cloning to maximizing an advantage evaluation.
• Score: 28.262400646275793
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While deep reinforcement learning has achieved promising results in challenging decision-making tasks, the main bones of its success --- deep neural networks are mostly black-boxes. A feasible way to gain insight into a black-box model is to distill it into an interpretable model such as a decision tree, which consists of if-then rules and is easy to grasp and be verified. However, the traditional model distillation is usually a supervised learning task under a stationary data distribution assumption, which is violated in reinforcement learning. Therefore, a typical policy distillation that clones model behaviors with even a small error could bring a data distribution shift, resulting in an unsatisfied distilled policy model with low fidelity or low performance. In this paper, we propose to address this issue by changing the distillation objective from behavior cloning to maximizing an advantage evaluation. The novel distillation objective maximizes an approximated cumulative reward and focuses more on disastrous behaviors in critical states, which controls the data shift effect. We evaluate our method on several Gym tasks, a commercial fight game, and a self-driving car simulator. The empirical results show that the proposed method can preserve a higher cumulative reward than behavior cloning and learn a more consistent policy to the original one. Moreover, by examining the extracted rules from the distilled decision trees, we demonstrate that the proposed method delivers reasonable and robust decisions.

Related papers

• The Role of Causality in Algorithmic Recourse [10.770213929153371]
  Algorithmic recourse aims to provide individuals with actionable changes to improve their predicted outcomes in high-stakes classification settings.<n>Most existing approaches focus only on a model's prediction, without accounting for whether the changes lead to genuine improvement in an individual's true qualifications.<n>We formalize this failure mode through a causal performative framework for recourse.
  arXiv  Detail & Related papers  (2026-07-30T16:42:51Z)
• Critic-Driven Voronoi-Quantization for Distilling Deep RL Policies to Explainable Models [5.417332705560665]
  We introduce a new model-agnostic method called Critic-Driven Voronoi State Partitioning.<n>By exploiting the critic value network of the original policy, we introduce new subpolicies in regions with insufficient value.<n>We validate our approach on several well known benchmarks and proof that this distillation approaches the original policy using a reasonable sized set of linear functions.
  arXiv  Detail & Related papers  (2026-05-14T14:38:56Z)
• A Pontryagin Method of Model-based Reinforcement Learning via Hamiltonian Actor-Critic [16.49862942485022]
  Hamiltonian Actor-Critic (HAC) is a model-based approach that eliminates explicit value function learning.<n>By avoiding value approximation, HAC reduces sensitivity to model errors while admitting convergence guarantees.<n>In offline settings with limited data, HAC matches or exceeds state-of-the-art methods, highlighting its strong sample efficiency.
  arXiv  Detail & Related papers  (2026-03-30T20:14:19Z)
• Attribution-Guided Model Rectification of Unreliable Neural Network Behaviors [60.06461883533697]
  We leverage rank-one model editing to establish an attribution-guided model rectification framework.<n>We first distinguish our rectification setting from existing model editing, yielding a formulation that corrects unreliable behavior.<n>To target the primary source of misbehavior, we introduce an attribution-guided layer localization method.
  arXiv  Detail & Related papers  (2026-03-08T01:06:07Z)
• Explainable RL Policies by Distilling to Locally-Specialized Linear Policies with Voronoi State Partitioning [5.417332705560665]
  We propose a new model-agnostic method to divide the state space into regions where a simplified, human-understandable model can operate in.<n>In this paper, we use Voronoi partitioning to find regions where linear models can achieve similar performance to the original con- troller.
  arXiv  Detail & Related papers  (2025-11-17T12:58:38Z)
• Curiosity-Diffuser: Curiosity Guide Diffusion Models for Reliability [9.389039109130337]
  imitation policy based on neural network may generate hallucinations, leading to inaccurate behaviors that impact the safety of real-world applications. This paper proposes the Curiosity-Diffuser, aimed at guiding the conditional diffusion model to generate trajectories with lower curiosity. Curiosity-Diffuser significantly improves task performance and produces behaviors that are more similar to the training data.
  arXiv  Detail & Related papers  (2025-03-19T02:25:36Z)
• Distilling Reinforcement Learning Policies for Interpretable Robot Locomotion: Gradient Boosting Machines and Symbolic Regression [53.33734159983431]
  This paper introduces a novel approach to distill neural RL policies into more interpretable forms. We train expert neural network policies using RL and distill them into (i) GBMs, (ii) EBMs, and (iii) symbolic policies.
  arXiv  Detail & Related papers  (2024-03-21T11:54:45Z)
• Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RL [86.0987896274354]
  We first identify a fundamental pattern, self-excitation, as the primary cause of Q-value estimation divergence in offline RL. We then propose a novel Self-Excite Eigenvalue Measure (SEEM) metric to measure the evolving property of Q-network at training. For the first time, our theory can reliably decide whether the training will diverge at an early stage.
  arXiv  Detail & Related papers  (2023-10-06T17:57:44Z)
• Gradient Surgery for One-shot Unlearning on Generative Model [0.989293617504294]
  We introduce a simple yet effective approach to remove a data influence on the deep generative model. Inspired by works in multi-task learning, we propose to manipulate gradients to regularize the interplay of influence among samples.
  arXiv  Detail & Related papers  (2023-07-10T13:29:23Z)
• Self-Knowledge Distillation via Dropout [0.7883397954991659]
  We propose a simple and effective self-knowledge distillation method using a dropout (SD-Dropout) Our method does not require any additional trainable modules, does not rely on data, and requires only simple operations.
  arXiv  Detail & Related papers  (2022-08-11T05:08:55Z)
• Sample-Efficient Reinforcement Learning via Conservative Model-Based Actor-Critic [67.00475077281212]
  Model-based reinforcement learning algorithms are more sample efficient than their model-free counterparts. We propose a novel approach that achieves high sample efficiency without the strong reliance on accurate learned models. We show that CMBAC significantly outperforms state-of-the-art approaches in terms of sample efficiency on several challenging tasks.
  arXiv  Detail & Related papers  (2021-12-16T15:33:11Z)
• Churn Reduction via Distillation [54.5952282395487]
  We show an equivalence between training with distillation using the base model as the teacher and training with an explicit constraint on the predictive churn. We then show that distillation performs strongly for low churn training against a number of recent baselines.
  arXiv  Detail & Related papers  (2021-06-04T18:03:31Z)
• Black-Box Dissector: Towards Erasing-based Hard-Label Model Stealing Attack [90.6076825117532]
  Model stealing attack aims to create a substitute model that steals the ability of the victim target model. Most of the existing methods depend on the full probability outputs from the victim model, which is unavailable in most realistic scenarios. We propose a novel hard-label model stealing method termed emphblack-box dissector, which includes a CAM-driven erasing strategy to mine the hidden information in hard labels from the victim model.
  arXiv  Detail & Related papers  (2021-05-03T04:12:31Z)
• On the model-based stochastic value gradient for continuous reinforcement learning [50.085645237597056]
  We show that simple model-based agents can outperform state-of-the-art model-free agents in terms of both sample-efficiency and final reward. Our findings suggest that model-based policy evaluation deserves closer attention.
  arXiv  Detail & Related papers  (2020-08-28T17:58:29Z)
• Self-Knowledge Distillation with Progressive Refinement of Targets [1.1470070927586016]
  We propose a simple yet effective regularization method named progressive self-knowledge distillation (PS-KD) PS-KD progressively distills a model's own knowledge to soften hard targets during training. We show that PS-KD provides an effect of hard example mining by rescaling gradients according to difficulty in classifying examples.
  arXiv  Detail & Related papers  (2020-06-22T04:06:36Z)
• Modeling Survival in model-based Reinforcement Learning [0.0]
  This work presents the notion of survival by discussing cases in which the agent's goal is to survive. A substitute model for the reward function approxor is introduced that learns to avoid terminal states. Focusing on terminal states, as a small fraction of state-space, reduces the training effort drastically.
  arXiv  Detail & Related papers  (2020-04-18T15:49:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.