Distill2Explain: Differentiable decision trees for explainable reinforcement learning in energy application controllers

• URL: http://arxiv.org/abs/2403.11907v1
• Date: Mon, 18 Mar 2024 16:09:49 GMT
• Title: Distill2Explain: Differentiable decision trees for explainable reinforcement learning in energy application controllers
• Authors: Gargya Gokhale, Seyed Soroush Karimi Madahi, Bert Claessens, Chris Develder,
• Abstract summary: Residential sector is an important (potential) source of energy flexibility. A potential control framework for such a task is data-driven control, specifically model-free reinforcement learning (RL) RLs learn a good control policy by interacting with their environment, learning purely based on data and with minimal human intervention. We propose a novel method to obtain explainable RL policies by using differentiable decision trees.
• Score: 5.311053322050159
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Demand-side flexibility is gaining importance as a crucial element in the energy transition process. Accounting for about 25% of final energy consumption globally, the residential sector is an important (potential) source of energy flexibility. However, unlocking this flexibility requires developing a control framework that (1) easily scales across different houses, (2) is easy to maintain, and (3) is simple to understand for end-users. A potential control framework for such a task is data-driven control, specifically model-free reinforcement learning (RL). Such RL-based controllers learn a good control policy by interacting with their environment, learning purely based on data and with minimal human intervention. Yet, they lack explainability, which hampers user acceptance. Moreover, limited hardware capabilities of residential assets forms a hurdle (e.g., using deep neural networks). To overcome both those challenges, we propose a novel method to obtain explainable RL policies by using differentiable decision trees. Using a policy distillation approach, we train these differentiable decision trees to mimic standard RL-based controllers, leading to a decision tree-based control policy that is data-driven and easy to explain. As a proof-of-concept, we examine the performance and explainability of our proposed approach in a battery-based home energy management system to reduce energy costs. For this use case, we show that our proposed approach can outperform baseline rule-based policies by about 20-25%, while providing simple, explainable control policies. We further compare these explainable policies with standard RL policies and examine the performance trade-offs associated with this increased explainability.

Related papers

• CityLearn v2: Energy-flexible, resilient, occupant-centric, and carbon-aware management of grid-interactive communities [8.658740257657564]
  CityLearn provides an environment for benchmarking simple and advanced distributed energy resource control algorithms. This work details the v2 environment design and provides application examples that utilize reinforcement learning to manage battery energy storage system charging/discharging cycles, vehicle-to-grid control, and thermal comfort during heat pump power modulation.
  arXiv  Detail & Related papers  (2024-05-02T16:31:09Z)
• Distilling Reinforcement Learning Policies for Interpretable Robot Locomotion: Gradient Boosting Machines and Symbolic Regression [53.33734159983431]
  This paper introduces a novel approach to distill neural RL policies into more interpretable forms. We train expert neural network policies using RL and distill them into (i) GBMs, (ii) EBMs, and (iii) symbolic policies.
  arXiv  Detail & Related papers  (2024-03-21T11:54:45Z)
• Explainable Reinforcement Learning-based Home Energy Management Systems using Differentiable Decision Trees [4.573008040057806]
  The residential sector is another major and largely untapped source of flexibility, driven by the increased adoption of solar PV, home batteries, and EVs. We introduce a reinforcement learning-based approach using differentiable decision trees. This approach integrates the scalability of data-driven reinforcement learning with the explainability of (differentiable) decision trees. As a proof-of-concept, we analyze our method using a home energy management problem, comparing its performance with commercially available rule-based baseline and standard neural network-based RL controllers.
  arXiv  Detail & Related papers  (2024-03-18T16:40:41Z)
• Deployable Reinforcement Learning with Variable Control Rate [14.838483990647697]
  We propose a variant of Reinforcement Learning (RL) with variable control rate. In this approach, the policy decides the action the agent should take as well as the duration of the time step associated with that action. We show the efficacy of SEAC through a proof-of-concept simulation driving an agent with Newtonian kinematics.
  arXiv  Detail & Related papers  (2024-01-17T15:40:11Z)
• Conformal Policy Learning for Sensorimotor Control Under Distribution Shifts [61.929388479847525]
  This paper focuses on the problem of detecting and reacting to changes in the distribution of a sensorimotor controller's observables. The key idea is the design of switching policies that can take conformal quantiles as input. We show how to design such policies by using conformal quantiles to switch between base policies with different characteristics.
  arXiv  Detail & Related papers  (2023-11-02T17:59:30Z)
• Multi-market Energy Optimization with Renewables via Reinforcement Learning [1.0878040851638]
  This paper introduces a deep reinforcement learning framework for optimizing the operations of power plants pairing renewable energy with storage. The framework handles complexities such as time coupling by storage devices, uncertainty in renewable generation and energy prices, and non-linear storage models. It utilizes RL to incorporate complex storage models, overcoming restrictions of optimization-based methods that require convex and differentiable component models.
  arXiv  Detail & Related papers  (2023-06-13T21:35:24Z)
• Learning Resilient Radio Resource Management Policies with Graph Neural Networks [124.89036526192268]
  We formulate a resilient radio resource management problem with per-user minimum-capacity constraints. We show that we can parameterize the user selection and power control policies using a finite set of parameters. Thanks to such adaptation, our proposed method achieves a superior tradeoff between the average rate and the 5th percentile rate.
  arXiv  Detail & Related papers  (2022-03-07T19:40:39Z)
• Enforcing Policy Feasibility Constraints through Differentiable Projection for Energy Optimization [57.88118988775461]
  We propose PROjected Feasibility (PROF) to enforce convex operational constraints within neural policies. We demonstrate PROF on two applications: energy-efficient building operation and inverter control.
  arXiv  Detail & Related papers  (2021-05-19T01:58:10Z)
• A Relearning Approach to Reinforcement Learning for Control of Smart Buildings [1.8799681615947088]
  This paper demonstrates that continual relearning of control policies using incremental deep reinforcement learning (RL) can improve policy learning for non-stationary processes. We develop an incremental RL technique that simultaneously reduces building energy consumption without sacrificing overall comfort.
  arXiv  Detail & Related papers  (2020-08-04T23:31:05Z)
• Strictly Batch Imitation Learning by Energy-based Distribution Matching [104.33286163090179]
  Consider learning a policy purely on the basis of demonstrated behavior -- that is, with no access to reinforcement signals, no knowledge of transition dynamics, and no further interaction with the environment. One solution is simply to retrofit existing algorithms for apprenticeship learning to work in the offline setting. But such an approach leans heavily on off-policy evaluation or offline model estimation, and can be indirect and inefficient. We argue that a good solution should be able to explicitly parameterize a policy, implicitly learn from rollout dynamics, and operate in an entirely offline fashion.
  arXiv  Detail & Related papers  (2020-06-25T03:27:59Z)
• Multi-Agent Meta-Reinforcement Learning for Self-Powered and Sustainable Edge Computing Systems [87.4519172058185]
  An effective energy dispatch mechanism for self-powered wireless networks with edge computing capabilities is studied. A novel multi-agent meta-reinforcement learning (MAMRL) framework is proposed to solve the formulated problem. Experimental results show that the proposed MAMRL model can reduce up to 11% non-renewable energy usage and by 22.4% the energy cost.
  arXiv  Detail & Related papers  (2020-02-20T04:58:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.