Experimental evaluation of offline reinforcement learning for HVAC control in buildings

• URL: http://arxiv.org/abs/2408.07986v1
• Date: Thu, 15 Aug 2024 07:25:52 GMT
• Title: Experimental evaluation of offline reinforcement learning for HVAC control in buildings
• Authors: Jun Wang, Linyan Li, Qi Liu, Yu Yang,
• Abstract summary: Reinforcement learning (RL) techniques have been increasingly investigated for dynamic HVAC control in buildings. This paper comprehensively evaluates the strengths and limitations of state-of-the-art offline RL algorithms.
• Score: 12.542463083734614
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning (RL) techniques have been increasingly investigated for dynamic HVAC control in buildings. However, most studies focus on exploring solutions in online or off-policy scenarios without discussing in detail the implementation feasibility or effectiveness of dealing with purely offline datasets or trajectories. The lack of these works limits the real-world deployment of RL-based HVAC controllers, especially considering the abundance of historical data. To this end, this paper comprehensively evaluates the strengths and limitations of state-of-the-art offline RL algorithms by conducting analytical and numerical studies. The analysis is conducted from two perspectives: algorithms and dataset characteristics. As a prerequisite, the necessity of applying offline RL algorithms is first confirmed in two building environments. The ability of observation history modeling to reduce violations and enhance performance is subsequently studied. Next, the performance of RL-based controllers under datasets with different qualitative and quantitative conditions is investigated, including constraint satisfaction and power consumption. Finally, the sensitivity of certain hyperparameters is also evaluated. The results indicate that datasets of a certain suboptimality level and relatively small scale can be utilized to effectively train a well-performed RL-based HVAC controller. Specifically, such controllers can reduce at most 28.5% violation ratios of indoor temperatures and achieve at most 12.1% power savings compared to the baseline controller. In summary, this paper presents our well-structured investigations and new findings when applying offline reinforcement learning to building HVAC systems.

Related papers

• D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learning [99.33607114541861]
  We propose a new benchmark for offline RL that focuses on realistic simulations of robotic manipulation and locomotion environments. Our proposed benchmark covers state-based and image-based domains, and supports both offline RL and online fine-tuning evaluation.
  arXiv  Detail & Related papers  (2024-08-15T22:27:00Z)
• Go Beyond Black-box Policies: Rethinking the Design of Learning Agent for Interpretable and Verifiable HVAC Control [3.326392645107372]
  We overcome the bottleneck by redesigning HVAC controllers using decision trees extracted from thermal dynamics models and historical data. Our method saves 68.4% more energy and increases human comfort gain by 14.8% compared to the state-of-the-art method.
  arXiv  Detail & Related papers  (2024-02-29T22:42:23Z)
• An experimental evaluation of Deep Reinforcement Learning algorithms for HVAC control [40.71019623757305]
  Recent studies have shown that Deep Reinforcement Learning (DRL) algorithms can outperform traditional reactive controllers. This paper provides a critical and reproducible evaluation of several state-of-the-art DRL algorithms for HVAC control.
  arXiv  Detail & Related papers  (2024-01-11T08:40:26Z)
• Hybrid Reinforcement Learning for Optimizing Pump Sustainability in Real-World Water Distribution Networks [55.591662978280894]
  This article addresses the pump-scheduling optimization problem to enhance real-time control of real-world water distribution networks (WDNs) Our primary objectives are to adhere to physical operational constraints while reducing energy consumption and operational costs. Traditional optimization techniques, such as evolution-based and genetic algorithms, often fall short due to their lack of convergence guarantees.
  arXiv  Detail & Related papers  (2023-10-13T21:26:16Z)
• Look Beneath the Surface: Exploiting Fundamental Symmetry for Sample-Efficient Offline RL [29.885978495034703]
  offline reinforcement learning (RL) offers an appealing approach to real-world tasks by learning policies from pre-collected datasets. However, the performance of existing offline RL algorithms heavily depends on the scale and state-action space coverage of datasets. We provide a new insight that leveraging the fundamental symmetry of system dynamics can substantially enhance offline RL performance under small datasets.
  arXiv  Detail & Related papers  (2023-06-07T07:51:05Z)
• Revisiting the Minimalist Approach to Offline Reinforcement Learning [52.0035089982277]
  ReBRAC is a minimalistic algorithm that integrates design elements built on top of the TD3+BC method. We evaluate ReBRAC on 51 datasets with both proprioceptive and visual state spaces using D4RL and V-D4RL benchmarks.
  arXiv  Detail & Related papers  (2023-05-16T22:37:01Z)
• Efficient Deep Reinforcement Learning Requires Regulating Overfitting [91.88004732618381]
  We show that high temporal-difference (TD) error on the validation set of transitions is the main culprit that severely affects the performance of deep RL algorithms. We show that a simple online model selection method that targets the validation TD error is effective across state-based DMC and Gym tasks.
  arXiv  Detail & Related papers  (2023-04-20T17:11:05Z)
• A Reinforcement Learning-based Volt-VAR Control Dataset and Testing Environment [4.386026071380442]
  This paper introduces a suite of open-source datasets for RL-based VVC algorithm research that is sample efficient, safe, and robust. The dataset consists of two components: 1. a Gym-like VVC testing environment for the IEEE-13, 123, and 8500-bus test feeders and 2. a historical operational dataset for each of the feeders.
  arXiv  Detail & Related papers  (2022-04-20T14:44:55Z)
• On the Robustness of Controlled Deep Reinforcement Learning for Slice Placement [0.8459686722437155]
  We compare two Deep Reinforcement Learning algorithms: a pure DRL-based algorithm and a hybrid DRL as a hybrid DRL-heuristic algorithm. The evaluation results show that the proposed hybrid DRL-heuristic approach is more robust and reliable in case of unpredictable network load changes than pure DRL.
  arXiv  Detail & Related papers  (2021-08-05T10:24:33Z)
• RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning [108.9599280270704]
  We propose a benchmark called RL Unplugged to evaluate and compare offline RL methods. RL Unplugged includes data from a diverse range of domains including games and simulated motor control problems. We will release data for all our tasks and open-source all algorithms presented in this paper.
  arXiv  Detail & Related papers  (2020-06-24T17:14:51Z)
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning [119.49182500071288]
  We introduce benchmarks specifically designed for the offline setting, guided by key properties of datasets relevant to real-world applications of offline RL. By moving beyond simple benchmark tasks and data collected by partially-trained RL agents, we reveal important and unappreciated deficiencies of existing algorithms.
  arXiv  Detail & Related papers  (2020-04-15T17:18:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.