Counter-Dyna: Data-Efficient RL-Based HVAC Control using Counterfactual Building Models

• URL: http://arxiv.org/abs/2605.04555v1
• Date: Wed, 06 May 2026 06:58:41 GMT
• Title: Counter-Dyna: Data-Efficient RL-Based HVAC Control using Counterfactual Building Models
• Abstract summary: Model-based reinforcement learning (MBRL) offers a promising approach for data-efficient energy management in buildings.<n>Counter-Dyna is a method that enhances the data-efficiency of Dyna.<n>Our results show cost-saving potentials of 5.3% to 17.0% in a hypothetical deployment scenario.
• Score: 2.08239023124927
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Model-based reinforcement learning (MBRL) offers a promising approach for data-efficient energy management in buildings, combining the strengths of predictive modeling and reinforcement learning. While previous MBRL methods applied to HVAC control have reduced training data requirements, they still require several months of interaction with the building to learn a satisfactory control policy. A key reason is that existing surrogate models attempt to predict the entire state-space, including weather and electricity prices that are unaffected by control actions, or completely ignore these variables. Addressing these issues, we propose Counter-Dyna, a method that enhances the data-efficiency of Dyna, an MBRL method. We create data-efficient counterfactual surrogate models (CSM) by leveraging invariances in the state-space. Using a CSM in Dyna speeds up RL training measured in environment interaction data compared to previous results. In comparison with previous state-of-the-art that used 6-12 months of environment interactions, our method needs only 5 weeks. We evaluate our method in a large simulation study using the literature standard BOPTEST framework and proximal policy algorithm (PPO) as the RL algorithm. Our results show cost-saving potentials of 5.3% to 17.0% in a hypothetical deployment scenario. Our work is a significant step towards making real-world deployment of RL algorithms in HVAC control practically viable.

Related papers

• Reversal Q-Learning [57.43956630070019]
  Reversal Q-learning is an off-policy RL algorithm that trains a flow policy based on prior data.<n>We show that RQL leads to the best average offline RL performance compared to state-of-the-art flow-based offline RL algorithms.
  arXiv  Detail & Related papers  (2026-06-16T05:56:10Z)
• Data-Efficient RLVR via Off-Policy Influence Guidance [84.60336960383867]
  This work proposes a theoretically-grounded approach using influence functions to estimate the contribution of each data point to the learning objective.<n>We develop textbfCurriculum textbfRL with textbfOff-textbfPolicy textInfluence guidance (textbfCROPI), a multi-stage RL framework that iteratively selects the most influential data for the current policy.
  arXiv  Detail & Related papers  (2025-10-30T13:40:52Z)
• Comparative Field Deployment of Reinforcement Learning and Model Predictive Control for Residential HVAC [6.84547325352035]
  Reinforcement Learning promises greater automation and adaptability, yet its practical application in real-world residential settings remains largely undemonstrated.<n>This study's empirical results show that while RL reduces engineering overhead, it introduces practical trade-offs in model accuracy and operational robustness.
  arXiv  Detail & Related papers  (2025-10-01T21:36:04Z)
• Experimental evaluation of offline reinforcement learning for HVAC control in buildings [12.542463083734614]
  Reinforcement learning (RL) techniques have been increasingly investigated for dynamic HVAC control in buildings. This paper comprehensively evaluates the strengths and limitations of state-of-the-art offline RL algorithms.
  arXiv  Detail & Related papers  (2024-08-15T07:25:52Z)
• Go Beyond Black-box Policies: Rethinking the Design of Learning Agent for Interpretable and Verifiable HVAC Control [3.326392645107372]
  We overcome the bottleneck by redesigning HVAC controllers using decision trees extracted from thermal dynamics models and historical data. Our method saves 68.4% more energy and increases human comfort gain by 14.8% compared to the state-of-the-art method.
  arXiv  Detail & Related papers  (2024-02-29T22:42:23Z)
• Data-Efficient Task Generalization via Probabilistic Model-based Meta Reinforcement Learning [58.575939354953526]
  PACOH-RL is a novel model-based Meta-Reinforcement Learning (Meta-RL) algorithm designed to efficiently adapt control policies to changing dynamics. Existing Meta-RL methods require abundant meta-learning data, limiting their applicability in settings such as robotics. Our experiment results demonstrate that PACOH-RL outperforms model-based RL and model-based Meta-RL baselines in adapting to new dynamic conditions.
  arXiv  Detail & Related papers  (2023-11-13T18:51:57Z)
• Hybrid Reinforcement Learning for Optimizing Pump Sustainability in Real-World Water Distribution Networks [55.591662978280894]
  This article addresses the pump-scheduling optimization problem to enhance real-time control of real-world water distribution networks (WDNs) Our primary objectives are to adhere to physical operational constraints while reducing energy consumption and operational costs. Traditional optimization techniques, such as evolution-based and genetic algorithms, often fall short due to their lack of convergence guarantees.
  arXiv  Detail & Related papers  (2023-10-13T21:26:16Z)
• Efficient Learning of Voltage Control Strategies via Model-based Deep Reinforcement Learning [9.936452412191326]
  This article proposes a model-based deep reinforcement learning (DRL) method to design emergency control strategies for short-term voltage stability problems in power systems. Recent advances show promising results in model-free DRL-based methods for power systems, but model-free methods suffer from poor sample efficiency and training time. We propose a novel model-based-DRL framework where a deep neural network (DNN)-based dynamic surrogate model is utilized with the policy learning framework.
  arXiv  Detail & Related papers  (2022-12-06T02:50:53Z)
• Mastering the Unsupervised Reinforcement Learning Benchmark from Pixels [112.63440666617494]
  Reinforcement learning algorithms can succeed but require large amounts of interactions between the agent and the environment. We propose a new method to solve it, using unsupervised model-based RL, for pre-training the agent. We show robust performance on the Real-Word RL benchmark, hinting at resiliency to environment perturbations during adaptation.
  arXiv  Detail & Related papers  (2022-09-24T14:22:29Z)
• An Experimental Design Perspective on Model-Based Reinforcement Learning [73.37942845983417]
  In practical applications of RL, it is expensive to observe state transitions from the environment. We propose an acquisition function that quantifies how much information a state-action pair would provide about the optimal solution to a Markov decision process.
  arXiv  Detail & Related papers  (2021-12-09T23:13:57Z)
• Development of a Soft Actor Critic Deep Reinforcement Learning Approach for Harnessing Energy Flexibility in a Large Office Building [0.0]
  This research is concerned with the novel application and investigation of Soft Actor Critic' (SAC) based Deep Reinforcement Learning (DRL) SAC is a model-free DRL technique that is able to handle continuous action spaces.
  arXiv  Detail & Related papers  (2021-04-25T10:33:35Z)
• Information Theoretic Model Predictive Q-Learning [64.74041985237105]
  We present a novel theoretical connection between information theoretic MPC and entropy regularized RL. We develop a Q-learning algorithm that can leverage biased models.
  arXiv  Detail & Related papers  (2019-12-31T00:29:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.