Control of Rayleigh-Bénard Convection: Effectiveness of Reinforcement Learning in the Turbulent Regime

• URL: http://arxiv.org/abs/2504.12000v1
• Date: Wed, 16 Apr 2025 11:51:59 GMT
• Title: Control of Rayleigh-Bénard Convection: Effectiveness of Reinforcement Learning in the Turbulent Regime
• Authors: Thorben Markmann, Michiel Straat, Sebastian Peitz, Barbara Hammer,
• Abstract summary: We study the effectiveness of Reinforcement Learning (RL) for reducing convective heat transfer under increasing turbulence.<n>RL agents trained via single-agent Proximal Policy Optimization (PPO) are compared to linear proportional derivative (PD) controllers.<n>The RL agents reduced convection, measured by the Nusselt Number, by up to 33% in moderately turbulent systems and 10% in highly turbulent settings.
• Score: 5.138559709801885
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Data-driven flow control has significant potential for industry, energy systems, and climate science. In this work, we study the effectiveness of Reinforcement Learning (RL) for reducing convective heat transfer in the 2D Rayleigh-B\'enard Convection (RBC) system under increasing turbulence. We investigate the generalizability of control across varying initial conditions and turbulence levels and introduce a reward shaping technique to accelerate the training. RL agents trained via single-agent Proximal Policy Optimization (PPO) are compared to linear proportional derivative (PD) controllers from classical control theory. The RL agents reduced convection, measured by the Nusselt Number, by up to 33% in moderately turbulent systems and 10% in highly turbulent settings, clearly outperforming PD control in all settings. The agents showed strong generalization performance across different initial conditions and to a significant extent, generalized to higher degrees of turbulence. The reward shaping improved sample efficiency and consistently stabilized the Nusselt Number to higher turbulence levels.

Related papers

• Nuclear Microreactor Control with Deep Reinforcement Learning [0.40498500266986387]
  This study explores the application of deep reinforcement learning (RL) for real-time drum control in microreactors.<n>RL controllers can achieve similar or even superior load-following performance as traditional proportional-integral-derivative (PID) controllers.
  arXiv  Detail & Related papers  (2025-03-31T19:11:19Z)
• Improving a Proportional Integral Controller with Reinforcement Learning on a Throttle Valve Benchmark [2.8322124733515666]
  This paper presents a learning-based control strategy for non-linear throttle valves with an asymmetric controller. We exploit the recent advances in Reinforcement Learning with Guides to improve the closed-loop behavior by learning from the additional interactions with the valve. In all the experimental test cases, the resulting agent has a better sample efficiency than traditional RL agents and outperforms the PI controller.
  arXiv  Detail & Related papers  (2024-02-21T09:40:26Z)
• Reinforcement learning to maximise wind turbine energy generation [0.8437187555622164]
  We propose a reinforcement learning strategy to control wind turbine energy generation by actively changing the rotor speed, the rotor yaw angle and the blade pitch angle. A double deep Q-learning with a prioritized experience replay agent is coupled with a blade element momentum model and is trained to allow control for changing winds. The agent is trained to decide the best control (speed, yaw, pitch) for simple steady winds and is subsequently challenged with real dynamic turbulent winds, showing good performance.
  arXiv  Detail & Related papers  (2024-02-17T21:35:13Z)
• An experimental evaluation of Deep Reinforcement Learning algorithms for HVAC control [40.71019623757305]
  Recent studies have shown that Deep Reinforcement Learning (DRL) algorithms can outperform traditional reactive controllers. This paper provides a critical and reproducible evaluation of several state-of-the-art DRL algorithms for HVAC control.
  arXiv  Detail & Related papers  (2024-01-11T08:40:26Z)
• Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregression [70.78523583702209]
  We study training instabilities of behavior cloning with deep neural networks. We observe that minibatch SGD updates to the policy network during training result in sharp oscillations in long-horizon rewards.
  arXiv  Detail & Related papers  (2023-10-17T17:39:40Z)
• Hybrid Reinforcement Learning for Optimizing Pump Sustainability in Real-World Water Distribution Networks [55.591662978280894]
  This article addresses the pump-scheduling optimization problem to enhance real-time control of real-world water distribution networks (WDNs) Our primary objectives are to adhere to physical operational constraints while reducing energy consumption and operational costs. Traditional optimization techniques, such as evolution-based and genetic algorithms, often fall short due to their lack of convergence guarantees.
  arXiv  Detail & Related papers  (2023-10-13T21:26:16Z)
• Surrogate Empowered Sim2Real Transfer of Deep Reinforcement Learning for ORC Superheat Control [12.567922037611261]
  This paper proposes a Sim2Real transfer learning-based DRL control method for ORC superheat control. Experimental results show that the proposed method greatly improves the training speed of DRL in ORC control problems.
  arXiv  Detail & Related papers  (2023-08-05T01:59:44Z)
• Efficient Deep Reinforcement Learning Requires Regulating Overfitting [91.88004732618381]
  We show that high temporal-difference (TD) error on the validation set of transitions is the main culprit that severely affects the performance of deep RL algorithms. We show that a simple online model selection method that targets the validation TD error is effective across state-based DMC and Gym tasks.
  arXiv  Detail & Related papers  (2023-04-20T17:11:05Z)
• Learning to Optimize for Reinforcement Learning [58.01132862590378]
  Reinforcement learning (RL) is essentially different from supervised learning, and in practice, these learneds do not work well even in simple RL tasks. Agent-gradient distribution is non-independent and identically distributed, leading to inefficient meta-training. We show that, although only trained in toy tasks, our learned can generalize unseen complex tasks in Brax.
  arXiv  Detail & Related papers  (2023-02-03T00:11:02Z)
• Robust Deep Reinforcement Learning through Adversarial Loss [74.20501663956604]
  Recent studies have shown that deep reinforcement learning agents are vulnerable to small adversarial perturbations on the agent's inputs. We propose RADIAL-RL, a principled framework to train reinforcement learning agents with improved robustness against adversarial attacks.
  arXiv  Detail & Related papers  (2020-08-05T07:49:42Z)
• Understanding the Difficulty of Training Transformers [120.99980924577787]
  We show that unbalanced gradients are not the root cause of the instability of training. We propose Admin to stabilize the early stage's training and unleash its full potential in the late stage.
  arXiv  Detail & Related papers  (2020-04-17T13:59:07Z)
• Controlling Rayleigh-B\'enard convection via Reinforcement Learning [62.997667081978825]
  The identification of effective control strategies to suppress or enhance the convective heat exchange under fixed external thermal gradients is an outstanding fundamental and technological issue. In this work, we explore a novel approach, based on a state-of-the-art Reinforcement Learning (RL) algorithm. We show that our RL-based control is able to stabilize the conductive regime and bring the onset of convection up to a Rayleigh number.
  arXiv  Detail & Related papers  (2020-03-31T16:39:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.