Effective control of two-dimensional Rayleigh--B\'enard convection: invariant multi-agent reinforcement learning is all you need

• URL: http://arxiv.org/abs/2304.02370v2
• Date: Tue, 13 Jun 2023 08:55:45 GMT
• Title: Effective control of two-dimensional Rayleigh--B\'enard convection: invariant multi-agent reinforcement learning is all you need
• Authors: Colin Vignon, Jean Rabault, Joel Vasanth, Francisco Alc\'antara-\'Avila, Mikael Mortensen, Ricardo Vinuesa
• Abstract summary: Control of Rayleigh-B'enard convection (RBC) is a challenging topic for classical control-theory methods. We show that effective RBC control can be obtained by leveraging invariant multi-agent reinforcement learning (MARL) We show in a case study that MARL DRL is able to discover an advanced control strategy that destabilizes the spontaneous RBC double-cell pattern.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Rayleigh-B\'enard convection (RBC) is a recurrent phenomenon in several industrial and geoscience flows and a well-studied system from a fundamental fluid-mechanics viewpoint. However, controlling RBC, for example by modulating the spatial distribution of the bottom-plate heating in the canonical RBC configuration, remains a challenging topic for classical control-theory methods. In the present work, we apply deep reinforcement learning (DRL) for controlling RBC. We show that effective RBC control can be obtained by leveraging invariant multi-agent reinforcement learning (MARL), which takes advantage of the locality and translational invariance inherent to RBC flows inside wide channels. The MARL framework applied to RBC allows for an increase in the number of control segments without encountering the curse of dimensionality that would result from a naive increase in the DRL action-size dimension. This is made possible by the MARL ability for re-using the knowledge generated in different parts of the RBC domain. We show in a case study that MARL DRL is able to discover an advanced control strategy that destabilizes the spontaneous RBC double-cell pattern, changes the topology of RBC by coalescing adjacent convection cells, and actively controls the resulting coalesced cell to bring it to a new stable configuration. This modified flow configuration results in reduced convective heat transfer, which is beneficial in several industrial processes. Therefore, our work both shows the potential of MARL DRL for controlling large RBC systems, as well as demonstrates the possibility for DRL to discover strategies that move the RBC configuration between different topological configurations, yielding desirable heat-transfer characteristics. These results are useful for both gaining further understanding of the intrinsic properties of RBC, as well as for developing industrial applications.

Related papers

• Offline Behavior Distillation [57.6900189406964]
  Massive reinforcement learning (RL) data are typically collected to train policies offline without the need for interactions. We formulate offline behavior distillation (OBD), which synthesizes limited expert behavioral data from sub-optimal RL data. We propose two naive OBD objectives, DBC and PBC, which measure distillation performance via the decision difference between policies trained on distilled data and either offline data or a near-expert policy.
  arXiv  Detail & Related papers  (2024-10-30T06:28:09Z)
• Bellman Diffusion: Generative Modeling as Learning a Linear Operator in the Distribution Space [72.52365911990935]
  We introduce Bellman Diffusion, a novel DGM framework that maintains linearity in MDPs through gradient and scalar field modeling. Our results show that Bellman Diffusion achieves accurate field estimations and is a capable image generator, converging 1.5x faster than the traditional histogram-based baseline in distributional RL tasks.
  arXiv  Detail & Related papers  (2024-10-02T17:53:23Z)
• Multi-agent reinforcement learning for the control of three-dimensional Rayleigh-Bénard convection [0.7864304771129751]
  Multi-agent RL (MARL) has shown to be more effective than single-agent RL in controlling flows exhibiting locality and translational invariance. We present for the first time, an implementation of MARL-based control of three-dimensional Rayleigh-B'enard convection.
  arXiv  Detail & Related papers  (2024-07-31T12:41:20Z)
• Koopman-Based Surrogate Modelling of Turbulent Rayleigh-Bénard Convection [4.248022697109535]
  We use a Koopman-inspired architecture called the Linear Recurrent Autoencoder Network (LRAN) for learning reduced-order dynamics in convection flows. A traditional fluid dynamics method, the Kernel Dynamic Mode Decomposition (KDMD) is used to compare the LRAN. We obtained more accurate predictions with the LRAN than with KDMD in the most turbulent setting.
  arXiv  Detail & Related papers  (2024-05-10T12:15:02Z)
• Bayesian Reparameterization of Reward-Conditioned Reinforcement Learning with Energy-based Models [46.24690220893344]
  We show that current reward-conditioned reinforcement learning approaches are fundamentally limited. We propose a novel set of inductive biases for RCRL inspired by Bayes' theorem. We show that BR-RCRL achieves state-of-the-art performance on the Gym-Mujoco and Atari offline RL benchmarks.
  arXiv  Detail & Related papers  (2023-05-18T23:23:08Z)
• Low Emission Building Control with Zero-Shot Reinforcement Learning [70.70479436076238]
  Control via Reinforcement Learning (RL) has been shown to significantly improve building energy efficiency. We show it is possible to obtain emission-reducing policies without a priori--a paradigm we call zero-shot building control.
  arXiv  Detail & Related papers  (2022-08-12T17:13:25Z)
• Dynamic Mode Decomposition in Adaptive Mesh Refinement and Coarsening Simulations [58.720142291102135]
  Dynamic Mode Decomposition (DMD) is a powerful data-driven method used to extract coherent schemes. This paper proposes a strategy to enable DMD to extract from observations with different mesh topologies and dimensions.
  arXiv  Detail & Related papers  (2021-04-28T22:14:25Z)
• Towards Evaluating and Training Verifiably Robust Neural Networks [81.39994285743555]
  We study the relationship between IBP and CROWN, and prove that CROWN is always tighter than IBP when choosing appropriate bounding lines. We propose a relaxed version of CROWN, linear bound propagation (LBP), that can be used to verify large networks to obtain lower verified errors.
  arXiv  Detail & Related papers  (2021-04-01T13:03:48Z)
• Scalable Voltage Control using Structure-Driven Hierarchical Deep Reinforcement Learning [0.0]
  This paper presents a novel hierarchical deep reinforcement learning (DRL) based design for the voltage control of power grids. We exploit the area-wise division structure of the power system to propose a hierarchical DRL design that can be scaled to the larger grid models. We train area-wise decentralized RL agents to compute lower-level policies for the individual areas, and concurrently train a higher-level DRL agent that uses the updates of the lower-level policies to efficiently coordinate the control actions taken by the lower-level agents.
  arXiv  Detail & Related papers  (2021-01-29T21:30:59Z)
• Performance of Dual-Augmented Lagrangian Method and Common Spatial Patterns applied in classification of Motor-Imagery BCI [68.8204255655161]
  Motor-imagery based brain-computer interfaces (MI-BCI) have the potential to become ground-breaking technologies for neurorehabilitation. Due to the noisy nature of the used EEG signal, reliable BCI systems require specialized procedures for features optimization and extraction.
  arXiv  Detail & Related papers  (2020-10-13T20:50:13Z)
• Adaptive Energy Management for Real Driving Conditions via Transfer Reinforcement Learning [19.383907178714345]
  This article proposes a transfer reinforcement learning (RL) based adaptive energy managing approach for a hybrid electric vehicle (HEV) with parallel topology. The up-level characterizes how to transform the Q-value tables in the RL framework via driving cycle transformation (DCT) The lower-level determines how to set the corresponding control strategies with the transformed Q-value tables and TPMs.
  arXiv  Detail & Related papers  (2020-07-24T15:06:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.