Turbulence control in plane Couette flow using low-dimensional neural
ODE-based models and deep reinforcement learning
- URL: http://arxiv.org/abs/2301.12098v1
- Date: Sat, 28 Jan 2023 05:47:10 GMT
- Title: Turbulence control in plane Couette flow using low-dimensional neural
ODE-based models and deep reinforcement learning
- Authors: Alec J. Linot and Kevin Zeng and Michael D. Graham
- Abstract summary: "DManD-RL" (data-driven manifold dynamics-RL) generates a data-driven low-dimensional model of our system.
We train an RL control agent, yielding a 440-fold speedup over training on a numerical simulation.
The agent learns a policy that laminarizes 84% of unseen DNS test trajectories within 900 time units.
- Score: 0.0
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The high dimensionality and complex dynamics of turbulent flows remain an
obstacle to the discovery and implementation of control strategies. Deep
reinforcement learning (RL) is a promising avenue for overcoming these
obstacles, but requires a training phase in which the RL agent iteratively
interacts with the flow environment to learn a control policy, which can be
prohibitively expensive when the environment involves slow experiments or
large-scale simulations. We overcome this challenge using a framework we call
"DManD-RL" (data-driven manifold dynamics-RL), which generates a data-driven
low-dimensional model of our system that we use for RL training. With this
approach, we seek to minimize drag in a direct numerical simulation (DNS) of a
turbulent minimal flow unit of plane Couette flow at Re=400 using two slot jets
on one wall. We obtain, from DNS data with $\mathcal{O}(10^5)$ degrees of
freedom, a 25-dimensional DManD model of the dynamics by combining an
autoencoder and neural ordinary differential equation. Using this model as the
environment, we train an RL control agent, yielding a 440-fold speedup over
training on the DNS, with equivalent control performance. The agent learns a
policy that laminarizes 84% of unseen DNS test trajectories within 900 time
units, significantly outperforming classical opposition control (58%), despite
the actuation authority being much more restricted. The agent often achieves
laminarization through a counterintuitive strategy that drives the formation of
two low-speed streaks, with a spanwise wavelength that is too small to be
self-sustaining. The agent demonstrates the same performance when we limit
observations to wall shear rate.
Related papers
- DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning [61.10299147201369]
This paper introduces a novel autonomous RL approach, called DigiRL, for training in-the-wild device control agents.
We build a scalable and parallelizable Android learning environment equipped with a VLM-based evaluator.
We demonstrate the effectiveness of DigiRL using the Android-in-the-Wild dataset, where our 1.3B VLM trained with RL achieves a 49.5% absolute improvement.
arXiv Detail & Related papers (2024-06-14T17:49:55Z) - SINDy-RL: Interpretable and Efficient Model-Based Reinforcement Learning [5.59265003686955]
We introduce SINDy-RL, a framework for combining SINDy and deep reinforcement learning.
SINDy-RL achieves comparable performance to state-of-the-art DRL algorithms.
We demonstrate the effectiveness of our approaches on benchmark control environments and challenging fluids problems.
arXiv Detail & Related papers (2024-03-14T05:17:39Z) - Compressing Deep Reinforcement Learning Networks with a Dynamic
Structured Pruning Method for Autonomous Driving [63.155562267383864]
Deep reinforcement learning (DRL) has shown remarkable success in complex autonomous driving scenarios.
DRL models inevitably bring high memory consumption and computation, which hinders their wide deployment in resource-limited autonomous driving devices.
We introduce a novel dynamic structured pruning approach that gradually removes a DRL model's unimportant neurons during the training stage.
arXiv Detail & Related papers (2024-02-07T09:00:30Z) - Learning to Fly in Seconds [7.259696592534715]
We show how curriculum learning and a highly optimized simulator enhance sample complexity and lead to fast training times.
Our framework enables Simulation-to-Reality (Sim2Real) transfer for direct control after only 18 seconds of training on a consumer-grade laptop.
arXiv Detail & Related papers (2023-11-22T01:06:45Z) - Real-Time Model-Free Deep Reinforcement Learning for Force Control of a
Series Elastic Actuator [56.11574814802912]
State-of-the art robotic applications utilize series elastic actuators (SEAs) with closed-loop force control to achieve complex tasks such as walking, lifting, and manipulation.
Model-free PID control methods are more prone to instability due to nonlinearities in the SEA.
Deep reinforcement learning has proved to be an effective model-free method for continuous control tasks.
arXiv Detail & Related papers (2023-04-11T00:51:47Z) - Efficient Learning of Voltage Control Strategies via Model-based Deep
Reinforcement Learning [9.936452412191326]
This article proposes a model-based deep reinforcement learning (DRL) method to design emergency control strategies for short-term voltage stability problems in power systems.
Recent advances show promising results in model-free DRL-based methods for power systems, but model-free methods suffer from poor sample efficiency and training time.
We propose a novel model-based-DRL framework where a deep neural network (DNN)-based dynamic surrogate model is utilized with the policy learning framework.
arXiv Detail & Related papers (2022-12-06T02:50:53Z) - Mastering the Unsupervised Reinforcement Learning Benchmark from Pixels [112.63440666617494]
Reinforcement learning algorithms can succeed but require large amounts of interactions between the agent and the environment.
We propose a new method to solve it, using unsupervised model-based RL, for pre-training the agent.
We show robust performance on the Real-Word RL benchmark, hinting at resiliency to environment perturbations during adaptation.
arXiv Detail & Related papers (2022-09-24T14:22:29Z) - Data-driven control of spatiotemporal chaos with reduced-order neural
ODE-based models and reinforcement learning [0.0]
Deep learning is capable of discovering complex control strategies for high-dimensional systems, making it promising for flow control applications.
A major challenge associated with RL is that substantial training data must be generated by repeatedly interacting with the target system.
We use a data-driven reduced-order model (ROM) in place the true system during RL training to efficiently estimate the optimal policy.
We show that the ROM-based control strategy translates well to the true KSE and highlight that the RL agent discovers and stabilizes an underlying forced equilibrium solution of the KSE system.
arXiv Detail & Related papers (2022-05-01T23:25:44Z) - Accelerated Policy Learning with Parallel Differentiable Simulation [59.665651562534755]
We present a differentiable simulator and a new policy learning algorithm (SHAC)
Our algorithm alleviates problems with local minima through a smooth critic function.
We show substantial improvements in sample efficiency and wall-clock time over state-of-the-art RL and differentiable simulation-based algorithms.
arXiv Detail & Related papers (2022-04-14T17:46:26Z) - Online Convolutional Re-parameterization [51.97831675242173]
We present online convolutional re- parameterization (OREPA), a two-stage pipeline, aiming to reduce the huge training overhead by squeezing the complex training-time block into a single convolution.
Compared with the state-of-the-art re-param models, OREPA is able to save the training-time memory cost by about 70% and accelerate the training speed by around 2x.
We also conduct experiments on object detection and semantic segmentation and show consistent improvements on the downstream tasks.
arXiv Detail & Related papers (2022-04-02T09:50:19Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.