Enhancing sample efficiency in reinforcement-learning-based flow control: replacing the critic with an adaptive reduced-order model

• URL: http://arxiv.org/abs/2604.04986v1
• Date: Sun, 05 Apr 2026 01:48:32 GMT
• Title: Enhancing sample efficiency in reinforcement-learning-based flow control: replacing the critic with an adaptive reduced-order model
• Abstract summary: This work introduces an adaptive reduced-order-model (ROM)-based reinforcement learning framework for active flow control.<n>The ROM integrates a linear dynamical system and a neural ordinary differential equation (NODE) for estimating the nonlinearity in the flow.<n>It is validated on two canonical flow control problems: Blasius boundary layer flow and flow past a square cylinder.
• Score: 7.530352384088837
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Model-free deep reinforcement learning (DRL) methods suffer from poor sample efficiency. To overcome this limitation, this work introduces an adaptive reduced-order-model (ROM)-based reinforcement learning framework for active flow control. In contrast to conventional actor--critic architectures, the proposed approach leverages a ROM to estimate the gradient information required for controller optimization. The design of the ROM structure incorporates physical insights. The ROM integrates a linear dynamical system and a neural ordinary differential equation (NODE) for estimating the nonlinearity in the flow. The parameters of the linear component are identified via operator inference, while the NODE is trained in a data-driven manner using gradient-based optimization. During controller--environment interactions, the ROM is continuously updated with newly collected data, enabling adaptive refinement of the model. The controller is then optimized through differentiable simulation of the ROM. The proposed ROM-based DRL framework is validated on two canonical flow control problems: Blasius boundary layer flow and flow past a square cylinder. For the Blasius boundary layer, the proposed method effectively reduces to a single-episode system identification and controller optimization process, yet it yields controllers that outperform traditional linear designs and achieve performance comparable to DRL approaches with minimal data. For the flow past a square cylinder, the proposed method achieves superior drag reduction with significantly fewer exploration data compared with DRL approaches. The work addresses a key component of model-free DRL control algorithms and lays the foundation for designing more sample-efficient DRL-based active flow controllers.

Related papers

• Reversal Q-Learning [57.43956630070019]
  Reversal Q-learning is an off-policy RL algorithm that trains a flow policy based on prior data.<n>We show that RQL leads to the best average offline RL performance compared to state-of-the-art flow-based offline RL algorithms.
  arXiv  Detail & Related papers  (2026-06-16T05:56:10Z)
• Diffusion Controller: Framework, Algorithms and Parameterization [54.82539154511621]
  We present a unified control-theoretic view that casts reverse diffusion sampling as state-only control within (generalized) linearly-solvable Markov Decision Processes.<n>Under this framework, control acts by reweighting the pretrained reverse-time transition kernels, balancing terminal objectives against an $f$-divergence cost.<n>Experiments on Stable Diffusion v1.4 show consistent gains in preference-alignment win rates and improved quality-efficiency trade-offs.
  arXiv  Detail & Related papers  (2026-03-07T01:49:59Z)
• Optimal Derivative Feedback Control for an Active Magnetic Levitation System: An Experimental Study on Data-Driven Approaches [0.0]
  This paper presents the design and implementation of data-driven optimal derivative feedback controllers for an active magnetic levitation system.<n>For the direct model-free approach, a policy iteration procedure is proposed, which adds an iteration layer called the epoch loop to gather multiple sets of process data.<n>Results show that while both controllers can stabilize and improve the performance of the magnetic levitation system when compared to controllers designed from a nominal model, the direct model-free approach consistently outperforms the indirect solution when multiple epochs are allowed.
  arXiv  Detail & Related papers  (2026-02-06T18:42:01Z)
• Adaptive Linear Path Model-Based Diffusion [52.84663832658799]
  We introduce Linear Path Model-Based Diffusion (LP-MBD), which replaces the variance-preserving schedule with a flow-matching linear probability path.<n>We also propose Adaptive LP-MBD (ALP-MBD), which leverages reinforcement learning to adjust diffusion steps and noise levels according to task complexity and environmental conditions.
  arXiv  Detail & Related papers  (2026-02-02T21:33:03Z)
• Controllable Flow Matching for Online Reinforcement Learning [5.944099401274571]
  We propose CtrlFlow, a trajectory-level synthetic method using conditional flow matching (CFM)<n>Our method ensures optimal trajectory sampling by minimizing the control energy governed by the non-linear Controllability Gramian Matrix.<n>In online settings, CtrlFlow demonstrates the better performance on common MuJoCo benchmark tasks than dynamics models.
  arXiv  Detail & Related papers  (2025-11-10T08:01:20Z)
• ScaleWeaver: Weaving Efficient Controllable T2I Generation with Multi-Scale Reference Attention [86.93601565563954]
  ScaleWeaver is a framework designed to achieve high-fidelity, controllable generation upon advanced visual autoregressive( VAR) models.<n>The proposed Reference Attention module discards the unnecessary attention from image$rightarrow$condition, reducing computational cost.<n>Experiments show that ScaleWeaver delivers high-quality generation and precise control while attaining superior efficiency over diffusion-based methods.
  arXiv  Detail & Related papers  (2025-10-16T17:00:59Z)
• Invariant Control Strategies for Active Flow Control using Graph Neural Networks [0.0]
  We introduce graph neural networks (GNNs) as a promising architecture forReinforcement Learning (RL)-based flow control.<n>GNNs process unstructured, threedimensional flow data, preserving spatial relationships without the constraints of a Cartesian grid.<n>We show that GNN-based control policies achieve comparable performance to existing methods while benefiting from improved generalization properties.
  arXiv  Detail & Related papers  (2025-03-28T09:33:40Z)
• Provably Efficient Online RLHF with One-Pass Reward Modeling [70.82499103200402]
  Reinforcement Learning from Human Feedback has shown remarkable success in aligning Large Language Models with human preferences.<n>Online RLHF has emerged as a promising direction, enabling iterative data collection and refinement.<n>We propose a one-pass reward modeling method that eliminates the need to store historical data and achieves constant-time updates per iteration.
  arXiv  Detail & Related papers  (2025-02-11T02:36:01Z)
• Data efficient reinforcement learning and adaptive optimal perimeter control of network traffic dynamics [0.0]
  This work proposes an integral reinforcement learning (IRL) based approach to learning the macroscopic traffic dynamics for adaptive optimal perimeter control. To reduce the sampling complexity and use the available data more efficiently, the experience replay (ER) technique is introduced to the IRL algorithm. The convergence of the IRL-based algorithms and the stability of the controlled traffic dynamics are proven via the Lyapunov theory.
  arXiv  Detail & Related papers  (2022-09-13T04:28:49Z)
• Two-step reinforcement learning for model-free redesign of nonlinear optimal regulator [1.5624421399300306]
  Reinforcement learning (RL) is one of the promising approaches that enable model-free redesign of optimal controllers for nonlinear dynamical systems. We propose a model-free two-step design approach that improves the transient learning performance of RL in an optimal regulator redesign problem for unknown nonlinear systems.
  arXiv  Detail & Related papers  (2021-03-05T17:12:33Z)
• Optimization-driven Deep Reinforcement Learning for Robust Beamforming in IRS-assisted Wireless Communications [54.610318402371185]
  Intelligent reflecting surface (IRS) is a promising technology to assist downlink information transmissions from a multi-antenna access point (AP) to a receiver. We minimize the AP's transmit power by a joint optimization of the AP's active beamforming and the IRS's passive beamforming. We propose a deep reinforcement learning (DRL) approach that can adapt the beamforming strategies from past experiences.
  arXiv  Detail & Related papers  (2020-05-25T01:42:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.