Policy Optimization for PDE Control with a Warm Start

• URL: http://arxiv.org/abs/2403.01005v1
• Date: Fri, 1 Mar 2024 22:03:22 GMT
• Title: Policy Optimization for PDE Control with a Warm Start
• Authors: Xiangyuan Zhang, Saviz Mowlavi, Mouhacine Benosman, Tamer Ba\c{s}ar
• Abstract summary: Dimensionality reduction is crucial for controlling nonlinear partial differential equations (PDE) We augment the reduce-then-design procedure with a policy optimization step to compensate for the modeling error from dimensionality reduction. Our approach offers a cost-effective alternative to PDE control using end-to-end reinforcement learning.
• Score: 3.0811185425377743
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Dimensionality reduction is crucial for controlling nonlinear partial differential equations (PDE) through a "reduce-then-design" strategy, which identifies a reduced-order model and then implements model-based control solutions. However, inaccuracies in the reduced-order modeling can substantially degrade controller performance, especially in PDEs with chaotic behavior. To address this issue, we augment the reduce-then-design procedure with a policy optimization (PO) step. The PO step fine-tunes the model-based controller to compensate for the modeling error from dimensionality reduction. This augmentation shifts the overall strategy into reduce-then-design-then-adapt, where the model-based controller serves as a warm start for PO. Specifically, we study the state-feedback tracking control of PDEs that aims to align the PDE state with a specific constant target subject to a linear-quadratic cost. Through extensive experiments, we show that a few iterations of PO can significantly improve the model-based controller performance. Our approach offers a cost-effective alternative to PDE control using end-to-end reinforcement learning.

Related papers

• PDE-aware Optimizer for Physics-informed Neural Networks [0.0]
  We propose a PDE-aware that adapts parameter updates based on the variance of per-sample PDE residual.<n>This method addresses gradient misalignment without incurring the heavy computational costs of second-order gradients such as SOAP.<n>Our results demonstrate the effectiveness of PDE residual-aware adaptivity in enhancing stability in PINNs training.
  arXiv  Detail & Related papers  (2025-07-10T19:07:55Z)
• Physics-Informed Neural Networks for Control of Single-Phase Flow Systems Governed by Partial Differential Equations [4.776073133338117]
  We extend the Physics-Informed Neural Nets for Control (PINC) framework to integrate neural networks with physical conservation laws.<n>The PINC model for PDEs is structured into two stages: a steady-state network, which learns equilibrium solutions for a wide range of control inputs, and a transient network, which captures dynamic responses under time-varying boundary conditions.<n>We validate our approach through numerical experiments, demonstrating that the PINC model, which is trained exclusively using physical laws, accurately represents flow dynamics and enables real-time control applications.
  arXiv  Detail & Related papers  (2025-06-06T15:50:19Z)
• Predictive Lagrangian Optimization for Constrained Reinforcement Learning [15.082498910832529]
  Constrained optimization is popularly seen in reinforcement learning for addressing complex control tasks. In this paper, we propose a more generic equivalence framework to build the connection between constrained optimization and feedback control system.
  arXiv  Detail & Related papers  (2025-01-25T13:39:45Z)
• Bridging Autoencoders and Dynamic Mode Decomposition for Reduced-order Modeling and Control of PDEs [12.204795159651589]
  This paper explores a deep autocodingen learning method for reduced-order modeling and control of dynamical systems governed by Ptemporals. We first show that an objective for learning a linear autoen reduced-order model can be formulated to yield a solution closely resembling the result obtained through the dynamic mode decomposition with control algorithm. We then extend this linear autoencoding architecture to a deep autocoding framework, enabling the development of a nonlinear reduced-order model.
  arXiv  Detail & Related papers  (2024-09-09T22:56:40Z)
• Real-time optimal control of high-dimensional parametrized systems by deep learning-based reduced order models [3.5161229331588095]
  We propose a non-intrusive Deep Learning-based Reduced Order Modeling (DL-ROM) technique for the rapid control of systems described in terms of parametrized PDEs in multiple scenarios. After (i) data generation, (ii) dimensionality reduction, and (iii) neural networks training in the offline phase, optimal control strategies can be rapidly retrieved in an online phase for any scenario of interest.
  arXiv  Detail & Related papers  (2024-09-09T15:20:24Z)
• Adaptive Neural-Operator Backstepping Control of a Benchmark Hyperbolic PDE [3.3044728148521623]
  We present the first result on applying NOs in adaptive PDE control, presented for a benchmark 1-D hyperbolic PDE with recirculation. We also present numerical simulations demonstrating stability and observe speedups up to three orders of magnitude.
  arXiv  Detail & Related papers  (2024-01-15T17:52:15Z)
• Model-Based Reparameterization Policy Gradient Methods: Theory and Practical Algorithms [88.74308282658133]
  Reization (RP) Policy Gradient Methods (PGMs) have been widely adopted for continuous control tasks in robotics and computer graphics. Recent studies have revealed that, when applied to long-term reinforcement learning problems, model-based RP PGMs may experience chaotic and non-smooth optimization landscapes. We propose a spectral normalization method to mitigate the exploding variance issue caused by long model unrolls.
  arXiv  Detail & Related papers  (2023-10-30T18:43:21Z)
• Predictable MDP Abstraction for Unsupervised Model-Based RL [93.91375268580806]
  We propose predictable MDP abstraction (PMA) Instead of training a predictive model on the original MDP, we train a model on a transformed MDP with a learned action space. We theoretically analyze PMA and empirically demonstrate that PMA leads to significant improvements over prior unsupervised model-based RL approaches.
  arXiv  Detail & Related papers  (2023-02-08T07:37:51Z)
• Neural Control of Parametric Solutions for High-dimensional Evolution PDEs [6.649496716171139]
  We develop a novel computational framework to approximate solution operators of evolution partial differential equations (PDEs) We propose to approximate the solution operator of the PDE by learning the control vector field in the parameter space. This allows for substantially reduced computational cost to solve the evolution PDE with arbitrary initial conditions.
  arXiv  Detail & Related papers  (2023-01-31T19:26:25Z)
• Solving PDE-constrained Control Problems Using Operator Learning [14.30832827446317]
  We introduce surrogate models for PDE solution operators with special regularizers. Our framework can be applied to both data-driven and data-free cases.
  arXiv  Detail & Related papers  (2021-11-09T03:41:55Z)
• Evaluating model-based planning and planner amortization for continuous control [79.49319308600228]
  We take a hybrid approach, combining model predictive control (MPC) with a learned model and model-free policy learning. We find that well-tuned model-free agents are strong baselines even for high DoF control problems. We show that it is possible to distil a model-based planner into a policy that amortizes the planning without any loss of performance.
  arXiv  Detail & Related papers  (2021-10-07T12:00:40Z)
• Regret-optimal Estimation and Control [52.28457815067461]
  We show that the regret-optimal estimator and regret-optimal controller can be derived in state-space form. We propose regret-optimal analogs of Model-Predictive Control (MPC) and the Extended KalmanFilter (EKF) for systems with nonlinear dynamics.
  arXiv  Detail & Related papers  (2021-06-22T23:14:21Z)
• Gaussian Process-based Min-norm Stabilizing Controller for Control-Affine Systems with Uncertain Input Effects and Dynamics [90.81186513537777]
  We propose a novel compound kernel that captures the control-affine nature of the problem. We show that this resulting optimization problem is convex, and we call it Gaussian Process-based Control Lyapunov Function Second-Order Cone Program (GP-CLF-SOCP)
  arXiv  Detail & Related papers  (2020-11-14T01:27:32Z)
• Optimization-driven Deep Reinforcement Learning for Robust Beamforming in IRS-assisted Wireless Communications [54.610318402371185]
  Intelligent reflecting surface (IRS) is a promising technology to assist downlink information transmissions from a multi-antenna access point (AP) to a receiver. We minimize the AP's transmit power by a joint optimization of the AP's active beamforming and the IRS's passive beamforming. We propose a deep reinforcement learning (DRL) approach that can adapt the beamforming strategies from past experiences.
  arXiv  Detail & Related papers  (2020-05-25T01:42:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.