Digital Twin Calibration with Model-Based Reinforcement Learning

• URL: http://arxiv.org/abs/2501.02205v1
• Date: Sat, 04 Jan 2025 06:15:28 GMT
• Title: Digital Twin Calibration with Model-Based Reinforcement Learning
• Authors: Hua Zheng, Wei Xie, Ilya O. Ryzhov, Keilung Choy,
• Abstract summary: This paper presents a novel methodological framework, called the Actor-Simulator, that incorporates the calibration of digital twins into model-based reinforcement learning. Our approach jointly calibrates the digital twin and searches for an optimal control policy, thus accounting for and reducing model error. This dual-component approach provably converges to the optimal policy, and outperforms existing methods in extensive numerical experiments based on the biopharmaceutical manufacturing domain.
• Score: 3.0435175689911595
• License:
• Abstract: This paper presents a novel methodological framework, called the Actor-Simulator, that incorporates the calibration of digital twins into model-based reinforcement learning for more effective control of stochastic systems with complex nonlinear dynamics. Traditional model-based control often relies on restrictive structural assumptions (such as linear state transitions) and fails to account for parameter uncertainty in the model. These issues become particularly critical in industries such as biopharmaceutical manufacturing, where process dynamics are complex and not fully known, and only a limited amount of data is available. Our approach jointly calibrates the digital twin and searches for an optimal control policy, thus accounting for and reducing model error. We balance exploration and exploitation by using policy performance as a guide for data collection. This dual-component approach provably converges to the optimal policy, and outperforms existing methods in extensive numerical experiments based on the biopharmaceutical manufacturing domain.

Related papers

• Imitation Learning from Observations: An Autoregressive Mixture of Experts Approach [2.4427666827706074]
  This paper presents a novel approach to imitation learning from observations, where an autoregressive mixture of experts model is deployed to fit the underlying policy. The effectiveness of the proposed framework is validated using two autonomous driving datasets collected from human demonstrations.
  arXiv  Detail & Related papers  (2024-11-12T22:56:28Z)
• Learning epidemic trajectories through Kernel Operator Learning: from modelling to optimal control [0.0]
  We discuss the efficacy of Kernel Operator Learning (KOL) to reconstruct population dynamics during epidemic outbreaks. In particular, we introduce two surrogate models, named KOL-m and KOL-$partial$, which reconstruct in two different ways the evolution of the epidemics. We show how the two introduced approaches are suitable for realizing fast and robust forecasts and scenario analyses.
  arXiv  Detail & Related papers  (2024-04-17T07:21:17Z)
• Active Learning for Control-Oriented Identification of Nonlinear Systems [26.231260751633307]
  We present the first finite sample analysis of an active learning algorithm suitable for a general class of nonlinear dynamics. In certain settings, the excess control cost of our algorithm achieves the optimal rate, up to logarithmic factors. We validate our approach in simulation, showcasing the advantage of active, control-oriented exploration for controlling nonlinear systems.
  arXiv  Detail & Related papers  (2024-04-13T15:40:39Z)
• Data-driven Nonlinear Model Reduction using Koopman Theory: Integrated Control Form and NMPC Case Study [56.283944756315066]
  We propose generic model structures combining delay-coordinate encoding of measurements and full-state decoding to integrate reduced Koopman modeling and state estimation. A case study demonstrates that our approach provides accurate control models and enables real-time capable nonlinear model predictive control of a high-purity cryogenic distillation column.
  arXiv  Detail & Related papers  (2024-01-09T11:54:54Z)
• When Parameter-efficient Tuning Meets General-purpose Vision-language Models [65.19127815275307]
  PETAL revolutionizes the training process by requiring only 0.5% of the total parameters, achieved through a unique mode approximation technique. Our experiments reveal that PETAL not only outperforms current state-of-the-art methods in most scenarios but also surpasses full fine-tuning models in effectiveness.
  arXiv  Detail & Related papers  (2023-12-16T17:13:08Z)
• Enhanced multi-fidelity modelling for digital twin and uncertainty quantification [0.0]
  Data-driven models play a crucial role in digital twins, enabling real-time updates and predictions. The fidelity of available data and the scarcity of accurate sensor data often hinder the efficient learning of surrogate models. We propose a novel framework that begins by developing a robust multi-fidelity surrogate model.
  arXiv  Detail & Related papers  (2023-06-26T05:58:17Z)
• Learning-enhanced Nonlinear Model Predictive Control using Knowledge-based Neural Ordinary Differential Equations and Deep Ensembles [5.650647159993238]
  In this work, we leverage deep learning tools, namely knowledge-based neural ordinary differential equations (KNODE) and deep ensembles, to improve the prediction accuracy of a model predictive control (MPC) In particular, we learn an ensemble of KNODE models, which we refer to as the KNODE ensemble, to obtain an accurate prediction of the true system dynamics. We show that the KNODE ensemble provides more accurate predictions and illustrate the efficacy and closed-loop performance of the proposed nonlinear MPC framework.
  arXiv  Detail & Related papers  (2022-11-24T23:51:18Z)
• Causal Inference via Nonlinear Variable Decorrelation for Healthcare Applications [60.26261850082012]
  We introduce a novel method with a variable decorrelation regularizer to handle both linear and nonlinear confounding. We employ association rules as new representations using association rule mining based on the original features to increase model interpretability.
  arXiv  Detail & Related papers  (2022-09-29T17:44:14Z)
• Probabilistic robust linear quadratic regulators with Gaussian processes [73.0364959221845]
  Probabilistic models such as Gaussian processes (GPs) are powerful tools to learn unknown dynamical systems from data for subsequent use in control design. We present a novel controller synthesis for linearized GP dynamics that yields robust controllers with respect to a probabilistic stability margin.
  arXiv  Detail & Related papers  (2021-05-17T08:36:18Z)
• COMBO: Conservative Offline Model-Based Policy Optimization [120.55713363569845]
  Uncertainty estimation with complex models, such as deep neural networks, can be difficult and unreliable. We develop a new model-based offline RL algorithm, COMBO, that regularizes the value function on out-of-support state-actions. We find that COMBO consistently performs as well or better as compared to prior offline model-free and model-based methods.
  arXiv  Detail & Related papers  (2021-02-16T18:50:32Z)
• How Training Data Impacts Performance in Learning-based Control [67.7875109298865]
  This paper derives an analytical relationship between the density of the training data and the control performance. We formulate a quality measure for the data set, which we refer to as $rho$-gap. We show how the $rho$-gap can be applied to a feedback linearizing control law.
  arXiv  Detail & Related papers  (2020-05-25T12:13:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.