Reinforcement Learning for Control of Valves

• URL: http://arxiv.org/abs/2012.14668v2
• Date: Thu, 4 Feb 2021 11:49:57 GMT
• Title: Reinforcement Learning for Control of Valves
• Authors: Rajesh Siraskar
• Abstract summary: This paper is a study of reinforcement learning (RL) as an optimal-control strategy for control of nonlinear valves. It is evaluated against the PID (proportional-integral-derivative) strategy, using a unified framework.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper is a study of reinforcement learning (RL) as an optimal-control strategy for control of nonlinear valves. It is evaluated against the PID (proportional-integral-derivative) strategy, using a unified framework. RL is an autonomous learning mechanism that learns by interacting with its environment. It is gaining increasing attention in the world of control systems as a means of building optimal-controllers for challenging dynamic and nonlinear processes. Published RL research often uses open-source tools (Python and OpenAI Gym environments). We use MATLAB's recently launched (R2019a) Reinforcement Learning Toolbox to develop the valve controller; trained using the DDPG (Deep Deterministic Policy-Gradient) algorithm and Simulink to simulate the nonlinear valve and create the experimental test-bench for evaluation. Simulink allows industrial engineers to quickly adapt and experiment with other systems of their choice. Results indicate that the RL controller is extremely good at tracking the signal with speed and produces a lower error with respect to the reference signal. The PID, however, is better at disturbance rejection and hence provides a longer life for the valves. Successful machine learning involves tuning many hyperparameters requiring significant investment of time and efforts. We introduce "Graded Learning" as a simplified, application oriented adaptation of the more formal and algorithmic "Curriculum for Reinforcement Learning". It is shown via experiments that it helps converge the learning task of complex non-linear real world systems. Finally, experiential learnings gained from this research are corroborated against published research.

Related papers

• Autonomous Vehicle Controllers From End-to-End Differentiable Simulation [60.05963742334746]
  We propose a differentiable simulator and design an analytic policy gradients (APG) approach to training AV controllers. Our proposed framework brings the differentiable simulator into an end-to-end training loop, where gradients of environment dynamics serve as a useful prior to help the agent learn a more grounded policy. We find significant improvements in performance and robustness to noise in the dynamics, as well as overall more intuitive human-like handling.
  arXiv  Detail & Related papers  (2024-09-12T11:50:06Z)
• Laboratory Experiments of Model-based Reinforcement Learning for Adaptive Optics Control [0.565395466029518]
  We implement and adapt an RL method called Policy Optimization for AO (PO4AO) to the GHOST test bench at ESO headquarters. We study the predictive and self-calibrating aspects of the method. New implementation on GHOST running PyTorch introduces only around 700 microseconds in addition to hardware, pipeline, and Python interface latency.
  arXiv  Detail & Related papers  (2023-12-30T14:11:43Z)
• Learning to Fly in Seconds [7.259696592534715]
  We show how curriculum learning and a highly optimized simulator enhance sample complexity and lead to fast training times. Our framework enables Simulation-to-Reality (Sim2Real) transfer for direct control after only 18 seconds of training on a consumer-grade laptop.
  arXiv  Detail & Related papers  (2023-11-22T01:06:45Z)
• Tracking Control for a Spherical Pendulum via Curriculum Reinforcement Learning [27.73555826776087]
  Reinforcement Learning (RL) allows learning non-trivial robot control laws purely from data. In this paper, we pair a recent algorithm for automatically building curricula with RL on massively parallelized simulations. We demonstrate the potential of curriculum RL to jointly learn state estimation and control for non-linear tracking tasks.
  arXiv  Detail & Related papers  (2023-09-25T12:48:47Z)
• Meta Reinforcement Learning for Adaptive Control: An Offline Approach [3.131740922192114]
  We formulate a meta reinforcement learning (meta-RL) control strategy that takes advantage of known, offline information for training. Our meta-RL agent has a recurrent structure that accumulates "context" for its current dynamics through a hidden state variable. In tests reported here, the meta-RL agent was trained entirely offline, yet produced excellent results in novel settings.
  arXiv  Detail & Related papers  (2022-03-17T23:58:52Z)
• RvS: What is Essential for Offline RL via Supervised Learning? [77.91045677562802]
  Recent work has shown that supervised learning alone, without temporal difference (TD) learning, can be remarkably effective for offline RL. In every environment suite we consider simply maximizing likelihood with two-layer feedforward is competitive. They also probe the limits of existing RvS methods, which are comparatively weak on random data.
  arXiv  Detail & Related papers  (2021-12-20T18:55:16Z)
• On the Theory of Reinforcement Learning with Once-per-Episode Feedback [120.5537226120512]
  We introduce a theory of reinforcement learning in which the learner receives feedback only once at the end of an episode. This is arguably more representative of real-world applications than the traditional requirement that the learner receive feedback at every time step.
  arXiv  Detail & Related papers  (2021-05-29T19:48:51Z)
• Learning to Fly -- a Gym Environment with PyBullet Physics for Reinforcement Learning of Multi-agent Quadcopter Control [0.0]
  We propose an open-source environment for multiple quadcopters based on the Bullet physics engine. Its multi-agent and vision based reinforcement learning interfaces, as well as the support of realistic collisions and aerodynamic effects, make it, to the best of our knowledge, a first of its kind.
  arXiv  Detail & Related papers  (2021-03-03T02:47:59Z)
• Learning Dexterous Manipulation from Suboptimal Experts [69.8017067648129]
  Relative Entropy Q-Learning (REQ) is a simple policy algorithm that combines ideas from successful offline and conventional RL algorithms. We show how REQ is also effective for general off-policy RL, offline RL, and RL from demonstrations.
  arXiv  Detail & Related papers  (2020-10-16T18:48:49Z)
• Discovering Reinforcement Learning Algorithms [53.72358280495428]
  Reinforcement learning algorithms update an agent's parameters according to one of several possible rules. This paper introduces a new meta-learning approach that discovers an entire update rule. It includes both 'what to predict' (e.g. value functions) and 'how to learn from it' by interacting with a set of environments.
  arXiv  Detail & Related papers  (2020-07-17T07:38:39Z)
• Information Theoretic Model Predictive Q-Learning [64.74041985237105]
  We present a novel theoretical connection between information theoretic MPC and entropy regularized RL. We develop a Q-learning algorithm that can leverage biased models.
  arXiv  Detail & Related papers  (2019-12-31T00:29:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.