Beyond PID Controllers: PPO with Neuralized PID Policy for Proton Beam Intensity Control in Mu2e

• URL: http://arxiv.org/abs/2312.17372v1
• Date: Thu, 28 Dec 2023 21:35:20 GMT
• Title: Beyond PID Controllers: PPO with Neuralized PID Policy for Proton Beam Intensity Control in Mu2e
• Authors: Chenwei Xu, Jerry Yao-Chieh Hu, Aakaash Narayanan, Mattson Thieme, Vladimir Nagaslaev, Mark Austin, Jeremy Arnold, Jose Berlioz, Pierrick Hanlet, Aisha Ibrahim, Dennis Nicklaus, Jovan Mitrevski, Jason Michael St.John, Gauri Pradhan, Andrea Saewert, Kiyomi Seiya, Brian Schupbach, Randy Thurman-Keup, Nhan Tran, Rui Shi, Seda Ogrenci, Alexis Maya-Isabelle Shuping, Kyle Hazelwood and Han Liu
• Abstract summary: We introduce a novel Proximal Policy Optimization (PPO) algorithm aimed at maintaining a uniform proton beam intensity delivery in the Muon to Electron Conversion Experiment (Mu2e) at Fermi National Accelerator Laboratory (Fermilab) Our primary objective is to regulate the spill process to ensure a consistent intensity profile, with the ultimate goal of creating an automated controller capable of providing real-time feedback and calibration of the Spill Regulation System (SRS) parameters on a millisecond timescale.
• Score: 3.860979702631594
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce a novel Proximal Policy Optimization (PPO) algorithm aimed at addressing the challenge of maintaining a uniform proton beam intensity delivery in the Muon to Electron Conversion Experiment (Mu2e) at Fermi National Accelerator Laboratory (Fermilab). Our primary objective is to regulate the spill process to ensure a consistent intensity profile, with the ultimate goal of creating an automated controller capable of providing real-time feedback and calibration of the Spill Regulation System (SRS) parameters on a millisecond timescale. We treat the Mu2e accelerator system as a Markov Decision Process suitable for Reinforcement Learning (RL), utilizing PPO to reduce bias and enhance training stability. A key innovation in our approach is the integration of a neuralized Proportional-Integral-Derivative (PID) controller into the policy function, resulting in a significant improvement in the Spill Duty Factor (SDF) by 13.6%, surpassing the performance of the current PID controller baseline by an additional 1.6%. This paper presents the preliminary offline results based on a differentiable simulator of the Mu2e accelerator. It paves the groundwork for real-time implementations and applications, representing a crucial step towards automated proton beam intensity control for the Mu2e experiment.

Related papers

• One-Step Diffusion Policy: Fast Visuomotor Policies via Diffusion Distillation [80.71541671907426]
  OneStep Diffusion Policy (OneDP) is a novel approach that distills knowledge from pre-trained diffusion policies into a single-step action generator. OneDP significantly accelerates response times for robotic control tasks.
  arXiv  Detail & Related papers  (2024-10-28T17:54:31Z)
• Multistep Criticality Search and Power Shaping in Microreactors with Reinforcement Learning [0.3562485774739681]
  We introduce the use of reinforcement learning (RL) algorithms for intelligent control in nuclear microreactors. RL agent is trained using proximal policy optimization (PPO) and advantage actor-critic (A2C) Results demonstrate the excellent performance of PPO in identifying optimal drum positions.
  arXiv  Detail & Related papers  (2024-06-22T20:14:56Z)
• Parameter-Adaptive Approximate MPC: Tuning Neural-Network Controllers without Retraining [50.00291020618743]
  This work introduces a novel, parameter-adaptive AMPC architecture capable of online tuning without recomputing large datasets and retraining. We showcase the effectiveness of parameter-adaptive AMPC by controlling the swing-ups of two different real cartpole systems with a severely resource-constrained microcontroller (MCU) Taken together, these contributions represent a marked step toward the practical application of AMPC in real-world systems.
  arXiv  Detail & Related papers  (2024-04-08T20:02:19Z)
• Real-Time Adaptive Safety-Critical Control with Gaussian Processes in High-Order Uncertain Models [14.790031018404942]
  This paper presents an adaptive online learning framework for systems with uncertain parameters. We first integrate a forgetting factor to refine a variational sparse GP algorithm. In the second phase, we propose a safety filter based on high-order control barrier functions.
  arXiv  Detail & Related papers  (2024-02-29T08:25:32Z)
• Self-Tuning PID Control via a Hybrid Actor-Critic-Based Neural Structure for Quadcopter Control [0.0]
  Proportional-Integrator-Derivative (PID) controller is used in a wide range of industrial and experimental processes. Due to the uncertainty of model parameters and external disturbances, real systems such as Quadrotors need more robust and reliable PID controllers. In this research, a self-tuning PID controller using a Reinforcement-Learning-based Neural Network has been investigated.
  arXiv  Detail & Related papers  (2023-07-03T19:35:52Z)
• TempoRL: laser pulse temporal shape optimization with Deep Reinforcement Learning [0.577478614918139]
  High Power Laser's (HPL) optimal performance is essential for the success of a wide variety of experimental tasks related to light-matter interactions. Traditionally, HPL parameters are optimised in an automated fashion relying on black-box numerical methods. Model-free Deep Reinforcement Learning (DRL) offers a promising alternative framework for optimising HPL performance.
  arXiv  Detail & Related papers  (2023-04-20T22:15:27Z)
• Real-Time Model-Free Deep Reinforcement Learning for Force Control of a Series Elastic Actuator [56.11574814802912]
  State-of-the art robotic applications utilize series elastic actuators (SEAs) with closed-loop force control to achieve complex tasks such as walking, lifting, and manipulation. Model-free PID control methods are more prone to instability due to nonlinearities in the SEA. Deep reinforcement learning has proved to be an effective model-free method for continuous control tasks.
  arXiv  Detail & Related papers  (2023-04-11T00:51:47Z)
• Performance-Driven Controller Tuning via Derivative-Free Reinforcement Learning [6.5158195776494]
  We tackle the controller tuning problem using a novel derivative-free reinforcement learning framework. We conduct numerical experiments on two concrete examples from autonomous driving, namely, adaptive cruise control with PID controller and trajectory tracking with MPC controller. Experimental results show that the proposed method outperforms popular baselines and highlight its strong potential for controller tuning.
  arXiv  Detail & Related papers  (2022-09-11T13:01:14Z)
• Collaborative Intelligent Reflecting Surface Networks with Multi-Agent Reinforcement Learning [63.83425382922157]
  Intelligent reflecting surface (IRS) is envisioned to be widely applied in future wireless networks. In this paper, we investigate a multi-user communication system assisted by cooperative IRS devices with the capability of energy harvesting.
  arXiv  Detail & Related papers  (2022-03-26T20:37:14Z)
• Gaussian Process-based Min-norm Stabilizing Controller for Control-Affine Systems with Uncertain Input Effects and Dynamics [90.81186513537777]
  We propose a novel compound kernel that captures the control-affine nature of the problem. We show that this resulting optimization problem is convex, and we call it Gaussian Process-based Control Lyapunov Function Second-Order Cone Program (GP-CLF-SOCP)
  arXiv  Detail & Related papers  (2020-11-14T01:27:32Z)
• Optimization-driven Deep Reinforcement Learning for Robust Beamforming in IRS-assisted Wireless Communications [54.610318402371185]
  Intelligent reflecting surface (IRS) is a promising technology to assist downlink information transmissions from a multi-antenna access point (AP) to a receiver. We minimize the AP's transmit power by a joint optimization of the AP's active beamforming and the IRS's passive beamforming. We propose a deep reinforcement learning (DRL) approach that can adapt the beamforming strategies from past experiences.
  arXiv  Detail & Related papers  (2020-05-25T01:42:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.