Towards Safe Load Balancing based on Control Barrier Functions and Deep Reinforcement Learning

• URL: http://arxiv.org/abs/2401.05525v1
• Date: Wed, 10 Jan 2024 19:43:12 GMT
• Title: Towards Safe Load Balancing based on Control Barrier Functions and Deep Reinforcement Learning
• Authors: Lam Dinh, Pham Tran Anh Quang, J\'er\'emie Leguay
• Abstract summary: We propose a safe learning-based load balancing algorithm for Software Defined-Wide Area Network (SD-WAN) It is empowered by Deep Reinforcement Learning (DRL) combined with a Control Barrier Function (CBF) We show that our approach delivers near-optimal Quality-of-Service (QoS) in terms of end-to-end delay while respecting safety requirements related to link capacity constraints.
• Score: 0.691367883100748
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Deep Reinforcement Learning (DRL) algorithms have recently made significant strides in improving network performance. Nonetheless, their practical use is still limited in the absence of safe exploration and safe decision-making. In the context of commercial solutions, reliable and safe-to-operate systems are of paramount importance. Taking this problem into account, we propose a safe learning-based load balancing algorithm for Software Defined-Wide Area Network (SD-WAN), which is empowered by Deep Reinforcement Learning (DRL) combined with a Control Barrier Function (CBF). It safely projects unsafe actions into feasible ones during both training and testing, and it guides learning towards safe policies. We successfully implemented the solution on GPU to accelerate training by approximately 110x times and achieve model updates for on-policy methods within a few seconds, making the solution practical. We show that our approach delivers near-optimal Quality-of-Service (QoS performance in terms of end-to-end delay while respecting safety requirements related to link capacity constraints. We also demonstrated that on-policy learning based on Proximal Policy Optimization (PPO) performs better than off-policy learning with Deep Deterministic Policy Gradient (DDPG) when both are combined with a CBF for safe load balancing.

Related papers

• Safe Load Balancing in Software-Defined-Networking [1.2521494095948067]
  Control Barrier (CBF) designed on top of Deep Reinforcement Learning (DRL) algorithms for load-balancing. We show that our DRL-CBF approach is capable of meeting safety requirements during training and testing.
  arXiv  Detail & Related papers  (2024-10-22T09:34:22Z)
• Safety Correction from Baseline: Towards the Risk-aware Policy in Robotics via Dual-agent Reinforcement Learning [64.11013095004786]
  We propose a dual-agent safe reinforcement learning strategy consisting of a baseline and a safe agent. Such a decoupled framework enables high flexibility, data efficiency and risk-awareness for RL-based control. The proposed method outperforms the state-of-the-art safe RL algorithms on difficult robot locomotion and manipulation tasks.
  arXiv  Detail & Related papers  (2022-12-14T03:11:25Z)
• Evaluating Model-free Reinforcement Learning toward Safety-critical Tasks [70.76757529955577]
  This paper revisits prior work in this scope from the perspective of state-wise safe RL. We propose Unrolling Safety Layer (USL), a joint method that combines safety optimization and safety projection. To facilitate further research in this area, we reproduce related algorithms in a unified pipeline and incorporate them into SafeRL-Kit.
  arXiv  Detail & Related papers  (2022-12-12T06:30:17Z)
• SAFER: Data-Efficient and Safe Reinforcement Learning via Skill Acquisition [59.94644674087599]
  We propose SAFEty skill pRiors (SAFER), an algorithm that accelerates policy learning on complex control tasks under safety constraints. Through principled training on an offline dataset, SAFER learns to extract safe primitive skills. In the inference stage, policies trained with SAFER learn to compose safe skills into successful policies.
  arXiv  Detail & Related papers  (2022-02-10T05:43:41Z)
• Model-Based Safe Reinforcement Learning with Time-Varying State and Control Constraints: An Application to Intelligent Vehicles [13.40143623056186]
  This paper proposes a safe RL algorithm for optimal control of nonlinear systems with time-varying state and control constraints. A multi-step policy evaluation mechanism is proposed to predict the policy's safety risk under time-varying safety constraints and guide the policy to update safely. The proposed algorithm outperforms several state-of-the-art RL algorithms in the simulated Safety Gym environment.
  arXiv  Detail & Related papers  (2021-12-18T10:45:31Z)
• Improving Safety in Deep Reinforcement Learning using Unsupervised Action Planning [4.2955354157580325]
  One of the key challenges to deep reinforcement learning (deep RL) is to ensure safety at both training and testing phases. We propose a novel technique of unsupervised action planning to improve the safety of on-policy reinforcement learning algorithms. Our results show that the proposed safety RL algorithm can achieve higher rewards compared with multiple baselines in both discrete and continuous control problems.
  arXiv  Detail & Related papers  (2021-09-29T10:26:29Z)
• Safe Reinforcement Learning Using Advantage-Based Intervention [45.79740561754542]
  Many sequential decision problems involve finding a policy that maximizes total reward while obeying safety constraints. We propose a new algorithm, SAILR, that uses an intervention mechanism based on advantage functions to keep the agent safe throughout training. Our method comes with strong guarantees on safety during both training and deployment.
  arXiv  Detail & Related papers  (2021-06-16T20:28:56Z)
• Constrained Markov Decision Processes via Backward Value Functions [43.649330976089004]
  We model the problem of learning with constraints as a Constrained Markov Decision Process. A key contribution of our approach is to translate cumulative cost constraints into state-based constraints. We provide theoretical guarantees under which the agent converges while ensuring safety over the course of training.
  arXiv  Detail & Related papers  (2020-08-26T20:56:16Z)
• Chance-Constrained Trajectory Optimization for Safe Exploration and Learning of Nonlinear Systems [81.7983463275447]
  Learning-based control algorithms require data collection with abundant supervision for training. We present a new approach for optimal motion planning with safe exploration that integrates chance-constrained optimal control with dynamics learning and feedback control.
  arXiv  Detail & Related papers  (2020-05-09T05:57:43Z)
• Cautious Reinforcement Learning with Logical Constraints [78.96597639789279]
  An adaptive safe padding forces Reinforcement Learning (RL) to synthesise optimal control policies while ensuring safety during the learning process. Theoretical guarantees are available on the optimality of the synthesised policies and on the convergence of the learning algorithm.
  arXiv  Detail & Related papers  (2020-02-26T00:01:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.