Federated Deep Reinforcement Learning for the Distributed Control of
NextG Wireless Networks
- URL: http://arxiv.org/abs/2112.03465v1
- Date: Tue, 7 Dec 2021 03:13:20 GMT
- Title: Federated Deep Reinforcement Learning for the Distributed Control of
NextG Wireless Networks
- Authors: Peyman Tehrani, Francesco Restuccia and Marco Levorato
- Abstract summary: Next Generation (NextG) networks are expected to support demanding internet tactile applications such as augmented reality and connected autonomous vehicles.
Data-driven approaches can improve the ability of the network to adapt to the current operating conditions.
Deep RL (DRL) has been shown to achieve good performance even in complex environments.
- Score: 16.12495409295754
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Next Generation (NextG) networks are expected to support demanding tactile
internet applications such as augmented reality and connected autonomous
vehicles. Whereas recent innovations bring the promise of larger link capacity,
their sensitivity to the environment and erratic performance defy traditional
model-based control rationales. Zero-touch data-driven approaches can improve
the ability of the network to adapt to the current operating conditions. Tools
such as reinforcement learning (RL) algorithms can build optimal control policy
solely based on a history of observations. Specifically, deep RL (DRL), which
uses a deep neural network (DNN) as a predictor, has been shown to achieve good
performance even in complex environments and with high dimensional inputs.
However, the training of DRL models require a large amount of data, which may
limit its adaptability to ever-evolving statistics of the underlying
environment. Moreover, wireless networks are inherently distributed systems,
where centralized DRL approaches would require excessive data exchange, while
fully distributed approaches may result in slower convergence rates and
performance degradation. In this paper, to address these challenges, we propose
a federated learning (FL) approach to DRL, which we refer to federated DRL
(F-DRL), where base stations (BS) collaboratively train the embedded DNN by
only sharing models' weights rather than training data. We evaluate two
distinct versions of F-DRL, value and policy based, and show the superior
performance they achieve compared to distributed and centralized DRL.
Related papers
- A Deep Q-Network Based on Radial Basis Functions for Multi-Echelon
Inventory Management [6.149034764951798]
This paper addresses a multi-echelon inventory management problem with a complex network topology.
It develops a DRL model whose Q-network is based on radial basis functions.
It produces a better policy in the multi-echelon system and competitive performance in the serial system where the base-stock policy is optimal.
arXiv Detail & Related papers (2024-01-29T04:11:56Z) - Hybrid Reinforcement Learning for Optimizing Pump Sustainability in
Real-World Water Distribution Networks [55.591662978280894]
This article addresses the pump-scheduling optimization problem to enhance real-time control of real-world water distribution networks (WDNs)
Our primary objectives are to adhere to physical operational constraints while reducing energy consumption and operational costs.
Traditional optimization techniques, such as evolution-based and genetic algorithms, often fall short due to their lack of convergence guarantees.
arXiv Detail & Related papers (2023-10-13T21:26:16Z) - Digital Twin Assisted Deep Reinforcement Learning for Online Admission
Control in Sliced Network [19.152875040151976]
We propose a digital twin (DT) accelerated DRL solution to address this issue.
A neural network-based DT is established with a customized output layer for queuing systems, trained through supervised learning, and then employed to assist the training phase of the DRL model.
Extensive simulations show that the DT-accelerated DRL improves resource utilization by over 40% compared to the directly trained state-of-the-art dueling deep Q-learning model.
arXiv Detail & Related papers (2023-10-07T09:09:19Z) - How Does Forecasting Affect the Convergence of DRL Techniques in O-RAN
Slicing? [20.344810727033327]
We propose a novel forecasting-aided DRL approach and its respective O-RAN practical deployment workflow to enhance DRL convergence.
Our approach shows up to 22.8%, 86.3%, and 300% improvements in the average initial reward value, convergence rate, and number of converged scenarios respectively.
arXiv Detail & Related papers (2023-09-01T14:30:04Z) - DRL-GAN: A Hybrid Approach for Binary and Multiclass Network Intrusion
Detection [2.7122540465034106]
Intrusion detection systems (IDS) are an essential security technology for detecting these attacks.
We implement a novel hybrid technique using synthetic data produced by a Generative Adversarial Network (GAN) to use as input for training a Deep Reinforcement Learning (DRL) model.
Our findings demonstrate that training the DRL on specific synthetic datasets can result in better performance in correctly classifying minority classes over training on the true imbalanced dataset.
arXiv Detail & Related papers (2023-01-05T19:51:24Z) - On the Robustness of Controlled Deep Reinforcement Learning for Slice
Placement [0.8459686722437155]
We compare two Deep Reinforcement Learning algorithms: a pure DRL-based algorithm and a hybrid DRL as a hybrid DRL-heuristic algorithm.
The evaluation results show that the proposed hybrid DRL-heuristic approach is more robust and reliable in case of unpredictable network load changes than pure DRL.
arXiv Detail & Related papers (2021-08-05T10:24:33Z) - Adaptive Stochastic ADMM for Decentralized Reinforcement Learning in
Edge Industrial IoT [106.83952081124195]
Reinforcement learning (RL) has been widely investigated and shown to be a promising solution for decision-making and optimal control processes.
We propose an adaptive ADMM (asI-ADMM) algorithm and apply it to decentralized RL with edge-computing-empowered IIoT networks.
Experiment results show that our proposed algorithms outperform the state of the art in terms of communication costs and scalability, and can well adapt to complex IoT environments.
arXiv Detail & Related papers (2021-06-30T16:49:07Z) - Behavioral Priors and Dynamics Models: Improving Performance and Domain
Transfer in Offline RL [82.93243616342275]
We introduce Offline Model-based RL with Adaptive Behavioral Priors (MABE)
MABE is based on the finding that dynamics models, which support within-domain generalization, and behavioral priors, which support cross-domain generalization, are complementary.
In experiments that require cross-domain generalization, we find that MABE outperforms prior methods.
arXiv Detail & Related papers (2021-06-16T20:48:49Z) - Instabilities of Offline RL with Pre-Trained Neural Representation [127.89397629569808]
In offline reinforcement learning (RL), we seek to utilize offline data to evaluate (or learn) policies in scenarios where the data are collected from a distribution that substantially differs from that of the target policy to be evaluated.
Recent theoretical advances have shown that such sample-efficient offline RL is indeed possible provided certain strong representational conditions hold.
This work studies these issues from an empirical perspective to gauge how stable offline RL methods are.
arXiv Detail & Related papers (2021-03-08T18:06:44Z) - Reinforcement Learning for Datacenter Congestion Control [50.225885814524304]
Successful congestion control algorithms can dramatically improve latency and overall network throughput.
Until today, no such learning-based algorithms have shown practical potential in this domain.
We devise an RL-based algorithm with the aim of generalizing to different configurations of real-world datacenter networks.
We show that this scheme outperforms alternative popular RL approaches, and generalizes to scenarios that were not seen during training.
arXiv Detail & Related papers (2021-02-18T13:49:28Z) - Deep Learning for Ultra-Reliable and Low-Latency Communications in 6G
Networks [84.2155885234293]
We first summarize how to apply data-driven supervised deep learning and deep reinforcement learning in URLLC.
To address these open problems, we develop a multi-level architecture that enables device intelligence, edge intelligence, and cloud intelligence for URLLC.
arXiv Detail & Related papers (2020-02-22T14:38:11Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.