Towards Practical Operation of Deep Reinforcement Learning Agents in Real-World Network Management at Open RAN Edges

• URL: http://arxiv.org/abs/2410.23086v2
• Date: Fri, 18 Jul 2025 16:19:31 GMT
• Title: Towards Practical Operation of Deep Reinforcement Learning Agents in Real-World Network Management at Open RAN Edges
• Authors: Haiyuan Li, Hari Madhukumar, Peizheng Li, Yuelin Liu, Yiran Teng, Yulei Wu, Ning Wang, Shuangyi Yan, Dimitra Simeonidou,
• Abstract summary: Deep Reinforcement Learning (DRL) has emerged as a powerful solution for meeting the growing demands for connectivity, reliability, low latency and operational efficiency in advanced networks.<n>We first present an orchestration framework that integrates ETSI Multi-access Edge Computing (MEC) with Open RAN, enabling seamless adoption of DRL-based strategies across different time scales.<n>We then identify three critical challenges hindering DRL's real-world deployment, including (1) asynchronous requests from unpredictable or bursty traffic, (2) adaptability and generalization across heterogeneous topologies and evolving service demands, and (3) prolonged convergence and service interruptions due to exploration in
• Score: 5.345501810244355
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep Reinforcement Learning (DRL) has emerged as a powerful solution for meeting the growing demands for connectivity, reliability, low latency and operational efficiency in advanced networks. However, most research has focused on theoretical analysis and simulations, with limited investigation into real-world deployment. To bridge the gap and support practical DRL deployment for network management, we first present an orchestration framework that integrates ETSI Multi-access Edge Computing (MEC) with Open RAN, enabling seamless adoption of DRL-based strategies across different time scales while enhancing agent lifecycle management. We then identify three critical challenges hindering DRL's real-world deployment, including (1) asynchronous requests from unpredictable or bursty traffic, (2) adaptability and generalization across heterogeneous topologies and evolving service demands, and (3) prolonged convergence and service interruptions due to exploration in live operational environments. To address these challenges, we propose a three-fold solution strategy: (a) advanced time-series integration for handling asynchronized traffic, (b) flexible architecture design such as multi-agent DRL and incremental learning to support heterogeneous scenarios, and (c) simulation-driven deployment with transfer learning to reduce convergence time and service disruptions. Lastly, the feasibility of the MEC-O-RAN architecture is validated on an urban-wide testing infrastructure, and two real-world use cases are presented, showcasing the three identified challenges and demonstrating the effectiveness of the proposed solutions.

Related papers

• Multi-Agent Reinforcement Learning for Sample-Efficient Deep Neural Network Mapping [54.65536245955678]
  We present a decentralized multi-agent reinforcement learning (MARL) framework designed to overcome the challenge of sample inefficiency.<n>We introduce an agent clustering algorithm that assigns similar mapping parameters to the same agents based on correlation analysis.<n> Experimental results show our MARL approach improves sample efficiency by 30-300x over standard single-agent RL.
  arXiv  Detail & Related papers  (2025-07-22T05:51:07Z)
• Offline and Distributional Reinforcement Learning for Wireless Communications [5.771885923067511]
  Traditional online reinforcement learning (RL) and deep RL methods face limitations in real-time wireless networks. We focus on offline and distributional RL, two advanced RL techniques that can overcome these challenges. We introduce a novel framework that combines offline and distributional RL for wireless communication applications.
  arXiv  Detail & Related papers  (2025-04-04T09:24:39Z)
• Iterative Multi-Agent Reinforcement Learning: A Novel Approach Toward Real-World Multi-Echelon Inventory Optimization [0.6990493129893112]
  Multi-echelon inventory optimization (MEIO) is critical for effective supply chain management, but its inherent complexity can pose significant challenges.<n>Recent research has found deep reinforcement learning (DRL) to be a promising alternative to traditional reinforcement learning.<n>This thesis investigates DRL's applicability to MEIO problems of increasing complexity.
  arXiv  Detail & Related papers  (2025-03-23T20:52:21Z)
• DeepSeek-Inspired Exploration of RL-based LLMs and Synergy with Wireless Networks: A Survey [62.697565282841026]
  Reinforcement learning (RL)-based large language models (LLMs) have gained significant attention. Wireless networks require the empowerment of RL-based LLMs. Wireless networks provide a vital infrastructure for the efficient training, deployment, and distributed inference of RL-based LLMs.
  arXiv  Detail & Related papers  (2025-03-13T01:59:11Z)
• Improve the Training Efficiency of DRL for Wireless Communication Resource Allocation: The Role of Generative Diffusion Models [2.702550149035333]
  We propose Diffusion-based Deep Reinforcement Learning (D2RL) to overcome fundamental DRL training bottlenecks for wireless networks.<n>D2RL achieves faster convergence and reduced computational costs over conventional DRL methods for resource allocation in wireless communications.<n>This work underscores the transformative potential of GDMs in overcoming fundamental DRL training bottlenecks for wireless networks.
  arXiv  Detail & Related papers  (2025-02-11T03:09:45Z)
• Achieving Network Resilience through Graph Neural Network-enabled Deep Reinforcement Learning [64.20847540439318]
  Deep reinforcement learning (DRL) has been widely used in many important tasks of communication networks. Some studies have combined graph neural networks (GNNs) with DRL, which use the GNNs to extract unstructured features of the network. This paper explores the solution of combining GNNs with DRL to build a resilient network.
  arXiv  Detail & Related papers  (2025-01-19T15:22:17Z)
• Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Review [50.67937325077047]
  This paper is devoted to a comprehensive review of realizing the sample efficiency and generalization of RL algorithms through transfer and inverse reinforcement learning (T-IRL) Our findings denote that a majority of recent research works have dealt with the aforementioned challenges by utilizing human-in-the-loop and sim-to-real strategies. Under the IRL structure, training schemes that require a low number of experience transitions and extension of such frameworks to multi-agent and multi-intention problems have been the priority of researchers in recent years.
  arXiv  Detail & Related papers  (2024-11-15T15:18:57Z)
• Dynamic Spectrum Access for Ambient Backscatter Communication-assisted D2D Systems with Quantum Reinforcement Learning [68.63990729719369]
  The wireless spectrum is becoming scarce, resulting in low spectral efficiency for D2D communications. This paper aims to integrate the ambient backscatter communication technology into D2D devices to allow them to backscatter ambient RF signals. We develop a novel quantum reinforcement learning (RL) algorithm that can achieve a faster convergence rate with fewer training parameters.
  arXiv  Detail & Related papers  (2024-10-23T15:36:43Z)
• D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learning [99.33607114541861]
  We propose a new benchmark for offline RL that focuses on realistic simulations of robotic manipulation and locomotion environments. Our proposed benchmark covers state-based and image-based domains, and supports both offline RL and online fine-tuning evaluation.
  arXiv  Detail & Related papers  (2024-08-15T22:27:00Z)
• Toward Enhanced Reinforcement Learning-Based Resource Management via Digital Twin: Opportunities, Applications, and Challenges [40.73920295596231]
  This article presents a digital twin (DT)-enhanced reinforcement learning (RL) framework aimed at optimizing performance and reliability in network resource management. To deal with the challenges, a comprehensive DT-based framework is proposed to enhance the convergence speed and performance for unified RL-based resource management. The proposed framework provides safe action exploration, more accurate estimates of long-term returns, faster training convergence, higher convergence performance, and real-time adaptation to varying network conditions.
  arXiv  Detail & Related papers  (2024-06-12T04:14:24Z)
• Closed-form congestion control via deep symbolic regression [1.5961908901525192]
  Reinforcement Learning (RL) algorithms can handle challenges in ultra-low-latency and high throughput scenarios. The adoption of neural network models in real deployments still poses some challenges regarding real-time inference and interpretability. This paper proposes a methodology to deal with such challenges while maintaining the performance and generalization capabilities.
  arXiv  Detail & Related papers  (2024-03-28T14:31:37Z)
• Hybrid Reinforcement Learning for Optimizing Pump Sustainability in Real-World Water Distribution Networks [55.591662978280894]
  This article addresses the pump-scheduling optimization problem to enhance real-time control of real-world water distribution networks (WDNs) Our primary objectives are to adhere to physical operational constraints while reducing energy consumption and operational costs. Traditional optimization techniques, such as evolution-based and genetic algorithms, often fall short due to their lack of convergence guarantees.
  arXiv  Detail & Related papers  (2023-10-13T21:26:16Z)
• Digital Twin Assisted Deep Reinforcement Learning for Online Admission Control in Sliced Network [19.152875040151976]
  We propose a digital twin (DT) accelerated DRL solution to address this issue. A neural network-based DT is established with a customized output layer for queuing systems, trained through supervised learning, and then employed to assist the training phase of the DRL model. Extensive simulations show that the DT-accelerated DRL improves resource utilization by over 40% compared to the directly trained state-of-the-art dueling deep Q-learning model.
  arXiv  Detail & Related papers  (2023-10-07T09:09:19Z)
• MARLIN: Soft Actor-Critic based Reinforcement Learning for Congestion Control in Real Networks [63.24965775030673]
  We propose a novel Reinforcement Learning (RL) approach to design generic Congestion Control (CC) algorithms. Our solution, MARLIN, uses the Soft Actor-Critic algorithm to maximize both entropy and return. We trained MARLIN on a real network with varying background traffic patterns to overcome the sim-to-real mismatch.
  arXiv  Detail & Related papers  (2023-02-02T18:27:20Z)
• Toward Safe and Accelerated Deep Reinforcement Learning for Next-Generation Wireless Networks [21.618559590818236]
  We discuss two key practical challenges that are faced but rarely tackled when developing DRL-based RRM solutions. In particular, we discuss the need to have safe and accelerated DRL-based RRM solutions that mitigate the slow convergence and performance instability exhibited by DRL algorithms.
  arXiv  Detail & Related papers  (2022-09-16T04:50:49Z)
• Artificial Intelligence Empowered Multiple Access for Ultra Reliable and Low Latency THz Wireless Networks [76.89730672544216]
  Terahertz (THz) wireless networks are expected to catalyze the beyond fifth generation (B5G) era. To satisfy the ultra-reliability and low-latency demands of several B5G applications, novel mobility management approaches are required. This article presents a holistic MAC layer approach that enables intelligent user association and resource allocation, as well as flexible and adaptive mobility management.
  arXiv  Detail & Related papers  (2022-08-17T03:00:24Z)
• Accelerating Deep Reinforcement Learning for Digital Twin Network Optimization with Evolutionary Strategies [0.0]
  The community proposed the Digital Twin Networks (DTN) as a key enabler of efficient network management. Deep Reinforcement Learning (DRL) showed a high performance when applied to solve network optimization problems. In this paper, we explore the use of Evolutionary Strategies (ES) to train DRL agents for solving a routing optimization problem.
  arXiv  Detail & Related papers  (2022-02-01T11:56:55Z)
• Reinforcement Learning-Empowered Mobile Edge Computing for 6G Edge Intelligence [76.96698721128406]
  Mobile edge computing (MEC) considered a novel paradigm for computation and delay-sensitive tasks in fifth generation (5G) networks and beyond. This paper provides a comprehensive research review on free-enabled RL and offers insight for development.
  arXiv  Detail & Related papers  (2022-01-27T10:02:54Z)
• Single-Shot Pruning for Offline Reinforcement Learning [47.886329599997474]
  Deep Reinforcement Learning (RL) is a powerful framework for solving complex real-world problems. One way to tackle this problem is to prune neural networks leaving only the necessary parameters. We close the gap between RL and single-shot pruning techniques and present a general pruning approach to the Offline RL.
  arXiv  Detail & Related papers  (2021-12-31T18:10:02Z)
• Federated Deep Reinforcement Learning for the Distributed Control of NextG Wireless Networks [16.12495409295754]
  Next Generation (NextG) networks are expected to support demanding internet tactile applications such as augmented reality and connected autonomous vehicles. Data-driven approaches can improve the ability of the network to adapt to the current operating conditions. Deep RL (DRL) has been shown to achieve good performance even in complex environments.
  arXiv  Detail & Related papers  (2021-12-07T03:13:20Z)
• Semantic-Aware Collaborative Deep Reinforcement Learning Over Wireless Cellular Networks [82.02891936174221]
  Collaborative deep reinforcement learning (CDRL) algorithms in which multiple agents can coordinate over a wireless network is a promising approach. In this paper, a novel semantic-aware CDRL method is proposed to enable a group of untrained agents with semantically-linked DRL tasks to collaborate efficiently across a resource-constrained wireless cellular network.
  arXiv  Detail & Related papers  (2021-11-23T18:24:47Z)
• DRL-based Slice Placement under Realistic Network Load Conditions [0.8459686722437155]
  We propose a network slice placement optimization solution based on Deep Reinforcement Learning (DRL) The solution is adapted to networks with large scale and under non-stationary traffic conditions (namely, the network load) We demonstrate the applicability of the proposed solution and its higher and stable performance over a non-controlled DRL-based solution.
  arXiv  Detail & Related papers  (2021-09-27T07:58:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.