Shaping Rewards, Shaping Routes: On Multi-Agent Deep Q-Networks for Routing in Satellite Constellation Networks

• URL: http://arxiv.org/abs/2408.01979v1
• Date: Sun, 4 Aug 2024 09:53:57 GMT
• Title: Shaping Rewards, Shaping Routes: On Multi-Agent Deep Q-Networks for Routing in Satellite Constellation Networks
• Authors: Manuel M. H. Roth, Anupama Hegde, Thomas Delamotte, Andreas Knopp,
• Abstract summary: We investigate the viability of multi-agent deep Q-networks for routing in satellite constellation networks. We focus on reward shaping and quantifying training convergence for joint optimization of latency and load balancing. We propose a novel hybrid solution based on centralized learning and decentralized control.
• Score: 2.774897240515734
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Effective routing in satellite mega-constellations has become crucial to facilitate the handling of increasing traffic loads, more complex network architectures, as well as the integration into 6G networks. To enhance adaptability as well as robustness to unpredictable traffic demands, and to solve dynamic routing environments efficiently, machine learning-based solutions are being considered. For network control problems, such as optimizing packet forwarding decisions according to Quality of Service requirements and maintaining network stability, deep reinforcement learning techniques have demonstrated promising results. For this reason, we investigate the viability of multi-agent deep Q-networks for routing in satellite constellation networks. We focus specifically on reward shaping and quantifying training convergence for joint optimization of latency and load balancing in static and dynamic scenarios. To address identified drawbacks, we propose a novel hybrid solution based on centralized learning and decentralized control.

Related papers

• A Distance Similarity-based Genetic Optimization Algorithm for Satellite Ground Network Planning Considering Feeding Mode [53.71516191515285]
  The low transmission efficiency of the satellite data relay back mission has become a problem that is currently constraining the construction of the system. We propose a distance similarity-based genetic optimization algorithm (DSGA), which considers the state characteristics between the tasks and introduces a weighted Euclidean distance method to determine the similarity between the tasks.
  arXiv  Detail & Related papers  (2024-08-29T06:57:45Z)
• Dynamic Routing for Integrated Satellite-Terrestrial Networks: A Constrained Multi-Agent Reinforcement Learning Approach [41.714453335170404]
  We study packet routing with ground stations and satellites working jointly to transmit packets. We propose a novel constrained Multi-Agent reinforcement learning (MARL) dynamic routing algorithm named CMADR. Results demonstrate that CMADR reduces the packet delay by a minimum of 21% and 15%, while meeting stringent energy consumption and packet loss rate constraints, outperforming several baseline algorithms.
  arXiv  Detail & Related papers  (2023-12-23T03:36:35Z)
• An Intelligent SDWN Routing Algorithm Based on Network Situational Awareness and Deep Reinforcement Learning [4.085916808788356]
  This article introduces an intelligent routing algorithm (DRL-PPONSA) based on deep reinforcement learning with network situational awareness. Experimental results show that DRL-PPONSA outperforms traditional routing methods in network throughput, delay, packet loss rate, and wireless node distance.
  arXiv  Detail & Related papers  (2023-05-12T14:18:09Z)
• Multi-Agent Reinforcement Learning for Network Routing in Integrated Access Backhaul Networks [0.0]
  We aim to maximize packet arrival ratio while minimizing their latency in IAB networks. To solve this problem, we develop a multi-agent partially observed Markov decision process (POMD) We show that A2C outperforms other reinforcement learning algorithms, leading to increased network efficiency and reduced selfish agent behavior.
  arXiv  Detail & Related papers  (2023-05-12T13:03:26Z)
• MARLIN: Soft Actor-Critic based Reinforcement Learning for Congestion Control in Real Networks [63.24965775030673]
  We propose a novel Reinforcement Learning (RL) approach to design generic Congestion Control (CC) algorithms. Our solution, MARLIN, uses the Soft Actor-Critic algorithm to maximize both entropy and return. We trained MARLIN on a real network with varying background traffic patterns to overcome the sim-to-real mismatch.
  arXiv  Detail & Related papers  (2023-02-02T18:27:20Z)
• Multi-agent Reinforcement Learning with Graph Q-Networks for Antenna Tuning [60.94661435297309]
  The scale of mobile networks makes it challenging to optimize antenna parameters using manual intervention or hand-engineered strategies. We propose a new multi-agent reinforcement learning algorithm to optimize mobile network configurations globally. We empirically demonstrate the performance of the algorithm on an antenna tilt tuning problem and a joint tilt and power control problem in a simulated environment.
  arXiv  Detail & Related papers  (2023-01-20T17:06:34Z)
• Machine Learning-Based User Scheduling in Integrated Satellite-HAPS-Ground Networks [82.58968700765783]
  Integrated space-air-ground networks promise to offer a valuable solution space for empowering the sixth generation of communication networks (6G) This paper showcases the prospects of machine learning in the context of user scheduling in integrated space-air-ground communications.
  arXiv  Detail & Related papers  (2022-05-27T13:09:29Z)
• A Multi-Domain VNE Algorithm based on Load Balancing in the IoT networks [22.63148849159129]
  This paper proposes a virtual network mapping strategy based on hybrid genetic algorithm. It uses a cross-probability and pheromone-based mutation gene selection strategy to improve the flexibility of the algorithm. It performs well in a number of performance metrics including mapping average quotation, link load balancing, mapping cost-benefit ratio, acceptance rate and running time.
  arXiv  Detail & Related papers  (2022-02-07T01:01:21Z)
• Offline Contextual Bandits for Wireless Network Optimization [107.24086150482843]
  In this paper, we investigate how to learn policies that can automatically adjust the configuration parameters of every cell in the network in response to the changes in the user demand. Our solution combines existent methods for offline learning and adapts them in a principled way to overcome crucial challenges arising in this context.
  arXiv  Detail & Related papers  (2021-11-11T11:31:20Z)
• Packet Routing with Graph Attention Multi-agent Reinforcement Learning [4.78921052969006]
  We develop a model-free and data-driven routing strategy by leveraging reinforcement learning (RL) Considering the graph nature of the network topology, we design a multi-agent RL framework in combination with Graph Neural Network (GNN)
  arXiv  Detail & Related papers  (2021-07-28T06:20:34Z)
• On Topology Optimization and Routing in Integrated Access and Backhaul Networks: A Genetic Algorithm-based Approach [70.85399600288737]
  We study the problem of topology optimization and routing in IAB networks. We develop efficient genetic algorithm-based schemes for both IAB node placement and non-IAB backhaul link distribution. We discuss the main challenges for enabling mesh-based IAB networks.
  arXiv  Detail & Related papers  (2021-02-14T21:52:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.