Continual Deep Reinforcement Learning for Decentralized Satellite Routing
- URL: http://arxiv.org/abs/2405.12308v1
- Date: Mon, 20 May 2024 18:12:36 GMT
- Title: Continual Deep Reinforcement Learning for Decentralized Satellite Routing
- Authors: Federico Lozano-Cuadra, Beatriz Soret, Israel Leyva-Mayorga, Petar Popovski,
- Abstract summary: This paper introduces a full solution for decentralized routing in Low Earth Orbit satellite constellations based on continual Deep Reinforcement Learning (DRL)
We follow a multi-agent approach, where each satellite acts as an independent decision-making agent.
Our solution adapts well to congestion conditions and exploits less loaded paths.
- Score: 34.67962234401005
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: This paper introduces a full solution for decentralized routing in Low Earth Orbit satellite constellations based on continual Deep Reinforcement Learning (DRL). This requires addressing multiple challenges, including the partial knowledge at the satellites and their continuous movement, and the time-varying sources of uncertainty in the system, such as traffic, communication links, or communication buffers. We follow a multi-agent approach, where each satellite acts as an independent decision-making agent, while acquiring a limited knowledge of the environment based on the feedback received from the nearby agents. The solution is divided into two phases. First, an offline learning phase relies on decentralized decisions and a global Deep Neural Network (DNN) trained with global experiences. Then, the online phase with local, on-board, and pre-trained DNNs requires continual learning to evolve with the environment, which can be done in two different ways: (1) Model anticipation, where the predictable conditions of the constellation are exploited by each satellite sharing local model with the next satellite; and (2) Federated Learning (FL), where each agent's model is merged first at the cluster level and then aggregated in a global Parameter Server. The results show that, without high congestion, the proposed Multi-Agent DRL framework achieves the same E2E performance as a shortest-path solution, but the latter assumes intensive communication overhead for real-time network-wise knowledge of the system at a centralized node, whereas ours only requires limited feedback exchange among first neighbour satellites. Importantly, our solution adapts well to congestion conditions and exploits less loaded paths. Moreover, the divergence of models over time is easily tackled by the synergy between anticipation, applied in short-term alignment, and FL, utilized for long-term alignment.
Related papers
- Collaborative Ground-Space Communications via Evolutionary Multi-objective Deep Reinforcement Learning [113.48727062141764]
We propose a distributed collaborative beamforming (DCB)-based uplink communication paradigm for enabling ground-space direct communications.
DCB treats the terminals that are unable to establish efficient direct connections with the low Earth orbit (LEO) satellites as distributed antennas.
We propose an evolutionary multi-objective deep reinforcement learning algorithm to obtain the desirable policies.
arXiv Detail & Related papers (2024-04-11T03:13:02Z) - Satellite Federated Edge Learning: Architecture Design and Convergence Analysis [47.057886812985984]
This paper introduces a novel FEEL algorithm, named FEDMEGA, tailored to mega-constellation networks.
By integrating inter-satellite links (ISL) for intra-orbit model aggregation, the proposed algorithm significantly reduces the usage of low data rate and intermittent GSL.
Our proposed method includes a ring all-reduce based intra-orbit aggregation mechanism, coupled with a network flow-based transmission scheme for global model aggregation.
arXiv Detail & Related papers (2024-04-02T11:59:58Z) - Multi-Agent Deep Reinforcement Learning for Distributed Satellite
Routing [7.793857269225969]
This paper introduces a Multi-Agent Deep Reinforcement Learning (MA-DRL) approach for routing in Low Earth Orbit Satellite Constellations (LSatCs)
Results show that MA-DRL efficiently learns optimal routes offline that are then loaded for an efficient distributed routing online.
arXiv Detail & Related papers (2024-02-27T16:36:53Z) - Communication-Efficient Federated Learning for LEO Satellite Networks
Integrated with HAPs Using Hybrid NOMA-OFDM [1.3121410433987561]
This paper proposes NomaFedHAP, a novel FL-SatCom approach tailored to LEO satellites.
NomaFedHAP utilizes high-altitude platforms (HAPs) as distributed parameter servers (PS) to enhance satellite visibility.
We derive a closed-form expression of the outage probability for satellites in near and far shells, as well as for the entire system.
arXiv Detail & Related papers (2024-01-01T07:07:27Z) - Adaptive Hierarchical SpatioTemporal Network for Traffic Forecasting [70.66710698485745]
We propose an Adaptive Hierarchical SpatioTemporal Network (AHSTN) to promote traffic forecasting.
AHSTN exploits the spatial hierarchy and modeling multi-scale spatial correlations.
Experiments on two real-world datasets show that AHSTN achieves better performance over several strong baselines.
arXiv Detail & Related papers (2023-06-15T14:50:27Z) - Federated learning for LEO constellations via inter-HAP links [0.0]
Low Earth Obit (LEO) satellite constellations have seen a sharp increase of deployment in recent years.
To apply machine learning (ML) in such applications, the traditional way of downloading satellite data such as imagery to a ground station (GS) is not desirable.
We show that existing FL solutions do not fit well in such LEO constellation scenarios because of significant challenges such as excessive convergence delay and unreliable wireless channels.
arXiv Detail & Related papers (2022-05-15T08:22:52Z) - Learning Emergent Random Access Protocol for LEO Satellite Networks [51.575090080749554]
We propose a novel grant-free random access solution for LEO SAT networks, dubbed emergent random access channel protocol (eRACH)
eRACH is a model-free approach that emerges through interaction with the non-stationary network environment.
Compared to RACH, we show from various simulations that our proposed eRACH yields 54.6% higher average network throughput.
arXiv Detail & Related papers (2021-12-03T07:44:45Z) - Integrating LEO Satellite and UAV Relaying via Reinforcement Learning
for Non-Terrestrial Networks [51.05735925326235]
A mega-constellation of low-earth orbit (LEO) satellites has the potential to enable long-range communication with low latency.
We study the problem of forwarding packets between two faraway ground terminals, through an LEO satellite selected from an orbiting constellation.
To maximize the end-to-end data rate, the satellite association and HAP location should be optimized.
We tackle this problem using deep reinforcement learning (DRL) with a novel action dimension reduction technique.
arXiv Detail & Related papers (2020-05-26T05:39:27Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.