Offline Reinforcement Learning and Sequence Modeling for Downlink Link Adaptation

• URL: http://arxiv.org/abs/2410.23031v2
• Date: Thu, 28 Nov 2024 23:00:31 GMT
• Title: Offline Reinforcement Learning and Sequence Modeling for Downlink Link Adaptation
• Authors: Samuele Peri, Alessio Russo, Gabor Fodor, Pablo Soldati,
• Abstract summary: Link adaptation (LA) is an essential function in modern wireless communication systems.<n>LA dynamically adjusts the transmission rate of a communication link to match time- and frequency-varying radio link conditions.<n>Recent research has introduced online reinforcement learning approaches as an alternative to the more commonly used rule-based algorithms.
• Score: 3.687363450234871
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Link adaptation (LA) is an essential function in modern wireless communication systems that dynamically adjusts the transmission rate of a communication link to match time- and frequency-varying radio link conditions. However, factors such as user mobility, fast fading, imperfect channel quality information, and aging of measurements make the modeling of LA challenging. To bypass the need for explicit modeling, recent research has introduced online reinforcement learning (RL) approaches as an alternative to the more commonly used rule-based algorithms. Yet, RL-based approaches face deployment challenges, as training in live networks can potentially degrade real-time performance. To address this challenge, this paper considers offline RL as a candidate to learn LA policies with minimal effects on the network operation. We propose three LA designs based on batch-constrained deep Q-learning, conservative Q-learning, and decision transformer. Our results show that offline RL algorithms can match the performance of state-of-the-art online RL methods when data is collected with a proper behavioral policy.

Related papers

• Offline and Distributional Reinforcement Learning for Wireless Communications [5.771885923067511]
  Traditional online reinforcement learning (RL) and deep RL methods face limitations in real-time wireless networks. We focus on offline and distributional RL, two advanced RL techniques that can overcome these challenges. We introduce a novel framework that combines offline and distributional RL for wireless communication applications.
  arXiv  Detail & Related papers  (2025-04-04T09:24:39Z)
• Resilient UAV Trajectory Planning via Few-Shot Meta-Offline Reinforcement Learning [5.771885923067511]
  This work proposes a novel, resilient, few-shot meta-offline RL algorithm combining offline RL and model-agnostic meta-learning. We show that the proposed few-shot meta-offline RL algorithm converges faster than baseline schemes. It is the only algorithm that can achieve optimal joint AoI and transmission power using an offline dataset.
  arXiv  Detail & Related papers  (2025-02-03T11:39:12Z)
• Continual Task Learning through Adaptive Policy Self-Composition [54.95680427960524]
  CompoFormer is a structure-based continual transformer model that adaptively composes previous policies via a meta-policy network. Our experiments reveal that CompoFormer outperforms conventional continual learning (CL) methods, particularly in longer task sequences.
  arXiv  Detail & Related papers  (2024-11-18T08:20:21Z)
• Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning [62.984693936073974]
  Value-based reinforcement learning can learn effective policies for a wide range of multi-turn problems. Current value-based RL methods have proven particularly challenging to scale to the setting of large language models. We propose a novel offline RL algorithm that addresses these drawbacks, casting Q-learning as a modified supervised fine-tuning problem.
  arXiv  Detail & Related papers  (2024-11-07T21:36:52Z)
• DRL Optimization Trajectory Generation via Wireless Network Intent-Guided Diffusion Models for Optimizing Resource Allocation [58.62766376631344]
  We propose a customized wireless network intent (WNI-G) model to address different state variations of wireless communication networks. Extensive simulation achieves greater stability in spectral efficiency and variations of traditional DRL models in dynamic communication systems.
  arXiv  Detail & Related papers  (2024-10-18T14:04:38Z)
• Closed-form congestion control via deep symbolic regression [1.5961908901525192]
  Reinforcement Learning (RL) algorithms can handle challenges in ultra-low-latency and high throughput scenarios. The adoption of neural network models in real deployments still poses some challenges regarding real-time inference and interpretability. This paper proposes a methodology to deal with such challenges while maintaining the performance and generalization capabilities.
  arXiv  Detail & Related papers  (2024-03-28T14:31:37Z)
• Retentive Decision Transformer with Adaptive Masking for Reinforcement Learning based Recommendation Systems [17.750449033873036]
  Reinforcement Learning-based Recommender Systems (RLRS) have shown promise across a spectrum of applications. Yet, they grapple with challenges, notably in crafting reward functions and harnessing large pre-existing datasets. Recent advancements in offline RLRS provide a solution for how to address these two challenges.
  arXiv  Detail & Related papers  (2024-03-26T12:08:58Z)
• Advancing RAN Slicing with Offline Reinforcement Learning [15.259182716723496]
  This paper introduces offlineReinforcement Learning to solve the RAN slicing problem. We show how offline RL can effectively learn near-optimal policies from sub-optimal datasets. We also present empirical evidence of the efficacy of offline RL in adapting to various service-level requirements.
  arXiv  Detail & Related papers  (2023-12-16T22:09:50Z)
• Towards Scalable Wireless Federated Learning: Challenges and Solutions [40.68297639420033]
  federated learning (FL) emerges as an effective distributed machine learning framework. We discuss the challenges and solutions of achieving scalable wireless FL from the perspectives of both network design and resource orchestration.
  arXiv  Detail & Related papers  (2023-10-08T08:55:03Z)
• A Neuromorphic Architecture for Reinforcement Learning from Real-Valued Observations [0.34410212782758043]
  Reinforcement Learning (RL) provides a powerful framework for decision-making in complex environments. This paper presents a novel Spiking Neural Network (SNN) architecture for solving RL problems with real-valued observations.
  arXiv  Detail & Related papers  (2023-07-06T12:33:34Z)
• ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensembles [52.34951901588738]
  We propose a novel framework called ENsemble-based Offline-To-Online (ENOTO) RL. By increasing the number of Q-networks, we seamlessly bridge offline pre-training and online fine-tuning without degrading performance. Experimental results demonstrate that ENOTO can substantially improve the training stability, learning efficiency, and final performance of existing offline RL methods.
  arXiv  Detail & Related papers  (2023-06-12T05:10:10Z)
• A Unified Framework for Alternating Offline Model Training and Policy Learning [62.19209005400561]
  In offline model-based reinforcement learning, we learn a dynamic model from historically collected data, and utilize the learned model and fixed datasets for policy learning. We develop an iterative offline MBRL framework, where we maximize a lower bound of the true expected return. With the proposed unified model-policy learning framework, we achieve competitive performance on a wide range of continuous-control offline reinforcement learning datasets.
  arXiv  Detail & Related papers  (2022-10-12T04:58:51Z)
• Phase Shift Design in RIS Empowered Wireless Networks: From Optimization to AI-Based Methods [83.98961686408171]
  Reconfigurable intelligent surfaces (RISs) have a revolutionary capability to customize the radio propagation environment for wireless networks. To fully exploit the advantages of RISs in wireless systems, the phases of the reflecting elements must be jointly designed with conventional communication resources. This paper provides a review of current optimization methods and artificial intelligence-based methods for handling the constraints imposed by RIS.
  arXiv  Detail & Related papers  (2022-04-28T09:26:14Z)
• MOORe: Model-based Offline-to-Online Reinforcement Learning [26.10368749930102]
  We propose a model-based Offline-to-Online Reinforcement learning (MOORe) algorithm. Experiment results show that our algorithm smoothly transfers from offline to online stages while enabling sample-efficient online adaption.
  arXiv  Detail & Related papers  (2022-01-25T03:14:57Z)
• Offline Contextual Bandits for Wireless Network Optimization [107.24086150482843]
  In this paper, we investigate how to learn policies that can automatically adjust the configuration parameters of every cell in the network in response to the changes in the user demand. Our solution combines existent methods for offline learning and adapts them in a principled way to overcome crucial challenges arising in this context.
  arXiv  Detail & Related papers  (2021-11-11T11:31:20Z)
• Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL [82.93243616342275]
  We introduce Offline Model-based RL with Adaptive Behavioral Priors (MABE) MABE is based on the finding that dynamics models, which support within-domain generalization, and behavioral priors, which support cross-domain generalization, are complementary. In experiments that require cross-domain generalization, we find that MABE outperforms prior methods.
  arXiv  Detail & Related papers  (2021-06-16T20:48:49Z)
• Reconfigurable Intelligent Surface Assisted Mobile Edge Computing with Heterogeneous Learning Tasks [53.1636151439562]
  Mobile edge computing (MEC) provides a natural platform for AI applications. We present an infrastructure to perform machine learning tasks at an MEC with the assistance of a reconfigurable intelligent surface (RIS) Specifically, we minimize the learning error of all participating users by jointly optimizing transmit power of mobile users, beamforming vectors of the base station, and the phase-shift matrix of the RIS.
  arXiv  Detail & Related papers  (2020-12-25T07:08:50Z)
• Optimization-driven Machine Learning for Intelligent Reflecting Surfaces Assisted Wireless Networks [82.33619654835348]
  Intelligent surface (IRS) has been employed to reshape the wireless channels by controlling individual scattering elements' phase shifts. Due to the large size of scattering elements, the passive beamforming is typically challenged by the high computational complexity. In this article, we focus on machine learning (ML) approaches for performance in IRS-assisted wireless networks.
  arXiv  Detail & Related papers  (2020-08-29T08:39:43Z)
• Optimization-driven Deep Reinforcement Learning for Robust Beamforming in IRS-assisted Wireless Communications [54.610318402371185]
  Intelligent reflecting surface (IRS) is a promising technology to assist downlink information transmissions from a multi-antenna access point (AP) to a receiver. We minimize the AP's transmit power by a joint optimization of the AP's active beamforming and the IRS's passive beamforming. We propose a deep reinforcement learning (DRL) approach that can adapt the beamforming strategies from past experiences.
  arXiv  Detail & Related papers  (2020-05-25T01:42:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.