Federated Reinforcement Learning: Techniques, Applications, and Open Challenges

• URL: http://arxiv.org/abs/2108.11887v1
• Date: Thu, 26 Aug 2021 16:22:49 GMT
• Title: Federated Reinforcement Learning: Techniques, Applications, and Open Challenges
• Authors: Jiaju Qi, Qihao Zhou, Lei Lei, Kan Zheng
• Abstract summary: Federated Reinforcement Learning (FRL) is an emerging and promising field in Reinforcement Learning (RL) FRL algorithms can be divided into two categories, i.e. Horizontal Federated Reinforcement Learning (HFRL) and Vertical Federated Reinforcement Learning (VFRL)
• Score: 4.749929332500373
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper presents a comprehensive survey of Federated Reinforcement Learning (FRL), an emerging and promising field in Reinforcement Learning (RL). Starting with a tutorial of Federated Learning (FL) and RL, we then focus on the introduction of FRL as a new method with great potential by leveraging the basic idea of FL to improve the performance of RL while preserving data-privacy. According to the distribution characteristics of the agents in the framework, FRL algorithms can be divided into two categories, i.e. Horizontal Federated Reinforcement Learning (HFRL) and Vertical Federated Reinforcement Learning (VFRL). We provide the detailed definitions of each category by formulas, investigate the evolution of FRL from a technical perspective, and highlight its advantages over previous RL algorithms. In addition, the existing works on FRL are summarized by application fields, including edge computing, communication, control optimization, and attack detection. Finally, we describe and discuss several key research directions that are crucial to solving the open problems within FRL.

Related papers

• Advances in Preference-based Reinforcement Learning: A Review [1.474723404975345]
  Preference-based reinforcement learning (PbRL) addresses that by utilizing human preferences as feedback from the experts instead of numeric rewards. We present a unified PbRL framework to include the newly emerging approaches that improve the scalability and efficiency of PbRL.
  arXiv  Detail & Related papers  (2024-08-21T18:57:12Z)
• Federated Representation Learning in the Under-Parameterized Regime [10.551397415936309]
  Federated representation learning (FRL) is a popular personalized federated learning (FL) framework where clients work together to train a common representation while retaining their personalized heads. We propose a novel FRL algorithm FLUTE, and theoretically characterize its sample complexity and convergence rate for linear models. Experimental results demonstrate that FLUTE outperforms state-of-the-art FRL solutions in both synthetic and real-world tasks.
  arXiv  Detail & Related papers  (2024-06-07T03:00:07Z)
• How Can LLM Guide RL? A Value-Based Approach [68.55316627400683]
  Reinforcement learning (RL) has become the de facto standard practice for sequential decision-making problems by improving future acting policies with feedback. Recent developments in large language models (LLMs) have showcased impressive capabilities in language understanding and generation, yet they fall short in exploration and self-improvement capabilities. We develop an algorithm named LINVIT that incorporates LLM guidance as a regularization factor in value-based RL, leading to significant reductions in the amount of data needed for learning.
  arXiv  Detail & Related papers  (2024-02-25T20:07:13Z)
• Reinforcement Learning-assisted Evolutionary Algorithm: A Survey and Research Opportunities [63.258517066104446]
  Reinforcement learning integrated as a component in the evolutionary algorithm has demonstrated superior performance in recent years. We discuss the RL-EA integration method, the RL-assisted strategy adopted by RL-EA, and its applications according to the existing literature. In the applications of RL-EA section, we also demonstrate the excellent performance of RL-EA on several benchmarks and a range of public datasets.
  arXiv  Detail & Related papers  (2023-08-25T15:06:05Z)
• Provable Reward-Agnostic Preference-Based Reinforcement Learning [61.39541986848391]
  Preference-based Reinforcement Learning (PbRL) is a paradigm in which an RL agent learns to optimize a task using pair-wise preference-based feedback over trajectories. We propose a theoretical reward-agnostic PbRL framework where exploratory trajectories that enable accurate learning of hidden reward functions are acquired.
  arXiv  Detail & Related papers  (2023-05-29T15:00:09Z)
• Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning [73.80728148866906]
  Quasimetric Reinforcement Learning (QRL) is a new RL method that utilizes quasimetric models to learn optimal value functions. On offline and online goal-reaching benchmarks, QRL also demonstrates improved sample efficiency and performance.
  arXiv  Detail & Related papers  (2023-04-03T17:59:58Z)
• Understanding Reinforcement Learning Algorithms: The Progress from Basic Q-learning to Proximal Policy Optimization [0.6091702876917281]
  reinforcement learning (RL) has a unique setting, jargon, and mathematics that can be intimidating for those new to the field or artificial intelligence. This paper provides a clear and concise overview of the fundamental principles of RL and covers the different types of RL algorithms. The presentation of the paper is aligned with the historical progress of the field, from the early 1980s Q-learning algorithm to the current state-of-the-art algorithms such as TD3, PPO, and offline RL.
  arXiv  Detail & Related papers  (2023-03-31T17:24:51Z)
• Local Environment Poisoning Attacks on Federated Reinforcement Learning [1.5020330976600738]
  Federated learning (FL) has become a popular tool for solving traditional Reinforcement Learning (RL) tasks. Federated mechanism exposes the system to poisoning by malicious agents that can mislead the trained policy. We propose a general framework to characterize FRL poisoning as an optimization problem and design a poisoning protocol that can be applied to policy-based FRL.
  arXiv  Detail & Related papers  (2023-03-05T17:44:23Z)
• Jump-Start Reinforcement Learning [68.82380421479675]
  We present a meta algorithm that can use offline data, demonstrations, or a pre-existing policy to initialize an RL policy. In particular, we propose Jump-Start Reinforcement Learning (JSRL), an algorithm that employs two policies to solve tasks. We show via experiments that JSRL is able to significantly outperform existing imitation and reinforcement learning algorithms.
  arXiv  Detail & Related papers  (2022-04-05T17:25:22Z)
• RL-DARTS: Differentiable Architecture Search for Reinforcement Learning [62.95469460505922]
  We introduce RL-DARTS, one of the first applications of Differentiable Architecture Search (DARTS) in reinforcement learning (RL) By replacing the image encoder with a DARTS supernet, our search method is sample-efficient, requires minimal extra compute resources, and is also compatible with off-policy and on-policy RL algorithms, needing only minor changes in preexisting code. We show that the supernet gradually learns better cells, leading to alternative architectures which can be highly competitive against manually designed policies, but also verify previous design choices for RL policies.
  arXiv  Detail & Related papers  (2021-06-04T03:08:43Z)
• Hierarchical Reinforcement Learning for Relay Selection and Power Optimization in Two-Hop Cooperative Relay Network [7.5377621697101205]
  We study the outage probability minimizing problem subjected to a total transmission power constraint in a two-hop cooperative relay network. We use reinforcement learning (RL) methods to learn strategies for relay selection and power allocation. We propose a hierarchical reinforcement learning (HRL) framework and training algorithm.
  arXiv  Detail & Related papers  (2020-11-10T04:47:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.