FedFormer: Contextual Federation with Attention in Reinforcement Learning

• URL: http://arxiv.org/abs/2205.13697v2
• Date: Mon, 30 May 2022 02:19:18 GMT
• Title: FedFormer: Contextual Federation with Attention in Reinforcement Learning
• Abstract summary: FedFormer is a novel federation strategy that utilizes Transformer Attention to contextually aggregate embeddings. We evaluate our methods on the Meta-World environment and find that our approach yields significant improvements over FedAvg and non-federated Soft Actor Critique single agent methods.
• Score: 21.05134144161283
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A core issue in federated reinforcement learning is defining how to aggregate insights from multiple agents into one. This is commonly done by taking the average of each participating agent's model weights into one common model (FedAvg). We instead propose FedFormer, a novel federation strategy that utilizes Transformer Attention to contextually aggregate embeddings from models originating from different learner agents. In so doing, we attentively weigh contributions of other agents with respect to the current agent's environment and learned relationships, thus providing more effective and efficient federation. We evaluate our methods on the Meta-World environment and find that our approach yields significant improvements over FedAvg and non-federated Soft Actor Critique single agent methods. Our results compared to Soft Actor Critique show that FedFormer performs better while still abiding by the privacy constraints of federated learning. In addition, we demonstrate nearly linear improvements in effectiveness with increased agent pools in certain tasks. This is contrasted by FedAvg, which fails to make noticeable improvements when scaled.

Related papers

• Low Variance Trust Region Optimization with Independent Actors and Sequential Updates in Cooperative Multi-agent Reinforcement Learning [5.8010446129208155]
  We introduce a clipping objective to control the upper bounds of the advantage fluctuation in sequential updates.<n>We derive two new practical algorithms using our clipping objective.<n>Our proposed method is highlighted with both stable convergence properties and the desired low advantage variance estimation.
  arXiv  Detail & Related papers  (2026-06-24T08:05:11Z)
• Leveraging Large Language Models for Effective and Explainable Multi-Agent Credit Assignment [4.406086834602686]
  We show how to reformulate credit assignment to the two pattern recognition problems of sequence improvement and attribution. Our approach utilizes a centralized reward-critic which numerically decomposes the environment reward based on the individual contribution of each agent. Both our methods far outperform the state-of-the-art on a variety of benchmarks, including Level-Based Foraging, Robotic Warehouse, and our new Spaceworld benchmark which incorporates collision-related safety constraints.
  arXiv  Detail & Related papers  (2025-02-24T05:56:47Z)
• FedStaleWeight: Buffered Asynchronous Federated Learning with Fair Aggregation via Staleness Reweighting [9.261784956541641]
  Asynchronous Federated Learning (AFL) methods have emerged as promising alternatives to their synchronous counterparts by the slowest agent. AFL model training heavily towards agents who can produce updates faster, leaving slower agents behind. We introduce FedStaleWeight, an algorithm addressing in aggregating asynchronous client updates by employing average staleness to compute fair re-weightings.
  arXiv  Detail & Related papers  (2024-06-05T02:52:22Z)
• FedMIA: An Effective Membership Inference Attack Exploiting "All for One" Principle in Federated Learning [17.141646895576145]
  Federated Learning (FL) is a promising approach for training machine learning models on decentralized data. Membership Inference Attacks (MIAs) aim to determine whether a specific data point belongs to a target client's training set. We introduce a three-step Membership Inference Attack (MIA) method, called FedMIA, which follows the "all for one"--leveraging updates from all clients across multiple communication rounds to enhance MIA effectiveness.
  arXiv  Detail & Related papers  (2024-02-09T09:58:35Z)
• Combating Exacerbated Heterogeneity for Robust Models in Federated Learning [91.88122934924435]
  Combination of adversarial training and federated learning can lead to the undesired robustness deterioration. We propose a novel framework called Slack Federated Adversarial Training (SFAT) We verify the rationality and effectiveness of SFAT on various benchmarked and real-world datasets.
  arXiv  Detail & Related papers  (2023-03-01T06:16:15Z)
• Beyond ADMM: A Unified Client-variance-reduced Adaptive Federated Learning Framework [82.36466358313025]
  We propose a primal-dual FL algorithm, termed FedVRA, that allows one to adaptively control the variance-reduction level and biasness of the global model. Experiments based on (semi-supervised) image classification tasks demonstrate superiority of FedVRA over the existing schemes.
  arXiv  Detail & Related papers  (2022-12-03T03:27:51Z)
• Learning From Good Trajectories in Offline Multi-Agent Reinforcement Learning [98.07495732562654]
  offline multi-agent reinforcement learning (MARL) aims to learn effective multi-agent policies from pre-collected datasets. One agent learned by offline MARL often inherits this random policy, jeopardizing the performance of the entire team. We propose a novel framework called Shared Individual Trajectories (SIT) to address this problem.
  arXiv  Detail & Related papers  (2022-11-28T18:11:26Z)
• Meta Knowledge Condensation for Federated Learning [65.20774786251683]
  Existing federated learning paradigms usually extensively exchange distributed models at a central solver to achieve a more powerful model. This would incur severe communication burden between a server and multiple clients especially when data distributions are heterogeneous. Unlike existing paradigms, we introduce an alternative perspective to significantly decrease the communication cost in federate learning.
  arXiv  Detail & Related papers  (2022-09-29T15:07:37Z)
• FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learning [87.08902493524556]
  Federated learning(FL) has recently attracted increasing attention from academia and industry. We propose FedDM to build the global training objective from multiple local surrogate functions. In detail, we construct synthetic sets of data on each client to locally match the loss landscape from original data.
  arXiv  Detail & Related papers  (2022-07-20T04:55:18Z)
• Federated Adversarial Training with Transformers [16.149924042225106]
  Federated learning (FL) has emerged to enable global model training over distributed clients' data while preserving its privacy. This paper investigates feasibility with different federated model aggregation methods and different vision transformer models with different tokenization and classification head techniques.
  arXiv  Detail & Related papers  (2022-06-05T09:07:09Z)
• Exploiting Heterogeneity in Robust Federated Best-Arm Identification [19.777265059976337]
  Fed-SEL is a simple communication-efficient algorithm that builds on successive elimination techniques and involves local sampling steps at the clients. We show that for certain heterogeneous problem instances, Fed-SEL outputs the best-arm after just one round of communication. As our final contribution, we develop variants of Fed-SEL, both for federated and peer-to-peer settings, that are robust to the presence of Byzantine clients.
  arXiv  Detail & Related papers  (2021-09-13T04:22:21Z)
• Deep Interactive Bayesian Reinforcement Learning via Meta-Learning [63.96201773395921]
  The optimal adaptive behaviour under uncertainty over the other agents' strategies can be computed using the Interactive Bayesian Reinforcement Learning framework. We propose to meta-learn approximate belief inference and Bayes-optimal behaviour for a given prior. We show empirically that our approach outperforms existing methods that use a model-free approach, sample from the approximate posterior, maintain memory-free models of others, or do not fully utilise the known structure of the environment.
  arXiv  Detail & Related papers  (2021-01-11T13:25:13Z)
• FedMGDA+: Federated Learning meets Multi-objective Optimization [22.36800379342697]
  Federated learning has emerged as a promising, massively distributed way to train a joint deep model over large amounts of edge devices. We formulate federated learning as multi-objective optimization and propose a new algorithm FedMGDA+ that is guaranteed to converge to stationary solutions.
  arXiv  Detail & Related papers  (2020-06-20T04:17:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.