Related papers: Personalized Federated Learning with Local Attention

Personalized Federated Learning with Local Attention

URL: http://arxiv.org/abs/2304.01783v2
Date: Fri, 14 Apr 2023 10:01:36 GMT
Title: Personalized Federated Learning with Local Attention
Authors: Sicong Liang, Junchao Tian, Shujun Yang, Yu Zhang
Abstract summary: Federated Learning (FL) aims to learn a single global model that enables the central server to help the model training in local clients without accessing their local data. Key challenge of FL is the heterogeneous label distribution and feature shift, which could lead to significant performance degradation of the learned models. We propose a simple yet effective algorithm, namely textbfpersonalized textbfFederated learning with textbfLocal textbfAttention (pFedLA) Two modules are proposed in pFedLA, i.e., the personalized
Score: 5.018560254008613
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Abstract: Federated Learning (FL) aims to learn a single global model that enables the central server to help the model training in local clients without accessing their local data. The key challenge of FL is the heterogeneity of local data in different clients, such as heterogeneous label distribution and feature shift, which could lead to significant performance degradation of the learned models. Although many studies have been proposed to address the heterogeneous label distribution problem, few studies attempt to explore the feature shift issue. To address this issue, we propose a simple yet effective algorithm, namely \textbf{p}ersonalized \textbf{Fed}erated learning with \textbf{L}ocal \textbf{A}ttention (pFedLA), by incorporating the attention mechanism into personalized models of clients while keeping the attention blocks client-specific. Specifically, two modules are proposed in pFedLA, i.e., the personalized single attention module and the personalized hybrid attention module. In addition, the proposed pFedLA method is quite flexible and general as it can be incorporated into any FL method to improve their performance without introducing additional communication costs. Extensive experiments demonstrate that the proposed pFedLA method can boost the performance of state-of-the-art FL methods on different tasks such as image classification and object detection tasks.