Architecture Agnostic Federated Learning for Neural Networks

• URL: http://arxiv.org/abs/2202.07757v1
• Date: Tue, 15 Feb 2022 22:16:06 GMT
• Title: Architecture Agnostic Federated Learning for Neural Networks
• Authors: Disha Makhija, Xing Han, Nhat Ho, Joydeep Ghosh
• Abstract summary: This work introduces a novel Federated Heterogeneous Neural Networks (FedHeNN) framework. FedHeNN allows each client to build a personalised model without enforcing a common architecture across clients. The key idea of FedHeNN is to use the instance-level representations obtained from peer clients to guide the simultaneous training on each client.
• Score: 19.813602191888837
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With growing concerns regarding data privacy and rapid increase in data volume, Federated Learning(FL) has become an important learning paradigm. However, jointly learning a deep neural network model in a FL setting proves to be a non-trivial task because of the complexities associated with the neural networks, such as varied architectures across clients, permutation invariance of the neurons, and presence of non-linear transformations in each layer. This work introduces a novel Federated Heterogeneous Neural Networks (FedHeNN) framework that allows each client to build a personalised model without enforcing a common architecture across clients. This allows each client to optimize with respect to local data and compute constraints, while still benefiting from the learnings of other (potentially more powerful) clients. The key idea of FedHeNN is to use the instance-level representations obtained from peer clients to guide the simultaneous training on each client. The extensive experimental results demonstrate that the FedHeNN framework is capable of learning better performing models on clients in both the settings of homogeneous and heterogeneous architectures across clients.

Related papers

• Federated Learning with Flexible Architectures [12.800116749927266]
  This paper introduces Federated Learning with Flexible Architectures (FedFA), an FL training algorithm that allows clients to train models of different widths and depths. FedFA incorporates the layer grafting technique to align clients' local architectures with the largest network architecture in the FL system during model aggregation.
  arXiv  Detail & Related papers  (2024-06-14T09:44:46Z)
• An Aggregation-Free Federated Learning for Tackling Data Heterogeneity [50.44021981013037]
  Federated Learning (FL) relies on the effectiveness of utilizing knowledge from distributed datasets. Traditional FL methods adopt an aggregate-then-adapt framework, where clients update local models based on a global model aggregated by the server from the previous training round. We introduce FedAF, a novel aggregation-free FL algorithm.
  arXiv  Detail & Related papers  (2024-04-29T05:55:23Z)
• FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup for Non-IID Data [54.81695390763957]
  Federated learning is an emerging distributed machine learning method. We propose a heterogeneous local variant of AMSGrad, named FedLALR, in which each client adjusts its learning rate. We show that our client-specified auto-tuned learning rate scheduling can converge and achieve linear speedup with respect to the number of clients.
  arXiv  Detail & Related papers  (2023-09-18T12:35:05Z)
• Neural Collapse Inspired Federated Learning with Non-iid Data [31.576588815816095]
  Non-independent and identically distributed (non-iid) characteristics cause significant differences in local updates and affect the performance of the central server. Inspired by the phenomenon of neural collapse, we force each client to be optimized toward an optimal global structure for classification. Our method can improve the performance with faster convergence speed on different-size datasets.
  arXiv  Detail & Related papers  (2023-03-27T05:29:53Z)
• FedClassAvg: Local Representation Learning for Personalized Federated Learning on Heterogeneous Neural Networks [21.613436984547917]
  We propose a novel personalized federated learning method called federated classifier averaging (FedClassAvg) FedClassAvg aggregates weights as an agreement on decision boundaries on feature spaces. We demonstrate it outperforms the current state-of-the-art algorithms on heterogeneous personalized federated learning tasks.
  arXiv  Detail & Related papers  (2022-10-25T08:32:08Z)
• Straggler-Resilient Personalized Federated Learning [55.54344312542944]
  Federated learning allows training models from samples distributed across a large network of clients while respecting privacy and communication restrictions. We develop a novel algorithmic procedure with theoretical speedup guarantees that simultaneously handles two of these hurdles. Our method relies on ideas from representation learning theory to find a global common representation using all clients' data and learn a user-specific set of parameters leading to a personalized solution for each client.
  arXiv  Detail & Related papers  (2022-06-05T01:14:46Z)
• SPIDER: Searching Personalized Neural Architecture for Federated Learning [17.61748275091843]
  Federated learning (FL) assists machine learning when data cannot be shared with a centralized server due to privacy and regulatory restrictions. Recent advancements in FL use predefined architecture-based learning for all the clients. We introduce SPIDER, an algorithmic framework that aims to Search Personalized neural architecture for federated learning.
  arXiv  Detail & Related papers  (2021-12-27T23:42:15Z)
• Comfetch: Federated Learning of Large Networks on Constrained Clients via Sketching [28.990067638230254]
  Federated learning (FL) is a popular paradigm for private and collaborative model training on the edge. We propose a novel algorithm, Comdirectional, which allows clients to train large networks using representations of the global neural network.
  arXiv  Detail & Related papers  (2021-09-17T04:48:42Z)
• Exploiting Shared Representations for Personalized Federated Learning [54.65133770989836]
  We propose a novel federated learning framework and algorithm for learning a shared data representation across clients and unique local heads for each client. Our algorithm harnesses the distributed computational power across clients to perform many local-updates with respect to the low-dimensional local parameters for every update of the representation. This result is of interest beyond federated learning to a broad class of problems in which we aim to learn a shared low-dimensional representation among data distributions.
  arXiv  Detail & Related papers  (2021-02-14T05:36:25Z)
• Toward Understanding the Influence of Individual Clients in Federated Learning [52.07734799278535]
  Federated learning allows clients to jointly train a global model without sending their private data to a central server. We defined a new notion called em-Influence, quantify this influence over parameters, and proposed an effective efficient model to estimate this metric.
  arXiv  Detail & Related papers  (2020-12-20T14:34:36Z)
• Federated Residual Learning [53.77128418049985]
  We study a new form of federated learning where the clients train personalized local models and make predictions jointly with the server-side shared model. Using this new federated learning framework, the complexity of the central shared model can be minimized while still gaining all the performance benefits that joint training provides.
  arXiv  Detail & Related papers  (2020-03-28T19:55:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.