Federated Learning with Heterogeneous Architectures using Graph HyperNetworks

• URL: http://arxiv.org/abs/2201.08459v1
• Date: Thu, 20 Jan 2022 21:36:25 GMT
• Title: Federated Learning with Heterogeneous Architectures using Graph HyperNetworks
• Authors: Or Litany, Haggai Maron, David Acuna, Jan Kautz, Gal Chechik, Sanja Fidler
• Abstract summary: We propose a new FL framework that accommodates heterogeneous client architecture by adopting a graph hypernetwork for parameter sharing. Unlike existing solutions, our framework does not limit the clients to share the same architecture type, makes no use of external data and does not require clients to disclose their model architecture.
• Score: 154.60662664160333
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Standard Federated Learning (FL) techniques are limited to clients with identical network architectures. This restricts potential use-cases like cross-platform training or inter-organizational collaboration when both data privacy and architectural proprietary are required. We propose a new FL framework that accommodates heterogeneous client architecture by adopting a graph hypernetwork for parameter sharing. A property of the graph hyper network is that it can adapt to various computational graphs, thereby allowing meaningful parameter sharing across models. Unlike existing solutions, our framework does not limit the clients to share the same architecture type, makes no use of external data and does not require clients to disclose their model architecture. Compared with distillation-based and non-graph hypernetwork baselines, our method performs notably better on standard benchmarks. We additionally show encouraging generalization performance to unseen architectures.

Related papers

• Federated Learning with Flexible Architectures [12.800116749927266]
  This paper introduces Federated Learning with Flexible Architectures (FedFA), an FL training algorithm that allows clients to train models of different widths and depths. FedFA incorporates the layer grafting technique to align clients' local architectures with the largest network architecture in the FL system during model aggregation.
  arXiv  Detail & Related papers  (2024-06-14T09:44:46Z)
• FedSheafHN: Personalized Federated Learning on Graph-structured Data [22.825083541211168]
  We propose a model called FedSheafHN, which embeds each client's local subgraph into a server-constructed collaboration graph. Our model improves the integration and interpretation of complex client characteristics. It also has fast model convergence and effective new clients generalization.
  arXiv  Detail & Related papers  (2024-05-25T04:51:41Z)
• MAP: Model Aggregation and Personalization in Federated Learning with Incomplete Classes [49.22075916259368]
  In some real-world applications, data samples are usually distributed on local devices. In this paper, we focus on a special kind of Non-I.I.D. scene where clients own incomplete classes. Our proposed algorithm named MAP could simultaneously achieve the aggregation and personalization goals in FL.
  arXiv  Detail & Related papers  (2024-04-14T12:22:42Z)
• Serving Deep Learning Model in Relational Databases [70.53282490832189]
  Serving deep learning (DL) models on relational data has become a critical requirement across diverse commercial and scientific domains. We highlight three pivotal paradigms: The state-of-the-art DL-centric architecture offloads DL computations to dedicated DL frameworks. The potential UDF-centric architecture encapsulates one or more tensor computations into User Defined Functions (UDFs) within the relational database management system (RDBMS)
  arXiv  Detail & Related papers  (2023-10-07T06:01:35Z)
• Scalable Collaborative Learning via Representation Sharing [53.047460465980144]
  Federated learning (FL) and Split Learning (SL) are two frameworks that enable collaborative learning while keeping the data private (on device) In FL, each data holder trains a model locally and releases it to a central server for aggregation. In SL, the clients must release individual cut-layer activations (smashed data) to the server and wait for its response (during both inference and back propagation). In this work, we present a novel approach for privacy-preserving machine learning, where the clients collaborate via online knowledge distillation using a contrastive loss.
  arXiv  Detail & Related papers  (2022-11-20T10:49:22Z)
• Architecture Agnostic Federated Learning for Neural Networks [19.813602191888837]
  This work introduces a novel Federated Heterogeneous Neural Networks (FedHeNN) framework. FedHeNN allows each client to build a personalised model without enforcing a common architecture across clients. The key idea of FedHeNN is to use the instance-level representations obtained from peer clients to guide the simultaneous training on each client.
  arXiv  Detail & Related papers  (2022-02-15T22:16:06Z)
• SPIDER: Searching Personalized Neural Architecture for Federated Learning [17.61748275091843]
  Federated learning (FL) assists machine learning when data cannot be shared with a centralized server due to privacy and regulatory restrictions. Recent advancements in FL use predefined architecture-based learning for all the clients. We introduce SPIDER, an algorithmic framework that aims to Search Personalized neural architecture for federated learning.
  arXiv  Detail & Related papers  (2021-12-27T23:42:15Z)
• Network Graph Based Neural Architecture Search [57.78724765340237]
  We search neural network by rewiring the corresponding graph and predict the architecture performance by graph properties. Because we do not perform machine learning over the entire graph space, the searching process is remarkably efficient.
  arXiv  Detail & Related papers  (2021-12-15T00:12:03Z)
• Dataless Model Selection with the Deep Frame Potential [45.16941644841897]
  We quantify networks by their intrinsic capacity for unique and robust representations. We propose the deep frame potential: a measure of coherence that is approximately related to representation stability but has minimizers that depend only on network structure. We validate its use as a criterion for model selection and demonstrate correlation with generalization error on a variety of common residual and densely connected network architectures.
  arXiv  Detail & Related papers  (2020-03-30T23:27:25Z)
• Tensor Graph Convolutional Networks for Multi-relational and Robust Learning [74.05478502080658]
  This paper introduces a tensor-graph convolutional network (TGCN) for scalable semi-supervised learning (SSL) from data associated with a collection of graphs, that are represented by a tensor. The proposed architecture achieves markedly improved performance relative to standard GCNs, copes with state-of-the-art adversarial attacks, and leads to remarkable SSL performance over protein-to-protein interaction networks.
  arXiv  Detail & Related papers  (2020-03-15T02:33:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.