FedEmb: A Vertical and Hybrid Federated Learning Algorithm using Network And Feature Embedding Aggregation

• URL: http://arxiv.org/abs/2312.00102v4
• Date: Wed, 10 Jan 2024 06:07:33 GMT
• Title: FedEmb: A Vertical and Hybrid Federated Learning Algorithm using Network And Feature Embedding Aggregation
• Authors: Fanfei Meng, Lele Zhang, Yu Chen, Yuxin Wang
• Abstract summary: Federated learning (FL) is an emerging paradigm for decentralized training of machine learning models on distributed clients. In this paper, we propose a generalized algorithm FedEmb, for modelling vertical and hybrid-based learning. The experimental results show that FedEmb is an effective method to tackle both split feature & subject space decentralized problems.
• Score: 24.78757412559944
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Federated learning (FL) is an emerging paradigm for decentralized training of machine learning models on distributed clients, without revealing the data to the central server. The learning scheme may be horizontal, vertical or hybrid (both vertical and horizontal). Most existing research work with deep neural network (DNN) modelling is focused on horizontal data distributions, while vertical and hybrid schemes are much less studied. In this paper, we propose a generalized algorithm FedEmb, for modelling vertical and hybrid DNN-based learning. The idea of our algorithm is characterised by higher inference accuracy, stronger privacy-preserving properties, and lower client-server communication bandwidth demands as compared with existing work. The experimental results show that FedEmb is an effective method to tackle both split feature & subject space decentralized problems, shows 0.3% to 4.2% inference accuracy improvement with limited privacy revealing for datasets stored in local clients, and reduces 88.9 % time complexity over vertical baseline method.

Related papers

• Hybrid FedGraph: An efficient hybrid federated learning algorithm using graph convolutional neural network [13.786989442742588]
  Federated learning is an emerging paradigm for decentralized training of machine learning models on distributed clients. We propose a graph convolutional neural network to capture feature-sharing information while learning features from a subset of clients. We also develop a simple but effective clustering algorithm that aggregates features produced by the deep neural networks of each client while preserving data privacy.
  arXiv  Detail & Related papers  (2024-04-15T04:02:39Z)
• FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup for Non-IID Data [54.81695390763957]
  Federated learning is an emerging distributed machine learning method. We propose a heterogeneous local variant of AMSGrad, named FedLALR, in which each client adjusts its learning rate. We show that our client-specified auto-tuned learning rate scheduling can converge and achieve linear speedup with respect to the number of clients.
  arXiv  Detail & Related papers  (2023-09-18T12:35:05Z)
• Graph Federated Learning Based on the Decentralized Framework [8.619889123184649]
  Graph-federated learning is mainly based on the classical federated learning framework i.e., the Client-Server framework. We introduce the decentralized framework to graph-federated learning. The proposed method is compared with FedAvg, Fedprox, GCFL, and GCFL+ to verify the effectiveness of the proposed method.
  arXiv  Detail & Related papers  (2023-07-19T07:40:51Z)
• Vertical Federated Learning over Cloud-RAN: Convergence Analysis and System Optimization [82.12796238714589]
  We propose a novel cloud radio access network (Cloud-RAN) based vertical FL system to enable fast and accurate model aggregation. We characterize the convergence behavior of the vertical FL algorithm considering both uplink and downlink transmissions. We establish a system optimization framework by joint transceiver and fronthaul quantization design, for which successive convex approximation and alternate convex search based system optimization algorithms are developed.
  arXiv  Detail & Related papers  (2023-05-04T09:26:03Z)
• Scalable Collaborative Learning via Representation Sharing [53.047460465980144]
  Federated learning (FL) and Split Learning (SL) are two frameworks that enable collaborative learning while keeping the data private (on device) In FL, each data holder trains a model locally and releases it to a central server for aggregation. In SL, the clients must release individual cut-layer activations (smashed data) to the server and wait for its response (during both inference and back propagation). In this work, we present a novel approach for privacy-preserving machine learning, where the clients collaborate via online knowledge distillation using a contrastive loss.
  arXiv  Detail & Related papers  (2022-11-20T10:49:22Z)
• Decentralized Training of Foundation Models in Heterogeneous Environments [77.47261769795992]
  Training foundation models, such as GPT-3 and PaLM, can be extremely expensive. We present the first study of training large foundation models with model parallelism in a decentralized regime over a heterogeneous network.
  arXiv  Detail & Related papers  (2022-06-02T20:19:51Z)
• Acceleration of Federated Learning with Alleviated Forgetting in Local Training [61.231021417674235]
  Federated learning (FL) enables distributed optimization of machine learning models while protecting privacy. We propose FedReg, an algorithm to accelerate FL with alleviated knowledge forgetting in the local training stage. Our experiments demonstrate that FedReg not only significantly improves the convergence rate of FL, especially when the neural network architecture is deep.
  arXiv  Detail & Related papers  (2022-03-05T02:31:32Z)
• A Vertical Federated Learning Framework for Horizontally Partitioned Labels [12.433809611989155]
  Most existing vertical federated learning methods have a strong assumption that at least one party holds the complete set of labels of all data samples. Existing vertical federated learning methods can only utilize partial labels, which may lead to inadequate model update in end-to-end backpropagation. We propose a novel vertical federated learning framework named Cascade Vertical Federated Learning (CVFL) to fully utilize all horizontally partitioned labels to train neural networks with privacy-preservation.
  arXiv  Detail & Related papers  (2021-06-18T11:10:11Z)
• Towards Heterogeneous Clients with Elastic Federated Learning [45.2715985913761]
  Federated learning involves training machine learning models over devices or data silos, such as edge processors or data warehouses, while keeping the data local. We propose Elastic Federated Learning (EFL), an unbiased algorithm to tackle the heterogeneity in the system. It is an efficient and effective algorithm that compresses both upstream and downstream communications.
  arXiv  Detail & Related papers  (2021-06-17T12:30:40Z)
• Semi-Decentralized Federated Edge Learning for Fast Convergence on Non-IID Data [14.269800282001464]
  Federated edge learning (FEEL) has emerged as an effective approach to reduce the large communication latency in Cloud-based machine learning solutions. We investigate a novel framework of FEEL, namely semi-decentralized federated edge learning (SD-FEEL) By allowing model aggregation across different edge clusters, SD-FEEL enjoys the benefit of FEEL in reducing the training latency.
  arXiv  Detail & Related papers  (2021-04-26T16:11:47Z)
• Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data [77.88594632644347]
  Decentralized training of deep learning models is a key element for enabling data privacy and on-device learning over networks. In realistic learning scenarios, the presence of heterogeneity across different clients' local datasets poses an optimization challenge. We propose a novel momentum-based method to mitigate this decentralized training difficulty.
  arXiv  Detail & Related papers  (2021-02-09T11:27:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.