Communication-Efficient Vertical Federated Learning with Limited Overlapping Samples

• URL: http://arxiv.org/abs/2303.16270v2
• Date: Thu, 30 Mar 2023 00:42:31 GMT
• Title: Communication-Efficient Vertical Federated Learning with Limited Overlapping Samples
• Authors: Jingwei Sun, Ziyue Xu, Dong Yang, Vishwesh Nath, Wenqi Li, Can Zhao, Daguang Xu, Yiran Chen, Holger R. Roth
• Abstract summary: We propose a vertical federated learning (VFL) framework called textbfone-shot VFL. In our proposed framework, the clients only need to communicate with the server once or only a few times. Our methods can improve the accuracy by more than 46.5% and reduce the communication cost by more than 330$times$ compared with state-of-the-art VFL methods.
• Score: 34.576230628844506
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated learning is a popular collaborative learning approach that enables clients to train a global model without sharing their local data. Vertical federated learning (VFL) deals with scenarios in which the data on clients have different feature spaces but share some overlapping samples. Existing VFL approaches suffer from high communication costs and cannot deal efficiently with limited overlapping samples commonly seen in the real world. We propose a practical vertical federated learning (VFL) framework called \textbf{one-shot VFL} that can solve the communication bottleneck and the problem of limited overlapping samples simultaneously based on semi-supervised learning. We also propose \textbf{few-shot VFL} to improve the accuracy further with just one more communication round between the server and the clients. In our proposed framework, the clients only need to communicate with the server once or only a few times. We evaluate the proposed VFL framework on both image and tabular datasets. Our methods can improve the accuracy by more than 46.5\% and reduce the communication cost by more than 330$\times$ compared with state-of-the-art VFL methods when evaluated on CIFAR-10. Our code will be made publicly available at \url{https://nvidia.github.io/NVFlare/research/one-shot-vfl}.

Related papers

• Vertical Federated Learning with Missing Features During Training and Inference [37.44022318612869]
  We propose a vertical federated learning method for efficient training and inference of neural network-based models. We show that our method achieves linear convergence to a neighborhood of the optimum even in the absence of missing features.
  arXiv  Detail & Related papers  (2024-10-29T22:09:31Z)
• Communication-efficient Vertical Federated Learning via Compressed Error Feedback [24.32409923443071]
  Lossy compression is commonly used on the information communicated between the server and clients during training. In horizontal FL, each subset of a training subset holds a subset of that subset of information. We propose a training method for vertical FL where each client holds a subset of that subset of information. Our method converges linearly when the objective function satisfies the Polyak-Lojasiewicz inequality.
  arXiv  Detail & Related papers  (2024-06-20T15:40:38Z)
• SpaFL: Communication-Efficient Federated Learning with Sparse Models and Low computational Overhead [75.87007729801304]
  SpaFL: a communication-efficient FL framework is proposed to optimize sparse model structures with low computational overhead. To optimize the pruning process itself, only thresholds are communicated between a server and clients instead of parameters. Global thresholds are used to update model parameters by extracting aggregated parameter importance.
  arXiv  Detail & Related papers  (2024-06-01T13:10:35Z)
• Prune at the Clients, Not the Server: Accelerated Sparse Training in Federated Learning [56.21666819468249]
  Resource constraints of clients and communication costs pose major problems for training large models in Federated Learning. We introduce Sparse-ProxSkip, which combines training and acceleration in a sparse setting. We demonstrate the good performance of Sparse-ProxSkip in extensive experiments.
  arXiv  Detail & Related papers  (2024-05-31T05:21:12Z)
• Achieving Dimension-Free Communication in Federated Learning via Zeroth-Order Optimization [15.73877955614998]
  This paper presents a novel communication algorithm - DeComFL - which reduces the communication cost from $mathscrO(d)$ to $mathscrO(d)$ by transmitting only a constant number of scalar values between clients. Empirical evaluations, encompassing both classic deep learning training and large language model fine-tuning, demonstrate significant reductions in communication overhead.
  arXiv  Detail & Related papers  (2024-05-24T18:07:05Z)
• Communication Efficient ConFederated Learning: An Event-Triggered SAGA Approach [67.27031215756121]
  Federated learning (FL) is a machine learning paradigm that targets model training without gathering the local data over various data sources. Standard FL, which employs a single server, can only support a limited number of users, leading to degraded learning capability. In this work, we consider a multi-server FL framework, referred to as emphConfederated Learning (CFL) in order to accommodate a larger number of users.
  arXiv  Detail & Related papers  (2024-02-28T03:27:10Z)
• Tunable Soft Prompts are Messengers in Federated Learning [55.924749085481544]
  Federated learning (FL) enables multiple participants to collaboratively train machine learning models using decentralized data sources. The lack of model privacy protection in FL becomes an unneglectable challenge. We propose a novel FL training approach that accomplishes information exchange among participants via tunable soft prompts.
  arXiv  Detail & Related papers  (2023-11-12T11:01:10Z)
• Knowledge-Enhanced Semi-Supervised Federated Learning for Aggregating Heterogeneous Lightweight Clients in IoT [34.128674870180596]
  Federated learning (FL) enables multiple clients to train models collaboratively without sharing local data. We propose pFedKnow, which generates lightweight personalized client models via neural network pruning techniques to reduce communication cost. Experiment results on both image and text datasets show that the proposed pFedKnow outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2023-03-05T13:19:10Z)
• Scalable Collaborative Learning via Representation Sharing [53.047460465980144]
  Federated learning (FL) and Split Learning (SL) are two frameworks that enable collaborative learning while keeping the data private (on device) In FL, each data holder trains a model locally and releases it to a central server for aggregation. In SL, the clients must release individual cut-layer activations (smashed data) to the server and wait for its response (during both inference and back propagation). In this work, we present a novel approach for privacy-preserving machine learning, where the clients collaborate via online knowledge distillation using a contrastive loss.
  arXiv  Detail & Related papers  (2022-11-20T10:49:22Z)
• SemiFL: Communication Efficient Semi-Supervised Federated Learning with Unlabeled Clients [34.24028216079336]
  We propose a new Federated Learning framework referred to as SemiFL. In SemiFL, clients have completely unlabeled data, while the server has a small amount of labeled data. We demonstrate various efficient strategies of SemiFL that enhance learning performance.
  arXiv  Detail & Related papers  (2021-06-02T19:22:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.