ConDistFL: Conditional Distillation for Federated Learning from Partially Annotated Data

• URL: http://arxiv.org/abs/2308.04070v1
• Date: Tue, 8 Aug 2023 06:07:49 GMT
• Title: ConDistFL: Conditional Distillation for Federated Learning from Partially Annotated Data
• Authors: Pochuan Wang, Chen Shen, Weichung Wang, Masahiro Oda, Chiou-Shann Fuh, Kensaku Mori, Holger R. Roth
• Abstract summary: "ConDistFL" is a framework to combine Federated Learning (FL) with knowledge distillation. We validate our framework on four distinct partially annotated abdominal CT datasets from the MSD and KiTS19 challenges. Our ablation study suggests that ConDistFL can perform well without frequent aggregation, reducing the communication cost of FL.
• Score: 5.210280120905009
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Developing a generalized segmentation model capable of simultaneously delineating multiple organs and diseases is highly desirable. Federated learning (FL) is a key technology enabling the collaborative development of a model without exchanging training data. However, the limited access to fully annotated training data poses a major challenge to training generalizable models. We propose "ConDistFL", a framework to solve this problem by combining FL with knowledge distillation. Local models can extract the knowledge of unlabeled organs and tumors from partially annotated data from the global model with an adequately designed conditional probability representation. We validate our framework on four distinct partially annotated abdominal CT datasets from the MSD and KiTS19 challenges. The experimental results show that the proposed framework significantly outperforms FedAvg and FedOpt baselines. Moreover, the performance on an external test dataset demonstrates superior generalizability compared to models trained on each dataset separately. Our ablation study suggests that ConDistFL can perform well without frequent aggregation, reducing the communication cost of FL. Our implementation will be available at https://github.com/NVIDIA/NVFlare/tree/dev/research/condist-fl.

Related papers

• Exploiting Label Skews in Federated Learning with Model Concatenation [39.38427550571378]
  Federated Learning (FL) has emerged as a promising solution to perform deep learning on different data owners without exchanging raw data. Among different non-IID types, label skews have been challenging and common in image classification and other tasks. We propose FedConcat, a simple and effective approach that degrades these local models as the base of the global model.
  arXiv  Detail & Related papers  (2023-12-11T10:44:52Z)
• Fake It Till Make It: Federated Learning with Consensus-Oriented Generation [52.82176415223988]
  We propose federated learning with consensus-oriented generation (FedCOG) FedCOG consists of two key components at the client side: complementary data generation and knowledge-distillation-based model training. Experiments on classical and real-world FL datasets show that FedCOG consistently outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-12-10T18:49:59Z)
• Adaptive Model Pruning and Personalization for Federated Learning over Wireless Networks [72.59891661768177]
  Federated learning (FL) enables distributed learning across edge devices while protecting data privacy. We consider a FL framework with partial model pruning and personalization to overcome these challenges. This framework splits the learning model into a global part with model pruning shared with all devices to learn data representations and a personalized part to be fine-tuned for a specific device.
  arXiv  Detail & Related papers  (2023-09-04T21:10:45Z)
• FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learning [87.08902493524556]
  Federated learning(FL) has recently attracted increasing attention from academia and industry. We propose FedDM to build the global training objective from multiple local surrogate functions. In detail, we construct synthetic sets of data on each client to locally match the loss landscape from original data.
  arXiv  Detail & Related papers  (2022-07-20T04:55:18Z)
• Federated Learning on Heterogeneous and Long-Tailed Data via Classifier Re-Training with Federated Features [24.679535905451758]
  Federated learning (FL) provides a privacy-preserving solution for distributed machine learning tasks. One challenging problem that severely damages the performance of FL models is the co-occurrence of data heterogeneity and long-tail distribution. We propose a novel privacy-preserving FL method for heterogeneous and long-tailed data via Federated Re-training with Federated Features (CReFF)
  arXiv  Detail & Related papers  (2022-04-28T10:35:11Z)
• Federated Learning from Only Unlabeled Data with Class-Conditional-Sharing Clients [98.22390453672499]
  Supervised federated learning (FL) enables multiple clients to share the trained model without sharing their labeled data. We propose federation of unsupervised learning (FedUL), where the unlabeled data are transformed into surrogate labeled data for each of the clients.
  arXiv  Detail & Related papers  (2022-04-07T09:12:00Z)
• Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning [86.59588262014456]
  Federated Learning (FL) is an emerging distributed learning paradigm under privacy constraint. We propose a data-free knowledge distillation method to fine-tune the global model in the server (FedFTG) Our FedFTG significantly outperforms the state-of-the-art (SOTA) FL algorithms and can serve as a strong plugin for enhancing FedAvg, FedProx, FedDyn, and SCAFFOLD.
  arXiv  Detail & Related papers  (2022-03-17T11:18:17Z)
• Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Data [8.935169114460663]
  This study develops a federated learning (FL) framework overcoming largely incremental communication costs. We propose a distillation-based semi-supervised FL algorithm that exchanges the outputs of local models among mobile devices. In DS-FL, the communication cost depends only on the output dimensions of the models and does not scale up according to the model size.
  arXiv  Detail & Related papers  (2020-08-14T03:47:27Z)
• Ensemble Distillation for Robust Model Fusion in Federated Learning [72.61259487233214]
  Federated Learning (FL) is a machine learning setting where many devices collaboratively train a machine learning model. In most of the current training schemes the central model is refined by averaging the parameters of the server model and the updated parameters from the client side. We propose ensemble distillation for model fusion, i.e. training the central classifier through unlabeled data on the outputs of the models from the clients.
  arXiv  Detail & Related papers  (2020-06-12T14:49:47Z)
• Federated learning with hierarchical clustering of local updates to improve training on non-IID data [3.3517146652431378]
  We show that learning a single joint model is often not optimal in the presence of certain types of non-iid data. We present a modification to FL by introducing a hierarchical clustering step (FL+HC) We show how FL+HC allows model training to converge in fewer communication rounds compared to FL without clustering.
  arXiv  Detail & Related papers  (2020-04-24T15:16:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.