Related papers: Model-Contrastive Federated Learning

Model-Contrastive Federated Learning

URL: http://arxiv.org/abs/2103.16257v1
Date: Tue, 30 Mar 2021 11:16:57 GMT
Title: Model-Contrastive Federated Learning
Authors: Qinbin Li, Bingsheng He, Dawn Song
Abstract summary: Federated learning enables multiple parties to collaboratively train a machine learning model without communicating their local data. We propose MOON: model-contrastive federated learning. Our experiments show that MOON significantly outperforms the other state-of-the-art federated learning algorithms on various image classification tasks.
Score: 92.9075661456444
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Abstract: Federated learning enables multiple parties to collaboratively train a machine learning model without communicating their local data. A key challenge in federated learning is to handle the heterogeneity of local data distribution across parties. Although many studies have been proposed to address this challenge, we find that they fail to achieve high performance in image datasets with deep learning models. In this paper, we propose MOON: model-contrastive federated learning. MOON is a simple and effective federated learning framework. The key idea of MOON is to utilize the similarity between model representations to correct the local training of individual parties, i.e., conducting contrastive learning in model-level. Our extensive experiments show that MOON significantly outperforms the other state-of-the-art federated learning algorithms on various image classification tasks.

Related papers

Leveraging Foundation Models for Multi-modal Federated Learning with Incomplete Modality [41.79433449873368]
We propose a novel multi-modal federated learning method, Federated Multi-modal contrastiVe training with Pre-trained completion (FedMVP) FedMVP integrates the large-scale pre-trained models to enhance the federated training. We demonstrate that the model achieves superior performance over two real-world image-text classification datasets.
arXiv Detail & Related papers (2024-06-16T19:18:06Z)
Federated Model Heterogeneous Matryoshka Representation Learning [33.04969829305812]
Model heterogeneous federated learning (MteroFL) enables FL clients to collaboratively train models with heterogeneous structures in a distributed fashion. Existing methods rely on training loss to transfer knowledge between a MteroFL server and a client model. We propose a new representation approach for supervised learning tasks using Matryoshka models.
arXiv Detail & Related papers (2024-06-01T16:37:08Z)
Personalized Federated Learning with Contextual Modulation and Meta-Learning [2.7716102039510564]
Federated learning has emerged as a promising approach for training machine learning models on decentralized data sources. We propose a novel framework that combines federated learning with meta-learning techniques to enhance both efficiency and generalization capabilities.
arXiv Detail & Related papers (2023-12-23T08:18:22Z)
Non-Invasive Fairness in Learning through the Lens of Data Drift [88.37640805363317]
We show how to improve the fairness of Machine Learning models without altering the data or the learning algorithm. We use a simple but key insight: the divergence of trends between different populations, and, consecutively, between a learned model and minority populations, is analogous to data drift. We explore two strategies (model-splitting and reweighing) to resolve this drift, aiming to improve the overall conformance of models to the underlying data.
arXiv Detail & Related papers (2023-03-30T17:30:42Z)
Scaling Vision-Language Models with Sparse Mixture of Experts [128.0882767889029]
We show that mixture-of-experts (MoE) techniques can achieve state-of-the-art performance on a range of benchmarks over dense models of equivalent computational cost. Our research offers valuable insights into stabilizing the training of MoE models, understanding the impact of MoE on model interpretability, and balancing the trade-offs between compute performance when scaling vision-language models.
arXiv Detail & Related papers (2023-03-13T16:00:31Z)
Dataless Knowledge Fusion by Merging Weights of Language Models [51.8162883997512]
Fine-tuning pre-trained language models has become the prevalent paradigm for building downstream NLP models. This creates a barrier to fusing knowledge across individual models to yield a better single model. We propose a dataless knowledge fusion method that merges models in their parameter space.
arXiv Detail & Related papers (2022-12-19T20:46:43Z)
FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learning [87.08902493524556]
Federated learning(FL) has recently attracted increasing attention from academia and industry. We propose FedDM to build the global training objective from multiple local surrogate functions. In detail, we construct synthetic sets of data on each client to locally match the loss landscape from original data.
arXiv Detail & Related papers (2022-07-20T04:55:18Z)
FedProc: Prototypical Contrastive Federated Learning on Non-IID data [24.1906520295278]
Federated learning allows multiple clients to collaborate to train deep learning models while keeping the training data locally. We propose FedProc: prototypical contrastive federated learning. We show that FedProc improves the accuracy by $1.6%sim7.9%$ with acceptable computation cost.
arXiv Detail & Related papers (2021-09-25T04:32:23Z)
Federated Residual Learning [53.77128418049985]
We study a new form of federated learning where the clients train personalized local models and make predictions jointly with the server-side shared model. Using this new federated learning framework, the complexity of the central shared model can be minimized while still gaining all the performance benefits that joint training provides.
arXiv Detail & Related papers (2020-03-28T19:55:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.