FedUV: Uniformity and Variance for Heterogeneous Federated Learning
- URL: http://arxiv.org/abs/2402.18372v2
- Date: Fri, 1 Mar 2024 21:53:26 GMT
- Title: FedUV: Uniformity and Variance for Heterogeneous Federated Learning
- Authors: Ha Min Son, Moon-Hyun Kim, Tai-Myoung Chung, Chao Huang, Xin Liu
- Abstract summary: Federated learning is a promising framework to train neural networks with widely distributed data.
Recent work has shown this is due to the final layer of the network being most prone to local bias.
We investigate the training dynamics of the classifier by applying SVD to the weights motivated by the observation that freezing weights results in constant singular values.
- Score: 5.9330433627374815
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Federated learning is a promising framework to train neural networks with
widely distributed data. However, performance degrades heavily with
heterogeneously distributed data. Recent work has shown this is due to the
final layer of the network being most prone to local bias, some finding success
freezing the final layer as an orthogonal classifier. We investigate the
training dynamics of the classifier by applying SVD to the weights motivated by
the observation that freezing weights results in constant singular values. We
find that there are differences when training in IID and non-IID settings.
Based on this finding, we introduce two regularization terms for local training
to continuously emulate IID settings: (1) variance in the dimension-wise
probability distribution of the classifier and (2) hyperspherical uniformity of
representations of the encoder. These regularizations promote local models to
act as if it were in an IID setting regardless of the local data distribution,
thus offsetting proneness to bias while being flexible to the data. On
extensive experiments in both label-shift and feature-shift settings, we verify
that our method achieves highest performance by a large margin especially in
highly non-IID cases in addition to being scalable to larger models and
datasets.
Related papers
- Robust training of implicit generative models for multivariate and heavy-tailed distributions with an invariant statistical loss [0.4249842620609682]
We build on the textitinvariant statistical loss (ISL) method introduced in citede2024training.
We extend it to handle heavy-tailed and multivariate data distributions.
We assess its performance in generative generative modeling and explore its potential as a pretraining technique for generative adversarial networks (GANs)
arXiv Detail & Related papers (2024-10-29T10:27:50Z) - Learning From Drift: Federated Learning on Non-IID Data via Drift
Regularization [11.813552364878868]
Federated learning algorithms perform reasonably well on independent and identically distributed (IID) data.
They suffer greatly from heterogeneous environments, i.e., Non-IID data.
We propose Learning from Drift (LfD), a novel method for effectively training the model in heterogeneous settings.
arXiv Detail & Related papers (2023-09-13T09:23:09Z) - Class-Balancing Diffusion Models [57.38599989220613]
Class-Balancing Diffusion Models (CBDM) are trained with a distribution adjustment regularizer as a solution.
Our method benchmarked the generation results on CIFAR100/CIFAR100LT dataset and shows outstanding performance on the downstream recognition task.
arXiv Detail & Related papers (2023-04-30T20:00:14Z) - Experimenting with Normalization Layers in Federated Learning on non-IID
scenarios [1.2599533416395765]
Federated Learning (FL) has been emerging as a method for privacy-preserving pooling of datasets employing collaborative training from different institutions.
One critical performance challenge of FL is operating on datasets not independently and identically distributed (non-IID) among the federation participants.
We benchmark five different normalization layers for training Neural Networks (NNs), two families of non-IID data skew, and two datasets.
Results show that Batch Normalization, widely employed for centralized DL, is not the best choice for FL, whereas Group and Layer Normalization consistently outperform Batch Normalization.
arXiv Detail & Related papers (2023-03-19T11:20:43Z) - Class Balancing GAN with a Classifier in the Loop [58.29090045399214]
We introduce a novel theoretically motivated Class Balancing regularizer for training GANs.
Our regularizer makes use of the knowledge from a pre-trained classifier to ensure balanced learning of all the classes in the dataset.
We demonstrate the utility of our regularizer in learning representations for long-tailed distributions via achieving better performance than existing approaches over multiple datasets.
arXiv Detail & Related papers (2021-06-17T11:41:30Z) - Decentralized Local Stochastic Extra-Gradient for Variational
Inequalities [125.62877849447729]
We consider distributed variational inequalities (VIs) on domains with the problem data that is heterogeneous (non-IID) and distributed across many devices.
We make a very general assumption on the computational network that covers the settings of fully decentralized calculations.
We theoretically analyze its convergence rate in the strongly-monotone, monotone, and non-monotone settings.
arXiv Detail & Related papers (2021-06-15T17:45:51Z) - No Fear of Heterogeneity: Classifier Calibration for Federated Learning
with Non-IID Data [78.69828864672978]
A central challenge in training classification models in the real-world federated system is learning with non-IID data.
We propose a novel and simple algorithm called Virtual Representations (CCVR), which adjusts the classifier using virtual representations sampled from an approximated ssian mixture model.
Experimental results demonstrate that CCVR state-of-the-art performance on popular federated learning benchmarks including CIFAR-10, CIFAR-100, and CINIC-10.
arXiv Detail & Related papers (2021-06-09T12:02:29Z) - Unsupervised Feature Learning by Cross-Level Instance-Group
Discrimination [68.83098015578874]
We integrate between-instance similarity into contrastive learning, not directly by instance grouping, but by cross-level discrimination.
CLD effectively brings unsupervised learning closer to natural data and real-world applications.
New state-of-the-art on self-supervision, semi-supervision, and transfer learning benchmarks, and beats MoCo v2 and SimCLR on every reported performance.
arXiv Detail & Related papers (2020-08-09T21:13:13Z) - Robust Federated Learning: The Case of Affine Distribution Shifts [41.27887358989414]
We develop a robust federated learning algorithm that achieves satisfactory performance against distribution shifts in users' samples.
We show that an affine distribution shift indeed suffices to significantly decrease the performance of the learnt classifier in a new test user.
arXiv Detail & Related papers (2020-06-16T03:43:59Z) - Diversity inducing Information Bottleneck in Model Ensembles [73.80615604822435]
In this paper, we target the problem of generating effective ensembles of neural networks by encouraging diversity in prediction.
We explicitly optimize a diversity inducing adversarial loss for learning latent variables and thereby obtain diversity in the output predictions necessary for modeling multi-modal data.
Compared to the most competitive baselines, we show significant improvements in classification accuracy, under a shift in the data distribution.
arXiv Detail & Related papers (2020-03-10T03:10:41Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.