Not all Minorities are Equal: Empty-Class-Aware Distillation for
Heterogeneous Federated Learning
- URL: http://arxiv.org/abs/2401.02329v1
- Date: Thu, 4 Jan 2024 16:06:31 GMT
- Title: Not all Minorities are Equal: Empty-Class-Aware Distillation for
Heterogeneous Federated Learning
- Authors: Kuangpu Guo, Yuhe Ding, Jian Liang, Ran He, Zilei Wang, Tieniu Tan
- Abstract summary: FedED integrates empty-class distillation and logit suppression simultaneously.
It addresses misclassifications in minority classes that may be biased toward majority classes.
- Score: 120.42853706967188
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Data heterogeneity, characterized by disparities in local data distribution
across clients, poses a significant challenge in federated learning.
Substantial efforts have been devoted to addressing the heterogeneity in local
label distribution. As minority classes suffer from worse accuracy due to
overfitting on local imbalanced data, prior methods often incorporate
class-balanced learning techniques during local training. Despite the improved
mean accuracy across all classes, we observe that empty classes-referring to
categories absent from a client's data distribution-are still not well
recognized. This paper introduces FedED, a novel approach in heterogeneous
federated learning that integrates both empty-class distillation and logit
suppression simultaneously. Specifically, empty-class distillation leverages
knowledge distillation during local training on each client to retain essential
information related to empty classes from the global model. Moreover, logit
suppression directly penalizes network logits for non-label classes,
effectively addressing misclassifications in minority classes that may be
biased toward majority classes. Extensive experiments validate the efficacy of
FedED, surpassing previous state-of-the-art methods across diverse datasets
with varying degrees of label distribution shift.
Related papers
- GeT: Generative Target Structure Debiasing for Domain Adaptation [67.17025068995835]
Domain adaptation (DA) aims to transfer knowledge from a fully labeled source to a scarcely labeled or totally unlabeled target under domain shift.
Recently, semi-supervised learning-based (SSL) techniques that leverage pseudo labeling have been increasingly used in DA.
In this paper, we propose GeT that learns a non-bias target embedding distribution with high quality pseudo labels.
arXiv Detail & Related papers (2023-08-20T08:52:43Z) - How Robust is Your Fairness? Evaluating and Sustaining Fairness under
Unseen Distribution Shifts [107.72786199113183]
We propose a novel fairness learning method termed CUrvature MAtching (CUMA)
CUMA achieves robust fairness generalizable to unseen domains with unknown distributional shifts.
We evaluate our method on three popular fairness datasets.
arXiv Detail & Related papers (2022-07-04T02:37:50Z) - Throwing Away Data Improves Worst-Class Error in Imbalanced
Classification [36.91428748713018]
Class imbalances pervade classification problems, yet their treatment differs in theory and practice.
We take on the challenge of developing learning theory able to describe the worst-class error of classifiers over linearly-separable data.
arXiv Detail & Related papers (2022-05-23T23:43:18Z) - Imbalanced Classification via Explicit Gradient Learning From Augmented
Data [0.0]
We propose a novel deep meta-learning technique to augment a given imbalanced dataset with new minority instances.
The advantage of the proposed method is demonstrated on synthetic and real-world datasets with various imbalance ratios.
arXiv Detail & Related papers (2022-02-21T22:16:50Z) - Online Fairness-Aware Learning with Imbalanced Data Streams [9.481178205985396]
We propose ours, an online fairness-aware approach that maintains a valid and fair classifier over the stream.
oursis an online boosting approach that changes the training distribution in an online fashion by monitoring stream's class imbalance.
Experiments on 8 real-world and 1 synthetic datasets demonstrate the superiority of our method over state-of-the-art fairness-aware stream approaches.
arXiv Detail & Related papers (2021-08-13T13:31:42Z) - Fair-Net: A Network Architecture For Reducing Performance Disparity
Between Identifiable Sub-Populations [0.522145960878624]
We introduce Fair-Net, a multitask neural network architecture that improves both classification accuracy and probability calibration across identifiable sub-populations.
Empirical studies with three real world benchmark datasets demonstrate that Fair-Net improves classification and calibration performance.
arXiv Detail & Related papers (2021-06-01T18:26:08Z) - PLM: Partial Label Masking for Imbalanced Multi-label Classification [59.68444804243782]
Neural networks trained on real-world datasets with long-tailed label distributions are biased towards frequent classes and perform poorly on infrequent classes.
We propose a method, Partial Label Masking (PLM), which utilizes this ratio during training.
Our method achieves strong performance when compared to existing methods on both multi-label (MultiMNIST and MSCOCO) and single-label (imbalanced CIFAR-10 and CIFAR-100) image classification datasets.
arXiv Detail & Related papers (2021-05-22T18:07:56Z) - Fairness in Semi-supervised Learning: Unlabeled Data Help to Reduce
Discrimination [53.3082498402884]
A growing specter in the rise of machine learning is whether the decisions made by machine learning models are fair.
We present a framework of fair semi-supervised learning in the pre-processing phase, including pseudo labeling to predict labels for unlabeled data.
A theoretical decomposition analysis of bias, variance and noise highlights the different sources of discrimination and the impact they have on fairness in semi-supervised learning.
arXiv Detail & Related papers (2020-09-25T05:48:56Z) - M2m: Imbalanced Classification via Major-to-minor Translation [79.09018382489506]
In most real-world scenarios, labeled training datasets are highly class-imbalanced, where deep neural networks suffer from generalizing to a balanced testing criterion.
In this paper, we explore a novel yet simple way to alleviate this issue by augmenting less-frequent classes via translating samples from more-frequent classes.
Our experimental results on a variety of class-imbalanced datasets show that the proposed method improves the generalization on minority classes significantly compared to other existing re-sampling or re-weighting methods.
arXiv Detail & Related papers (2020-04-01T13:21:17Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.