Partial Knowledge Distillation for Alleviating the Inherent Inter-Class Discrepancy in Federated Learning

• URL: http://arxiv.org/abs/2411.15403v1
• Date: Sat, 23 Nov 2024 01:16:46 GMT
• Title: Partial Knowledge Distillation for Alleviating the Inherent Inter-Class Discrepancy in Federated Learning
• Authors: Xiaoyu Gan, Xizi Chen, Jingyang Zhu, Xiaomeng Wang, Jingbo Jiang, Chi-Ying Tsui,
• Abstract summary: We observe that weak classes consistently exist even for class-balanced learning. The inherent inter-class accuracy discrepancy can reach over 36.9% for federated learning on the FashionMNIST and CIFAR-10 datasets. We propose a class-specific partial knowledge distillation method to improve the model's classification accuracy for weak classes.
• Score: 2.395881636777087
• License:
• Abstract: Substantial efforts have been devoted to alleviating the impact of the long-tailed class distribution in federated learning. In this work, we observe an interesting phenomenon that weak classes consistently exist even for class-balanced learning. These weak classes, different from the minority classes in the previous works, are inherent to data and remain fairly consistent for various network structures and learning paradigms. The inherent inter-class accuracy discrepancy can reach over 36.9% for federated learning on the FashionMNIST and CIFAR-10 datasets, even when the class distribution is balanced both globally and locally. In this study, we empirically analyze the potential reason for this phenomenon. Furthermore, a class-specific partial knowledge distillation method is proposed to improve the model's classification accuracy for weak classes. In this approach, knowledge transfer is initiated upon the occurrence of specific misclassifications within certain weak classes. Experimental results show that the accuracy of weak classes can be improved by 10.7%, reducing the inherent interclass discrepancy effectively.

Related papers

• CLOSER: Towards Better Representation Learning for Few-Shot Class-Incremental Learning [52.63674911541416]
  Few-shot class-incremental learning (FSCIL) faces several challenges, such as overfitting and forgetting. Our primary focus is representation learning on base classes to tackle the unique challenge of FSCIL. We find that trying to secure the spread of features within a more confined feature space enables the learned representation to strike a better balance between transferability and discriminability.
  arXiv  Detail & Related papers  (2024-10-08T02:23:16Z)
• Addressing Imbalance for Class Incremental Learning in Medical Image Classification [14.242875524728495]
  We introduce two plug-in methods to mitigate the adverse effects of imbalance. First, we propose a CIL-balanced classification loss to mitigate the classification bias toward majority classes. Second, we propose a distribution margin loss that not only alleviates the inter-class overlap in embedding space but also enforces the intra-class compactness.
  arXiv  Detail & Related papers  (2024-07-18T17:59:44Z)
• Learning Confidence Bounds for Classification with Imbalanced Data [42.690254618937196]
  We propose a novel framework that leverages learning theory and concentration inequalities to overcome the shortcomings of traditional solutions. Our method can effectively adapt to the varying degrees of imbalance across different classes, resulting in more robust and reliable classification outcomes.
  arXiv  Detail & Related papers  (2024-07-16T16:02:27Z)
• Exploring Vacant Classes in Label-Skewed Federated Learning [113.65301899666645]
  Label skews, characterized by disparities in local label distribution across clients, pose a significant challenge in federated learning. This paper introduces FedVLS, a novel approach to label-skewed federated learning that integrates vacant-class distillation and logit suppression simultaneously.
  arXiv  Detail & Related papers  (2024-01-04T16:06:31Z)
• Understanding the Detrimental Class-level Effects of Data Augmentation [63.1733767714073]
  achieving optimal average accuracy comes at the cost of significantly hurting individual class accuracy by as much as 20% on ImageNet. We present a framework for understanding how DA interacts with class-level learning dynamics. We show that simple class-conditional augmentation strategies improve performance on the negatively affected classes.
  arXiv  Detail & Related papers  (2023-12-07T18:37:43Z)
• Towards Fast and Stable Federated Learning: Confronting Heterogeneity via Knowledge Anchor [18.696420390977863]
  This paper systematically analyzes the forgetting degree of each class during local training across different communication rounds. Motivated by these findings, we propose a novel and straightforward algorithm called Federated Knowledge Anchor (FedKA)
  arXiv  Detail & Related papers  (2023-12-05T01:12:56Z)
• A Unified Generalization Analysis of Re-Weighting and Logit-Adjustment for Imbalanced Learning [129.63326990812234]
  We propose a technique named data-dependent contraction to capture how modified losses handle different classes. On top of this technique, a fine-grained generalization bound is established for imbalanced learning, which helps reveal the mystery of re-weighting and logit-adjustment.
  arXiv  Detail & Related papers  (2023-10-07T09:15:08Z)
• Prototype-Anchored Learning for Learning with Imperfect Annotations [83.7763875464011]
  It is challenging to learn unbiased classification models from imperfectly annotated datasets. We propose a prototype-anchored learning (PAL) method, which can be easily incorporated into various learning-based classification schemes. We verify the effectiveness of PAL on class-imbalanced learning and noise-tolerant learning by extensive experiments on synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2022-06-23T10:25:37Z)
• Fairness-aware Class Imbalanced Learning [57.45784950421179]
  We evaluate long-tail learning methods for tweet sentiment and occupation classification. We extend a margin-loss based approach with methods to enforce fairness.
  arXiv  Detail & Related papers  (2021-09-21T22:16:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.