A Unified Generalization Analysis of Re-Weighting and Logit-Adjustment for Imbalanced Learning

• URL: http://arxiv.org/abs/2310.04752v1
• Date: Sat, 7 Oct 2023 09:15:08 GMT
• Title: A Unified Generalization Analysis of Re-Weighting and Logit-Adjustment for Imbalanced Learning
• Authors: Zitai Wang and Qianqian Xu and Zhiyong Yang and Yuan He and Xiaochun Cao and Qingming Huang
• Abstract summary: We propose a technique named data-dependent contraction to capture how modified losses handle different classes. On top of this technique, a fine-grained generalization bound is established for imbalanced learning, which helps reveal the mystery of re-weighting and logit-adjustment.
• Score: 129.63326990812234
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Real-world datasets are typically imbalanced in the sense that only a few classes have numerous samples, while many classes are associated with only a few samples. As a result, a na\"ive ERM learning process will be biased towards the majority classes, making it difficult to generalize to the minority classes. To address this issue, one simple but effective approach is to modify the loss function to emphasize the learning on minority classes, such as re-weighting the losses or adjusting the logits via class-dependent terms. However, existing generalization analysis of such losses is still coarse-grained and fragmented, failing to explain some empirical results. To bridge this gap, we propose a novel technique named data-dependent contraction to capture how these modified losses handle different classes. On top of this technique, a fine-grained generalization bound is established for imbalanced learning, which helps reveal the mystery of re-weighting and logit-adjustment in a unified manner. Furthermore, a principled learning algorithm is developed based on the theoretical insights. Finally, the empirical results on benchmark datasets not only validate the theoretical results but also demonstrate the effectiveness of the proposed method.

Related papers

• Rethinking Classifier Re-Training in Long-Tailed Recognition: A Simple Logits Retargeting Approach [102.0769560460338]
  We develop a simple logits approach (LORT) without the requirement of prior knowledge of the number of samples per class. Our method achieves state-of-the-art performance on various imbalanced datasets, including CIFAR100-LT, ImageNet-LT, and iNaturalist 2018.
  arXiv  Detail & Related papers  (2024-03-01T03:27:08Z)
• On the Dynamics Under the Unhinged Loss and Beyond [104.49565602940699]
  We introduce the unhinged loss, a concise loss function, that offers more mathematical opportunities to analyze closed-form dynamics. The unhinged loss allows for considering more practical techniques, such as time-vary learning rates and feature normalization.
  arXiv  Detail & Related papers  (2023-12-13T02:11:07Z)
• Towards a Unified Analysis of Kernel-based Methods Under Covariate Shift [18.240776405802205]
  We propose a unified analysis of general nonparametric methods in a reproducing kernel Hilbert space. Our theoretical results are established for a general loss belonging to a rich loss function family. Our results concur with the optimal results in literature where the squared loss is used.
  arXiv  Detail & Related papers  (2023-10-12T11:33:15Z)
• CUDA: Curriculum of Data Augmentation for Long-Tailed Recognition [10.441880303257468]
  Class imbalance problems frequently occur in real-world tasks. To mitigate this problem, many approaches have aimed to balance among given classes by re-weighting or re-sampling training samples. These re-balancing methods increase the impact of minority classes and reduce the influence of majority classes on the output of models. Several methods have been developed that increase the representations of minority samples by the features of the majority samples.
  arXiv  Detail & Related papers  (2023-02-10T20:30:22Z)
• Class-Imbalanced Complementary-Label Learning via Weighted Loss [8.934943507699131]
  Complementary-label learning (CLL) is widely used in weakly supervised classification. It faces a significant challenge in real-world datasets when confronted with class-imbalanced training samples. We propose a novel problem setting that enables learning from class-imbalanced complementary labels for multi-class classification.
  arXiv  Detail & Related papers  (2022-09-28T16:02:42Z)
• Prototype-Anchored Learning for Learning with Imperfect Annotations [83.7763875464011]
  It is challenging to learn unbiased classification models from imperfectly annotated datasets. We propose a prototype-anchored learning (PAL) method, which can be easily incorporated into various learning-based classification schemes. We verify the effectiveness of PAL on class-imbalanced learning and noise-tolerant learning by extensive experiments on synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2022-06-23T10:25:37Z)
• Imbalanced Classification via Explicit Gradient Learning From Augmented Data [0.0]
  We propose a novel deep meta-learning technique to augment a given imbalanced dataset with new minority instances. The advantage of the proposed method is demonstrated on synthetic and real-world datasets with various imbalance ratios.
  arXiv  Detail & Related papers  (2022-02-21T22:16:50Z)
• Learning explanations that are hard to vary [75.30552491694066]
  We show that averaging across examples can favor memorization and patchwork' solutions that sew together different strategies. We then propose and experimentally validate a simple alternative algorithm based on a logical AND.
  arXiv  Detail & Related papers  (2020-09-01T10:17:48Z)
• M2m: Imbalanced Classification via Major-to-minor Translation [79.09018382489506]
  In most real-world scenarios, labeled training datasets are highly class-imbalanced, where deep neural networks suffer from generalizing to a balanced testing criterion. In this paper, we explore a novel yet simple way to alleviate this issue by augmenting less-frequent classes via translating samples from more-frequent classes. Our experimental results on a variety of class-imbalanced datasets show that the proposed method improves the generalization on minority classes significantly compared to other existing re-sampling or re-weighting methods.
  arXiv  Detail & Related papers  (2020-04-01T13:21:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.