Identifying Key Challenges of Hardness-Based Resampling

• URL: http://arxiv.org/abs/2504.07031v1
• Date: Wed, 09 Apr 2025 16:45:57 GMT
• Title: Identifying Key Challenges of Hardness-Based Resampling
• Authors: Pawel Pukowski, Venet Osmani,
• Abstract summary: Performance gap across classes remains a persistent challenge in machine learning.<n>One way to quantify class hardness is through sample complexity.<n>Harder classes need substantially more samples to achieve generalization.
• Score: 0.5678271181959529
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Performance gap across classes remains a persistent challenge in machine learning, often attributed to variations in class hardness. One way to quantify class hardness is through sample complexity - the minimum number of samples required to effectively learn a given class. Sample complexity theory suggests that class hardness is driven by differences in the amount of data required for generalization. That is, harder classes need substantially more samples to achieve generalization. Therefore, hardness-based resampling is a promising approach to mitigate these performance disparities. While resampling has been studied extensively in data-imbalanced settings, its impact on balanced datasets remains unexplored. This raises the fundamental question whether resampling is effective because it addresses data imbalance or hardness imbalance. We begin addressing this question by introducing class imbalance into balanced datasets and evaluate its effect on performance disparities. We oversample hard classes and undersample easy classes to bring hard classes closer to their sample complexity requirements while maintaining a constant dataset size for fairness. We estimate class-level hardness using the Area Under the Margin (AUM) hardness estimator and leverage it to compute resampling ratios. Using these ratios, we perform hardness-based resampling on the well-known CIFAR-10 and CIFAR-100 datasets. Contrary to theoretical expectations, our results show that hardness-based resampling does not meaningfully affect class-wise performance disparities. To explain this discrepancy, we conduct detailed analyses to identify key challenges unique to hardness-based imbalance, distinguishing it from traditional data-based imbalance. Our insights help explain why theoretical sample complexity expectations fail to translate into practical performance gains and we provide guidelines for future research.

Related papers

• SeMi: When Imbalanced Semi-Supervised Learning Meets Mining Hard Examples [54.760757107700755]
  Semi-Supervised Learning (SSL) can leverage abundant unlabeled data to boost model performance.<n>The class-imbalanced data distribution in real-world scenarios poses great challenges to SSL, resulting in performance degradation.<n>We propose a method that enhances the performance of Imbalanced Semi-Supervised Learning by Mining Hard Examples (SeMi)
  arXiv  Detail & Related papers  (2025-01-10T14:35:16Z)
• Conformal-in-the-Loop for Learning with Imbalanced Noisy Data [5.69777817429044]
  Class imbalance and label noise are pervasive in large-scale datasets.<n>Much of machine learning research assumes well-labeled, balanced data, which rarely reflects real world conditions.<n>We propose Conformal-in-the-Loop (CitL), a novel training framework that addresses both challenges with a conformal prediction-based approach.
  arXiv  Detail & Related papers  (2024-11-04T17:09:58Z)
• A Unified Generalization Analysis of Re-Weighting and Logit-Adjustment for Imbalanced Learning [129.63326990812234]
  We propose a technique named data-dependent contraction to capture how modified losses handle different classes. On top of this technique, a fine-grained generalization bound is established for imbalanced learning, which helps reveal the mystery of re-weighting and logit-adjustment.
  arXiv  Detail & Related papers  (2023-10-07T09:15:08Z)
• Simplicity Bias Leads to Amplified Performance Disparities [8.60453031364566]
  We show that SGD-trained models have a bias towards simplicity, leading them to prioritize learning a majority class. A model may prioritize any class or group of the dataset that it finds simple-at the expense of what it finds complex.
  arXiv  Detail & Related papers  (2022-12-13T15:24:41Z)
• Difficulty-Net: Learning to Predict Difficulty for Long-Tailed Recognition [5.977483447975081]
  We propose Difficulty-Net, which learns to predict the difficulty of classes using the model's performance in a meta-learning framework. We introduce two key concepts, namely the relative difficulty and the driver loss. Experiments on popular long-tailed datasets demonstrated the effectiveness of the proposed method.
  arXiv  Detail & Related papers  (2022-09-07T07:04:08Z)
• HardVis: Visual Analytics to Handle Instance Hardness Using Undersampling and Oversampling Techniques [48.82319198853359]
  HardVis is a visual analytics system designed to handle instance hardness mainly in imbalanced classification scenarios. Users can explore subsets of data from different perspectives to decide all those parameters. The efficacy and effectiveness of HardVis are demonstrated with a hypothetical usage scenario and a use case.
  arXiv  Detail & Related papers  (2022-03-29T17:04:16Z)
• Scale-Equivalent Distillation for Semi-Supervised Object Detection [57.59525453301374]
  Recent Semi-Supervised Object Detection (SS-OD) methods are mainly based on self-training, generating hard pseudo-labels by a teacher model on unlabeled data as supervisory signals. We analyze the challenges these methods meet with the empirical experiment results. We introduce a novel approach, Scale-Equivalent Distillation (SED), which is a simple yet effective end-to-end knowledge distillation framework robust to large object size variance and class imbalance.
  arXiv  Detail & Related papers  (2022-03-23T07:33:37Z)
• CMW-Net: Learning a Class-Aware Sample Weighting Mapping for Robust Deep Learning [55.733193075728096]
  Modern deep neural networks can easily overfit to biased training data containing corrupted labels or class imbalance. Sample re-weighting methods are popularly used to alleviate this data bias issue. We propose a meta-model capable of adaptively learning an explicit weighting scheme directly from data.
  arXiv  Detail & Related papers  (2022-02-11T13:49:51Z)
• Class-Wise Difficulty-Balanced Loss for Solving Class-Imbalance [6.875312133832079]
  We propose a novel loss function named Class-wise Difficulty-Balanced loss. It dynamically distributes weights to each sample according to the difficulty of the class that the sample belongs to. The results show that CDB loss consistently outperforms the recently proposed loss functions on class-imbalanced datasets.
  arXiv  Detail & Related papers  (2020-10-05T07:19:19Z)
• Long-Tailed Recognition Using Class-Balanced Experts [128.73438243408393]
  We propose an ensemble of class-balanced experts that combines the strength of diverse classifiers. Our ensemble of class-balanced experts reaches results close to state-of-the-art and an extended ensemble establishes a new state-of-the-art on two benchmarks for long-tailed recognition.
  arXiv  Detail & Related papers  (2020-04-07T20:57:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.