Benchmarking noisy label detection methods

• URL: http://arxiv.org/abs/2510.16211v1
• Date: Fri, 17 Oct 2025 20:55:26 GMT
• Title: Benchmarking noisy label detection methods
• Authors: Henrique Pickler, Jorge K. S. Kamassury, Danilo Silva,
• Abstract summary: Label noise is a common problem in real-world datasets, affecting both model training and validation.<n>We perform a comprehensive benchmark of detection methods by decomposing them into three fundamental components.<n>We identify that in-sample information gathering using average probability aggregation combined with the logit margin achieves the best results.
• Score: 0.3154269505086154
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Label noise is a common problem in real-world datasets, affecting both model training and validation. Clean data are essential for achieving strong performance and ensuring reliable evaluation. While various techniques have been proposed to detect noisy labels, there is no clear consensus on optimal approaches. We perform a comprehensive benchmark of detection methods by decomposing them into three fundamental components: label agreement function, aggregation method, and information gathering approach (in-sample vs out-of-sample). This decomposition can be applied to many existing detection methods, and enables systematic comparison across diverse approaches. To fairly compare methods, we propose a unified benchmark task, detecting a fraction of training samples equal to the dataset's noise rate. We also introduce a novel metric: the false negative rate at this fixed operating point. Our evaluation spans vision and tabular datasets under both synthetic and real-world noise conditions. We identify that in-sample information gathering using average probability aggregation combined with the logit margin as the label agreement function achieves the best results across most scenarios. Our findings provide practical guidance for designing new detection methods and selecting techniques for specific applications.

Related papers

• Combating Noisy Labels through Fostering Self- and Neighbor-Consistency [120.4394402099635]
  Label noise is pervasive in various real-world scenarios, posing challenges in supervised deep learning.<n>We propose a noise-robust method named Jo-SNC (textbfJoint sample selection and model regularization based on textbfSelf- and textbfNeighbor-textbfConsistency)<n>We design a self-adaptive, data-driven thresholding scheme to adjust per-class selection thresholds.
  arXiv  Detail & Related papers  (2026-01-19T07:55:29Z)
• Detecting Mislabeled and Corrupted Data via Pointwise Mutual Information [0.9821874476902969]
  This paper proposes a mutual information-based framework for data selection under hybrid noise scenarios.<n>We compute each sample's pointwise contribution to the overall mutual information and find that lower contributions indicate noisy or mislabeled instances.<n>Under label corruption, training on high-MI samples improves classification accuracy by up to 15% compared to random sampling.
  arXiv  Detail & Related papers  (2025-08-11T07:39:20Z)
• FNBench: Benchmarking Robust Federated Learning against Noisy Labels [23.660857480962104]
  This paper presents the first benchmark study on noisy labels in federated learning (FL)<n>We consider three diverse label noise patterns covering synthetic label noise, imperfect human-annotation errors and systematic errors.<n>Our evaluation incorporates eighteen state-of-the-art methods over five image recognition datasets and one text classification dataset.
  arXiv  Detail & Related papers  (2025-05-10T16:14:52Z)
• Extracting Clean and Balanced Subset for Noisy Long-tailed Classification [66.47809135771698]
  We develop a novel pseudo labeling method using class prototypes from the perspective of distribution matching. By setting a manually-specific probability measure, we can reduce the side-effects of noisy and long-tailed data simultaneously. Our method can extract this class-balanced subset with clean labels, which brings effective performance gains for long-tailed classification with label noise.
  arXiv  Detail & Related papers  (2024-04-10T07:34:37Z)
• LaplaceConfidence: a Graph-based Approach for Learning with Noisy Labels [17.66525177980075]
  We introduce LaplaceConfidence, a method to obtain label confidence (i.e., clean probabilities) utilizing the Laplacian energy. LaplaceConfidence is embedded into a holistic method for robust training, where co-training technique generates unbiased label confidence. Our experiments demonstrate that LaplaceConfidence outperforms state-of-the-art methods on benchmark datasets under both synthetic and real-world noise.
  arXiv  Detail & Related papers  (2023-07-31T12:44:30Z)
• Rethinking Noisy Label Learning in Real-world Annotation Scenarios from the Noise-type Perspective [38.24239397999152]
  We propose a novel sample selection-based approach for noisy label learning, called Proto-semi. Proto-semi divides all samples into the confident and unconfident datasets via warm-up. By leveraging the confident dataset, prototype vectors are constructed to capture class characteristics. Empirical evaluations on a real-world annotated dataset substantiate the robustness of Proto-semi in handling the problem of learning from noisy labels.
  arXiv  Detail & Related papers  (2023-07-28T10:57:38Z)
• Neighborhood Collective Estimation for Noisy Label Identification and Correction [92.20697827784426]
  Learning with noisy labels (LNL) aims at designing strategies to improve model performance and generalization by mitigating the effects of model overfitting to noisy labels. Recent advances employ the predicted label distributions of individual samples to perform noise verification and noisy label correction, easily giving rise to confirmation bias. We propose Neighborhood Collective Estimation, in which the predictive reliability of a candidate sample is re-estimated by contrasting it against its feature-space nearest neighbors.
  arXiv  Detail & Related papers  (2022-08-05T14:47:22Z)
• Robust Meta-learning with Sampling Noise and Label Noise via Eigen-Reptile [78.1212767880785]
  meta-learner is prone to overfitting since there are only a few available samples. When handling the data with noisy labels, the meta-learner could be extremely sensitive to label noise. We present Eigen-Reptile (ER) that updates the meta- parameters with the main direction of historical task-specific parameters.
  arXiv  Detail & Related papers  (2022-06-04T08:48:02Z)
• UNICON: Combating Label Noise Through Uniform Selection and Contrastive Learning [89.56465237941013]
  We propose UNICON, a simple yet effective sample selection method which is robust to high label noise. We obtain an 11.4% improvement over the current state-of-the-art on CIFAR100 dataset with a 90% noise rate.
  arXiv  Detail & Related papers  (2022-03-28T07:36:36Z)
• Towards Model-Agnostic Post-Hoc Adjustment for Balancing Ranking Fairness and Algorithm Utility [54.179859639868646]
  Bipartite ranking aims to learn a scoring function that ranks positive individuals higher than negative ones from labeled data. There have been rising concerns on whether the learned scoring function can cause systematic disparity across different protected groups. We propose a model post-processing framework for balancing them in the bipartite ranking scenario.
  arXiv  Detail & Related papers  (2020-06-15T10:08:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.