Why Can Accurate Models Be Learned from Inaccurate Annotations?

• URL: http://arxiv.org/abs/2505.16159v1
• Date: Thu, 22 May 2025 03:00:15 GMT
• Title: Why Can Accurate Models Be Learned from Inaccurate Annotations?
• Authors: Chongjie Si, Yidan Cui, Fuchao Yang, Xiaokang Yang, Wei Shen,
• Abstract summary: Despite the presence of erroneous labels, models trained on noisy data often retain the ability to make accurate predictions.<n>This intriguing phenomenon raises a fundamental yet largely unexplored question: why models can still extract correct label information from inaccurate annotations remains unexplored.<n>We propose LIP, a lightweight plug-in designed to help classifiers retain principal subspace information while mitigating noise induced by label inaccuracy.
• Score: 48.528799044535155
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learning from inaccurate annotations has gained significant attention due to the high cost of precise labeling. However, despite the presence of erroneous labels, models trained on noisy data often retain the ability to make accurate predictions. This intriguing phenomenon raises a fundamental yet largely unexplored question: why models can still extract correct label information from inaccurate annotations remains unexplored. In this paper, we conduct a comprehensive investigation into this issue. By analyzing weight matrices from both empirical and theoretical perspectives, we find that label inaccuracy primarily accumulates noise in lower singular components and subtly perturbs the principal subspace. Within a certain range, the principal subspaces of weights trained on inaccurate labels remain largely aligned with those learned from clean labels, preserving essential task-relevant information. We formally prove that the angles of principal subspaces exhibit minimal deviation under moderate label inaccuracy, explaining why models can still generalize effectively. Building on these insights, we propose LIP, a lightweight plug-in designed to help classifiers retain principal subspace information while mitigating noise induced by label inaccuracy. Extensive experiments on tasks with various inaccuracy conditions demonstrate that LIP consistently enhances the performance of existing algorithms. We hope our findings can offer valuable theoretical and practical insights to understand of model robustness under inaccurate supervision.

Related papers

• Practical estimation of the optimal classification error with soft labels and calibration [52.1410307583181]
  We extend a previous work that utilizes soft labels for estimating the Bayes error, the optimal error rate.<n>We tackle a more challenging problem setting: estimation with corrupted soft labels.<n>Our method is instance-free, i.e., we do not assume access to any input instances.
  arXiv  Detail & Related papers  (2025-05-27T06:04:57Z)
• Don't Blame the Data, Blame the Model: Understanding Noise and Bias When Learning from Subjective Annotations [9.221081428960318]
  We show that models that are only provided aggregated labels show low confidence on high-disagreement data instances. Our experiments show an improvement of confidence for the high-disagreement instances.
  arXiv  Detail & Related papers  (2024-03-06T22:30:04Z)
• Using Early Readouts to Mediate Featural Bias in Distillation [30.5299408494168]
  Deep networks tend to learn spurious feature-label correlations in real-world supervised learning tasks. We propose a novel early readout mechanism whereby we attempt to predict the label using representations from earlier network layers.
  arXiv  Detail & Related papers  (2023-10-28T04:58:15Z)
• Label-Retrieval-Augmented Diffusion Models for Learning from Noisy Labels [61.97359362447732]
  Learning from noisy labels is an important and long-standing problem in machine learning for real applications. In this paper, we reformulate the label-noise problem from a generative-model perspective. Our model achieves new state-of-the-art (SOTA) results on all the standard real-world benchmark datasets.
  arXiv  Detail & Related papers  (2023-05-31T03:01:36Z)
• Weak Proxies are Sufficient and Preferable for Fairness with Missing Sensitive Attributes [25.730297492625507]
  We develop an algorithm that is able to measure fairness (provably) accurately with only three properly identified proxies. Our results imply a set of practical guidelines for practitioners on how to use proxies properly.
  arXiv  Detail & Related papers  (2022-10-06T19:25:29Z)
• Prototype-Anchored Learning for Learning with Imperfect Annotations [83.7763875464011]
  It is challenging to learn unbiased classification models from imperfectly annotated datasets. We propose a prototype-anchored learning (PAL) method, which can be easily incorporated into various learning-based classification schemes. We verify the effectiveness of PAL on class-imbalanced learning and noise-tolerant learning by extensive experiments on synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2022-06-23T10:25:37Z)
• Learning from Noisy Labels for Entity-Centric Information Extraction [17.50856935207308]
  We propose a simple co-regularization framework for entity-centric information extraction. These models are jointly optimized with task-specific loss, and are regularized to generate similar predictions. In the end, we can take any of the trained models for inference.
  arXiv  Detail & Related papers  (2021-04-17T22:49:12Z)
• A Theoretical Analysis of Learning with Noisily Labeled Data [62.946840431501855]
  We first show that in the first epoch training, the examples with clean labels will be learned first. We then show that after the learning from clean data stage, continuously training model can achieve further improvement in testing error.
  arXiv  Detail & Related papers  (2021-04-08T23:40:02Z)
• Robustness of Accuracy Metric and its Inspirations in Learning with Noisy Labels [51.66448070984615]
  We show that maximizing training accuracy on sufficiently many noisy samples yields an approximately optimal classifier. For validation, we prove that a noisy validation set is reliable, addressing the critical demand of model selection. We show characterizations of models trained with noisy labels, motivated by our theoretical results, and verify the utility of a noisy validation set.
  arXiv  Detail & Related papers  (2020-12-08T03:37:47Z)
• An Investigation of how Label Smoothing Affects Generalization [22.663974656813824]
  We show how label smoothing provides in controlling the generalization loss. Our theory also predicts the existence of an optimal label smoothing point. Our findings will help both theoreticians and practitioners understand label smoothing.
  arXiv  Detail & Related papers  (2020-10-23T20:26:25Z)
• Rethinking the Value of Labels for Improving Class-Imbalanced Learning [20.953282288425118]
  Class-imbalanced learning can significantly benefit in both semi-supervised and self-supervised manners. We argue that imbalanced labels are not useful always. Our findings highlight the need to rethink the usage of imbalanced labels in realistic long-tailed tasks.
  arXiv  Detail & Related papers  (2020-06-13T01:35:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.