Learning From Biased Soft Labels

• URL: http://arxiv.org/abs/2302.08155v1
• Date: Thu, 16 Feb 2023 08:57:48 GMT
• Title: Learning From Biased Soft Labels
• Authors: Hua Yuan, Ning Xu, Yu Shi, Xin Geng and Yong Rui
• Abstract summary: A study has demonstrated that knowledge distillation and label smoothing can be unified as learning from soft labels. This paper studies whether biased soft labels are still effective.
• Score: 48.84637168570285
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Knowledge distillation has been widely adopted in a variety of tasks and has achieved remarkable successes. Since its inception, many researchers have been intrigued by the dark knowledge hidden in the outputs of the teacher model. Recently, a study has demonstrated that knowledge distillation and label smoothing can be unified as learning from soft labels. Consequently, how to measure the effectiveness of the soft labels becomes an important question. Most existing theories have stringent constraints on the teacher model or data distribution, and many assumptions imply that the soft labels are close to the ground-truth labels. This paper studies whether biased soft labels are still effective. We present two more comprehensive indicators to measure the effectiveness of such soft labels. Based on the two indicators, we give sufficient conditions to ensure biased soft label based learners are classifier-consistent and ERM learnable. The theory is applied to three weakly-supervised frameworks. Experimental results validate that biased soft labels can also teach good students, which corroborates the soundness of the theory.

Related papers

• Learning with Confidence: Training Better Classifiers from Soft Labels [0.0]
  In supervised machine learning, models are typically trained using data with hard labels, i.e., definite assignments of class membership. We investigate whether incorporating label uncertainty, represented as discrete probability distributions over the class labels, improves the predictive performance of classification models.
  arXiv  Detail & Related papers  (2024-09-24T13:12:29Z)
• Imprecise Label Learning: A Unified Framework for Learning with Various Imprecise Label Configurations [91.67511167969934]
  imprecise label learning (ILL) is a framework for the unification of learning with various imprecise label configurations. We demonstrate that ILL can seamlessly adapt to partial label learning, semi-supervised learning, noisy label learning, and, more importantly, a mixture of these settings.
  arXiv  Detail & Related papers  (2023-05-22T04:50:28Z)
• Beyond Hard Labels: Investigating data label distributions [0.9668407688201357]
  We compare the disparity of learning with hard and soft labels for a synthetic and a real-world dataset. The application of soft labels leads to improved performance and yields a more regular structure of the internal feature space.
  arXiv  Detail & Related papers  (2022-07-13T14:25:30Z)
• One Positive Label is Sufficient: Single-Positive Multi-Label Learning with Label Enhancement [71.9401831465908]
  We investigate single-positive multi-label learning (SPMLL) where each example is annotated with only one relevant label. A novel method named proposed, i.e., Single-positive MultI-label learning with Label Enhancement, is proposed. Experiments on benchmark datasets validate the effectiveness of the proposed method.
  arXiv  Detail & Related papers  (2022-06-01T14:26:30Z)
• A Theoretical Analysis of Learning with Noisily Labeled Data [62.946840431501855]
  We first show that in the first epoch training, the examples with clean labels will be learned first. We then show that after the learning from clean data stage, continuously training model can achieve further improvement in testing error.
  arXiv  Detail & Related papers  (2021-04-08T23:40:02Z)
• Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study [59.95267695402516]
  This work aims to empirically clarify that label smoothing is incompatible with knowledge distillation. We provide a novel connection on how label smoothing affects distributions of semantically similar and dissimilar classes. We study its one-sidedness and imperfection of the incompatibility view through massive analyses, visualizations and comprehensive experiments.
  arXiv  Detail & Related papers  (2021-04-01T17:59:12Z)
• Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective [63.87421152879726]
  We investigate the bias-variance tradeoff brought by distillation with soft labels. We propose the novel weighted soft labels to help the network adaptively handle the sample-wise bias-variance tradeoff.
  arXiv  Detail & Related papers  (2021-02-01T05:53:04Z)
• An Investigation of how Label Smoothing Affects Generalization [22.663974656813824]
  We show how label smoothing provides in controlling the generalization loss. Our theory also predicts the existence of an optimal label smoothing point. Our findings will help both theoreticians and practitioners understand label smoothing.
  arXiv  Detail & Related papers  (2020-10-23T20:26:25Z)
• Rethinking the Value of Labels for Improving Class-Imbalanced Learning [20.953282288425118]
  Class-imbalanced learning can significantly benefit in both semi-supervised and self-supervised manners. We argue that imbalanced labels are not useful always. Our findings highlight the need to rethink the usage of imbalanced labels in realistic long-tailed tasks.
  arXiv  Detail & Related papers  (2020-06-13T01:35:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.