Instance-based Label Smoothing For Better Calibrated Classification Networks

• URL: http://arxiv.org/abs/2110.05355v1
• Date: Mon, 11 Oct 2021 15:33:23 GMT
• Title: Instance-based Label Smoothing For Better Calibrated Classification Networks
• Authors: Mohamed Maher, Meelis Kull
• Abstract summary: Label smoothing is widely used in deep neural networks for multi-class classification. We take inspiration from both label smoothing and self-distillation. We propose two novel instance-based label smoothing approaches.
• Score: 3.388509725285237
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Label smoothing is widely used in deep neural networks for multi-class classification. While it enhances model generalization and reduces overconfidence by aiming to lower the probability for the predicted class, it distorts the predicted probabilities of other classes resulting in poor class-wise calibration. Another method for enhancing model generalization is self-distillation where the predictions of a teacher network trained with one-hot labels are used as the target for training a student network. We take inspiration from both label smoothing and self-distillation and propose two novel instance-based label smoothing approaches, where a teacher network trained with hard one-hot labels is used to determine the amount of per class smoothness applied to each instance. The assigned smoothing factor is non-uniformly distributed along with the classes according to their similarity with the actual class. Our methods show better generalization and calibration over standard label smoothing on various deep neural architectures and image classification datasets.

Related papers

• Posterior Label Smoothing for Node Classification [2.737276507021477]
  We propose a simple yet effective label smoothing for the transductive node classification task. We design the soft label to encapsulate the local context of the target node through the neighborhood label distribution. In the following analysis, we find that incorporating global label statistics in posterior computation is the key to the success of label smoothing.
  arXiv  Detail & Related papers  (2024-06-01T11:59:49Z)
• Generating Unbiased Pseudo-labels via a Theoretically Guaranteed Chebyshev Constraint to Unify Semi-supervised Classification and Regression [57.17120203327993]
  threshold-to-pseudo label process (T2L) in classification uses confidence to determine the quality of label. In nature, regression also requires unbiased methods to generate high-quality labels. We propose a theoretically guaranteed constraint for generating unbiased labels based on Chebyshev's inequality.
  arXiv  Detail & Related papers  (2023-11-03T08:39:35Z)
• Active Inference-Based Optimization of Discriminative Neural Network Classifiers [3.1219977244201056]
  We propose a novel algorithm to find candidate classification labels of the training samples from their prior probabilities. The proposed objective function could incorporate the candidate labels, the original reference labels, and the priors of the training samples while still being distribution-based.
  arXiv  Detail & Related papers  (2023-06-04T19:30:28Z)
• Self-Evolution Learning for Mixup: Enhance Data Augmentation on Few-Shot Text Classification Tasks [75.42002070547267]
  We propose a self evolution learning (SE) based mixup approach for data augmentation in text classification. We introduce a novel instance specific label smoothing approach, which linearly interpolates the model's output and one hot labels of the original samples to generate new soft for label mixing up.
  arXiv  Detail & Related papers  (2023-05-22T23:43:23Z)
• Transductive CLIP with Class-Conditional Contrastive Learning [68.51078382124331]
  We propose Transductive CLIP, a novel framework for learning a classification network with noisy labels from scratch. A class-conditional contrastive learning mechanism is proposed to mitigate the reliance on pseudo labels. ensemble labels is adopted as a pseudo label updating strategy to stabilize the training of deep neural networks with noisy labels.
  arXiv  Detail & Related papers  (2022-06-13T14:04:57Z)
• Prototypical Classifier for Robust Class-Imbalanced Learning [64.96088324684683]
  We propose textitPrototypical, which does not require fitting additional parameters given the embedding network. Prototypical produces balanced and comparable predictions for all classes even though the training set is class-imbalanced. We test our method on CIFAR-10LT, CIFAR-100LT and Webvision datasets, observing that Prototypical obtains substaintial improvements compared with state of the arts.
  arXiv  Detail & Related papers  (2021-10-22T01:55:01Z)
• PLM: Partial Label Masking for Imbalanced Multi-label Classification [59.68444804243782]
  Neural networks trained on real-world datasets with long-tailed label distributions are biased towards frequent classes and perform poorly on infrequent classes. We propose a method, Partial Label Masking (PLM), which utilizes this ratio during training. Our method achieves strong performance when compared to existing methods on both multi-label (MultiMNIST and MSCOCO) and single-label (imbalanced CIFAR-10 and CIFAR-100) image classification datasets.
  arXiv  Detail & Related papers  (2021-05-22T18:07:56Z)
• Self-Supervised Classification Network [3.8073142980733]
  Self-supervised end-to-end classification neural network learns labels and representations simultaneously. First unsupervised end-to-end classification network to perform well on the large-scale ImageNet dataset.
  arXiv  Detail & Related papers  (2021-03-19T19:29:42Z)
• Locally Adaptive Label Smoothing for Predictive Churn [40.17985689233356]
  Training modern neural networks can lead to high emphprediction churn. We present several baselines for reducing churn and show that training on soft labels obtained by adaptively smoothing each example's label often outperforms the baselines on churn.
  arXiv  Detail & Related papers  (2021-02-09T21:38:37Z)
• Class-Similarity Based Label Smoothing for Confidence Calibration [2.055949720959582]
  We propose a novel form of label smoothing to improve confidence calibration. Since different classes are of different intrinsic similarities, more similar classes should result in closer probability values in the final output. This motivates the development of a new smooth label where the label values are based on similarities with the reference class.
  arXiv  Detail & Related papers  (2020-06-24T20:26:22Z)
• Certified Robustness to Label-Flipping Attacks via Randomized Smoothing [105.91827623768724]
  Machine learning algorithms are susceptible to data poisoning attacks. We present a unifying view of randomized smoothing over arbitrary functions. We propose a new strategy for building classifiers that are pointwise-certifiably robust to general data poisoning attacks.
  arXiv  Detail & Related papers  (2020-02-07T21:28:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.