Regularization via Adaptive Pairwise Label Smoothing

• URL: http://arxiv.org/abs/2012.01559v1
• Date: Wed, 2 Dec 2020 22:08:10 GMT
• Title: Regularization via Adaptive Pairwise Label Smoothing
• Authors: Hongyu Guo
• Abstract summary: This paper introduces a novel label smoothing technique called Pairwise Label Smoothing (PLS) Unlike current LS methods, which typically require to find a global smoothing distribution mass through cross-validation search, PLS automatically learns the distribution mass for each input pair during training. We empirically show that PLS significantly outperforms LS and the baseline models, achieving up to 30% of relative classification error reduction.
• Score: 19.252319300590653
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Label Smoothing (LS) is an effective regularizer to improve the generalization of state-of-the-art deep models. For each training sample the LS strategy smooths the one-hot encoded training signal by distributing its distribution mass over the non ground-truth classes, aiming to penalize the networks from generating overconfident output distributions. This paper introduces a novel label smoothing technique called Pairwise Label Smoothing (PLS). The PLS takes a pair of samples as input. Smoothing with a pair of ground-truth labels enables the PLS to preserve the relative distance between the two truth labels while further soften that between the truth labels and the other targets, resulting in models producing much less confident predictions than the LS strategy. Also, unlike current LS methods, which typically require to find a global smoothing distribution mass through cross-validation search, PLS automatically learns the distribution mass for each input pair during training. We empirically show that PLS significantly outperforms LS and the baseline models, achieving up to 30% of relative classification error reduction. We also visually show that when achieving such accuracy gains the PLS tends to produce very low winning softmax scores.

Related papers

• A Channel-ensemble Approach: Unbiased and Low-variance Pseudo-labels is Critical for Semi-supervised Classification [61.473485511491795]
  Semi-supervised learning (SSL) is a practical challenge in computer vision. Pseudo-label (PL) methods, e.g., FixMatch and FreeMatch, obtain the State Of The Art (SOTA) performances in SSL. We propose a lightweight channel-based ensemble method to consolidate multiple inferior PLs into the theoretically guaranteed unbiased and low-variance one.
  arXiv  Detail & Related papers  (2024-03-27T09:49:37Z)
• RankMatch: A Novel Approach to Semi-Supervised Label Distribution Learning Leveraging Inter-label Correlations [52.549807652527306]
  This paper introduces RankMatch, an innovative approach for Semi-Supervised Label Distribution Learning (SSLDL) RankMatch effectively utilizes a small number of labeled examples in conjunction with a larger quantity of unlabeled data. We establish a theoretical generalization bound for RankMatch, and through extensive experiments, demonstrate its superiority in performance against existing SSLDL methods.
  arXiv  Detail & Related papers  (2023-12-11T12:47:29Z)
• Flexible Distribution Alignment: Towards Long-tailed Semi-supervised Learning with Proper Calibration [18.376601653387315]
  Longtailed semi-supervised learning (LTSSL) represents a practical scenario for semi-supervised applications. This problem is often aggravated by discrepancies between labeled and unlabeled class distributions. We introduce Flexible Distribution Alignment (FlexDA), a novel adaptive logit-adjusted loss framework.
  arXiv  Detail & Related papers  (2023-06-07T17:50:59Z)
• Learning in Imperfect Environment: Multi-Label Classification with Long-Tailed Distribution and Partial Labels [53.68653940062605]
  We introduce a novel task, Partial labeling and Long-Tailed Multi-Label Classification (PLT-MLC) We find that most LT-MLC and PL-MLC approaches fail to solve the degradation-MLC. We propose an end-to-end learning framework: textbfCOrrection $rightarrow$ textbfModificattextbfIon $rightarrow$ balantextbfCe.
  arXiv  Detail & Related papers  (2023-04-20T20:05:08Z)
• CLS: Cross Labeling Supervision for Semi-Supervised Learning [9.929229055862491]
  Cross Labeling Supervision ( CLS) is a framework that generalizes the typical pseudo-labeling process. CLS allows the creation of both pseudo and complementary labels to support both positive and negative learning.
  arXiv  Detail & Related papers  (2022-02-17T08:09:40Z)
• Midpoint Regularization: from High Uncertainty Training to Conservative Classification [19.252319300590653]
  Label Smoothing (LS) improves model generalization through penalizing models from generating overconfident output distributions. We extend this technique by considering example pairs, coined PLS. PLS first creates midpoint samples by averaging random sample pairs and then learns a smoothing distribution during training for each of these midpoint samples, resulting in midpoints with high uncertainty labels for training.
  arXiv  Detail & Related papers  (2021-06-26T00:31:46Z)
• Understanding (Generalized) Label Smoothing when Learning with Noisy Labels [57.37057235894054]
  Label smoothing (LS) is an arising learning paradigm that uses the positively weighted average of both the hard training labels and uniformly distributed soft labels. We provide understandings for the properties of generalized label smoothing (GLS) when learning with noisy labels.
  arXiv  Detail & Related papers  (2021-06-08T07:32:29Z)
• PLM: Partial Label Masking for Imbalanced Multi-label Classification [59.68444804243782]
  Neural networks trained on real-world datasets with long-tailed label distributions are biased towards frequent classes and perform poorly on infrequent classes. We propose a method, Partial Label Masking (PLM), which utilizes this ratio during training. Our method achieves strong performance when compared to existing methods on both multi-label (MultiMNIST and MSCOCO) and single-label (imbalanced CIFAR-10 and CIFAR-100) image classification datasets.
  arXiv  Detail & Related papers  (2021-05-22T18:07:56Z)
• Weakly Supervised Label Smoothing [15.05158252504978]
  We study Label Smoothing (LS), a widely used regularization technique, in the context of neural learning to rank (L2R) models. Inspired by our investigation of LS in the context of neural L2R models, we propose a novel technique called Weakly Supervised Label Smoothing (WSLS)
  arXiv  Detail & Related papers  (2020-12-15T19:36:52Z)
• Distribution Aligning Refinery of Pseudo-label for Imbalanced Semi-supervised Learning [126.31716228319902]
  We develop Distribution Aligning Refinery of Pseudo-label (DARP) algorithm. We show that DARP is provably and efficiently compatible with state-of-the-art SSL schemes.
  arXiv  Detail & Related papers  (2020-07-17T09:16:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.