Regularization via Structural Label Smoothing
- URL: http://arxiv.org/abs/2001.01900v2
- Date: Sat, 4 Jul 2020 23:22:56 GMT
- Title: Regularization via Structural Label Smoothing
- Authors: Weizhi Li, Gautam Dasarathy and Visar Berisha
- Abstract summary: Regularization is an effective way to promote the generalization performance of machine learning models.
In this paper, we focus on label smoothing, a form of output distribution regularization that prevents overfitting of a neural network.
We show that such label smoothing imposes a quantifiable bias in the Bayes error rate of the training data.
- Score: 22.74769739125912
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Regularization is an effective way to promote the generalization performance
of machine learning models. In this paper, we focus on label smoothing, a form
of output distribution regularization that prevents overfitting of a neural
network by softening the ground-truth labels in the training data in an attempt
to penalize overconfident outputs. Existing approaches typically use
cross-validation to impose this smoothing, which is uniform across all training
data. In this paper, we show that such label smoothing imposes a quantifiable
bias in the Bayes error rate of the training data, with regions of the feature
space with high overlap and low marginal likelihood having a lower bias and
regions of low overlap and high marginal likelihood having a higher bias. These
theoretical results motivate a simple objective function for data-dependent
smoothing to mitigate the potential negative consequences of the operation
while maintaining its desirable properties as a regularizer. We call this
approach Structural Label Smoothing (SLS). We implement SLS and empirically
validate on synthetic, Higgs, SVHN, CIFAR-10, and CIFAR-100 datasets. The
results confirm our theoretical insights and demonstrate the effectiveness of
the proposed method in comparison to traditional label smoothing.
Related papers
- A Channel-ensemble Approach: Unbiased and Low-variance Pseudo-labels is Critical for Semi-supervised Classification [61.473485511491795]
Semi-supervised learning (SSL) is a practical challenge in computer vision.
Pseudo-label (PL) methods, e.g., FixMatch and FreeMatch, obtain the State Of The Art (SOTA) performances in SSL.
We propose a lightweight channel-based ensemble method to consolidate multiple inferior PLs into the theoretically guaranteed unbiased and low-variance one.
arXiv Detail & Related papers (2024-03-27T09:49:37Z) - All Points Matter: Entropy-Regularized Distribution Alignment for
Weakly-supervised 3D Segmentation [67.30502812804271]
Pseudo-labels are widely employed in weakly supervised 3D segmentation tasks where only sparse ground-truth labels are available for learning.
We propose a novel learning strategy to regularize the generated pseudo-labels and effectively narrow the gaps between pseudo-labels and model predictions.
arXiv Detail & Related papers (2023-05-25T08:19:31Z) - Boosting Differentiable Causal Discovery via Adaptive Sample Reweighting [62.23057729112182]
Differentiable score-based causal discovery methods learn a directed acyclic graph from observational data.
We propose a model-agnostic framework to boost causal discovery performance by dynamically learning the adaptive weights for the Reweighted Score function, ReScore.
arXiv Detail & Related papers (2023-03-06T14:49:59Z) - Adaptive Label Smoothing with Self-Knowledge in Natural Language
Generation [16.878277421402945]
We propose a regularization scheme that brings dynamic nature into the smoothing parameter.
A model in training self-regulates the extent of smoothing on the fly during forward propagation.
arXiv Detail & Related papers (2022-10-22T11:52:38Z) - FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning [46.95063831057502]
We propose emphFreeMatch to define and adjust the confidence threshold in a self-adaptive manner according to the model's learning status.
FreeMatch achieves textbf5.78%, textbf13.59%, and textbf1.28% error rate reduction over the latest state-of-the-art method FlexMatch on CIFAR-10 with 1 label per class.
arXiv Detail & Related papers (2022-05-15T10:07:52Z) - Robust Neural Network Classification via Double Regularization [2.41710192205034]
We propose a novel double regularization of the neural network training loss that combines a penalty on the complexity of the classification model and an optimal reweighting of training observations.
We demonstrate DRFit, for neural net classification of (i) MNIST and (ii) CIFAR-10, in both cases with simulated mislabeling.
arXiv Detail & Related papers (2021-12-15T13:19:20Z) - RATT: Leveraging Unlabeled Data to Guarantee Generalization [96.08979093738024]
We introduce a method that leverages unlabeled data to produce generalization bounds.
We prove that our bound is valid for 0-1 empirical risk minimization.
This work provides practitioners with an option for certifying the generalization of deep nets even when unseen labeled data is unavailable.
arXiv Detail & Related papers (2021-05-01T17:05:29Z) - In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label
Selection Framework for Semi-Supervised Learning [53.1047775185362]
Pseudo-labeling (PL) is a general SSL approach that does not have this constraint but performs relatively poorly in its original formulation.
We argue that PL underperforms due to the erroneous high confidence predictions from poorly calibrated models.
We propose an uncertainty-aware pseudo-label selection (UPS) framework which improves pseudo labeling accuracy by drastically reducing the amount of noise encountered in the training process.
arXiv Detail & Related papers (2021-01-15T23:29:57Z) - Delving Deep into Label Smoothing [112.24527926373084]
Label smoothing is an effective regularization tool for deep neural networks (DNNs)
We present an Online Label Smoothing (OLS) strategy, which generates soft labels based on the statistics of the model prediction for the target category.
arXiv Detail & Related papers (2020-11-25T08:03:11Z) - Deep Active Learning for Biased Datasets via Fisher Kernel
Self-Supervision [5.352699766206807]
Active learning (AL) aims to minimize labeling efforts for data-demanding deep neural networks (DNNs)
We propose a low-complexity method for feature density matching using self-supervised Fisher kernel (FK)
Our method outperforms state-of-the-art methods on MNIST, SVHN, and ImageNet classification while requiring only 1/10th of processing.
arXiv Detail & Related papers (2020-03-01T03:56:32Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.