Disturbing Target Values for Neural Network Regularization
- URL: http://arxiv.org/abs/2110.05003v1
- Date: Mon, 11 Oct 2021 05:14:02 GMT
- Title: Disturbing Target Values for Neural Network Regularization
- Authors: Yongho Kim, Hanna Lukashonak, Paweena Tarepakdee, Klavdia Zavalich,
Mofassir ul Islam Arif
- Abstract summary: Directional DisturbLabel (DDL) is a novel regularization technique that makes use of the class probabilities to infer the confident labels.
DDL uses the model behavior during training to regularize it in a more directed manner.
In this paper, 6 and 8 datasets are used to validate the robustness of our methods in classification and regression tasks respectively.
- Score: 1.5574423250822542
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Diverse regularization techniques have been developed such as L2
regularization, Dropout, DisturbLabel (DL) to prevent overfitting. DL, a
newcomer on the scene, regularizes the loss layer by flipping a small share of
the target labels at random and training the neural network on this distorted
data so as to not learn the training data. It is observed that high confidence
labels during training cause the overfitting problem and DL selects disturb
labels at random regardless of the confidence of labels. To solve this
shortcoming of DL, we propose Directional DisturbLabel (DDL) a novel
regularization technique that makes use of the class probabilities to infer the
confident labels and using these labels to regularize the model. This active
regularization makes use of the model behavior during training to regularize it
in a more directed manner. To address regression problems, we also propose
DisturbValue (DV), and DisturbError (DE). DE uses only predefined confident
labels to disturb target values. DV injects noise into a portion of target
values at random similar to DL. In this paper, 6 and 8 datasets are used to
validate the robustness of our methods in classification and regression tasks
respectively. Finally, we demonstrate that our methods are either comparable to
or outperform DisturbLabel, L2 regularization, and Dropout. Also, we achieve
the best performance in more than half the datasets by combining our methods
with either L2 regularization or Dropout.
Related papers
- Generating Unbiased Pseudo-labels via a Theoretically Guaranteed
Chebyshev Constraint to Unify Semi-supervised Classification and Regression [57.17120203327993]
threshold-to-pseudo label process (T2L) in classification uses confidence to determine the quality of label.
In nature, regression also requires unbiased methods to generate high-quality labels.
We propose a theoretically guaranteed constraint for generating unbiased labels based on Chebyshev's inequality.
arXiv Detail & Related papers (2023-11-03T08:39:35Z) - Partial-Label Regression [54.74984751371617]
Partial-label learning is a weakly supervised learning setting that allows each training example to be annotated with a set of candidate labels.
Previous studies on partial-label learning only focused on the classification setting where candidate labels are all discrete.
In this paper, we provide the first attempt to investigate partial-label regression, where each training example is annotated with a set of real-valued candidate labels.
arXiv Detail & Related papers (2023-06-15T09:02:24Z) - All Points Matter: Entropy-Regularized Distribution Alignment for
Weakly-supervised 3D Segmentation [67.30502812804271]
Pseudo-labels are widely employed in weakly supervised 3D segmentation tasks where only sparse ground-truth labels are available for learning.
We propose a novel learning strategy to regularize the generated pseudo-labels and effectively narrow the gaps between pseudo-labels and model predictions.
arXiv Detail & Related papers (2023-05-25T08:19:31Z) - Pseudo-Label Noise Suppression Techniques for Semi-Supervised Semantic
Segmentation [21.163070161951868]
Semi-consuming learning (SSL) can reduce the need for large labelled datasets by incorporating unsupervised data into the training.
Current SSL approaches use an initially supervised trained model to generate predictions for unlabelled images, called pseudo-labels.
We use three mechanisms to control pseudo-label noise and errors.
arXiv Detail & Related papers (2022-10-19T09:46:27Z) - Two Wrongs Don't Make a Right: Combating Confirmation Bias in Learning
with Label Noise [6.303101074386922]
Robust Label Refurbishment (Robust LR) is a new hybrid method that integrates pseudo-labeling and confidence estimation techniques to refurbish noisy labels.
We show that our method successfully alleviates the damage of both label noise and confirmation bias.
For example, Robust LR achieves up to 4.5% absolute top-1 accuracy improvement over the previous best on the real-world noisy dataset WebVision.
arXiv Detail & Related papers (2021-12-06T12:10:17Z) - Self-Tuning for Data-Efficient Deep Learning [75.34320911480008]
Self-Tuning is a novel approach to enable data-efficient deep learning.
It unifies the exploration of labeled and unlabeled data and the transfer of a pre-trained model.
It outperforms its SSL and TL counterparts on five tasks by sharp margins.
arXiv Detail & Related papers (2021-02-25T14:56:19Z) - Two-phase Pseudo Label Densification for Self-training based Domain
Adaptation [93.03265290594278]
We propose a novel Two-phase Pseudo Label Densification framework, referred to as TPLD.
In the first phase, we use sliding window voting to propagate the confident predictions, utilizing intrinsic spatial-correlations in the images.
In the second phase, we perform a confidence-based easy-hard classification.
To ease the training process and avoid noisy predictions, we introduce the bootstrapping mechanism to the original self-training loss.
arXiv Detail & Related papers (2020-12-09T02:35:25Z) - Learning to Purify Noisy Labels via Meta Soft Label Corrector [49.92310583232323]
Recent deep neural networks (DNNs) can easily overfit to biased training data with noisy labels.
Label correction strategy is commonly used to alleviate this issue.
We propose a meta-learning model which could estimate soft labels through meta-gradient descent step.
arXiv Detail & Related papers (2020-08-03T03:25:17Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.