Two-Phase Learning for Overcoming Noisy Labels
- URL: http://arxiv.org/abs/2012.04337v1
- Date: Tue, 8 Dec 2020 10:25:29 GMT
- Title: Two-Phase Learning for Overcoming Noisy Labels
- Authors: Hwanjun Song, Minseok Kim, Dongmin Park, Jae-Gil Lee
- Abstract summary: We propose a novel two-phase learning method, which automatically transitions its learning phase at the point when the network begins to memorize false-labeled samples.
MorPH significantly outperforms five state-of-the art methods in terms of test error and training time.
- Score: 16.390094129357774
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: To counter the challenge associated with noise labels, the learning strategy
of deep neural networks must be differentiated over the learning period during
the training process. Therefore, we propose a novel two-phase learning method,
MORPH, which automatically transitions its learning phase at the point when the
network begins to rapidly memorize false-labeled samples. In the first phase,
MORPH starts to update the network for all the training samples before the
transition point. Without any supervision, the learning phase is converted to
the next phase on the basis of the estimated best transition point.
Subsequently, MORPH resumes the training of the network only for a maximal safe
set, which maintains the collection of almost certainly true-labeled samples at
each epoch. Owing to its two-phase learning, MORPH realizes noise-free training
for any type of label noise for practical use. Moreover, extensive experiments
using six datasets verify that MORPH significantly outperforms five
state-of-the art methods in terms of test error and training time.
Related papers
- On Emergence of Clean-Priority Learning in Early Stopped Neural Networks [18.725557157004214]
When random label noise is added to a training dataset, the prediction error of a neural network on a label-noise-free test dataset deteriorates.
This behaviour is believed to be a result of neural networks learning the pattern of clean data first and fitting the noise later in the training.
We show both theoretically and experimentally, as the clean-priority learning goes on, the dominance of the gradients of clean samples over those of noisy samples diminishes.
arXiv Detail & Related papers (2023-06-05T01:45:22Z) - PADDLES: Phase-Amplitude Spectrum Disentangled Early Stopping for
Learning with Noisy Labels [49.43183579415899]
Convolutional Neural Networks (CNNs) have demonstrated superiority in learning patterns, but are sensitive to label noises and may overfit noisy labels during training.
The early stopping strategy averts updating CNNs during the early training phase and is widely employed in the presence of noisy labels.
Our proposed Phase-AmplituDe DisentangLed Early Stopping (PADDLES) method is shown to be effective on both synthetic and real-world label-noise datasets.
arXiv Detail & Related papers (2022-12-07T05:03:13Z) - On the Soft-Subnetwork for Few-shot Class Incremental Learning [67.0373924836107]
We propose a few-shot class incremental learning (FSCIL) method referred to as emphSoft-SubNetworks (SoftNet).
Our objective is to learn a sequence of sessions incrementally, where each session only includes a few training instances per class while preserving the knowledge of the previously learned ones.
We provide comprehensive empirical validations demonstrating that our SoftNet effectively tackles the few-shot incremental learning problem by surpassing the performance of state-of-the-art baselines over benchmark datasets.
arXiv Detail & Related papers (2022-09-15T04:54:02Z) - Intersection of Parallels as an Early Stopping Criterion [64.8387564654474]
We propose a method to spot an early stopping point in the training iterations without the need for a validation set.
For a wide range of learning rates, our method, called Cosine-Distance Criterion (CDC), leads to better generalization on average than all the methods that we compare against.
arXiv Detail & Related papers (2022-08-19T19:42:41Z) - A Theoretical Analysis of Learning with Noisily Labeled Data [62.946840431501855]
We first show that in the first epoch training, the examples with clean labels will be learned first.
We then show that after the learning from clean data stage, continuously training model can achieve further improvement in testing error.
arXiv Detail & Related papers (2021-04-08T23:40:02Z) - Phases of learning dynamics in artificial neural networks: with or
without mislabeled data [3.3576886095389296]
We study dynamics of gradient descent that drives learning in neural networks.
Without mislabeled data, we find that the SGD learning dynamics transitions from a fast learning phase to a slow exploration phase.
We find that individual sample losses for the two datasets are most separated during phase II.
arXiv Detail & Related papers (2021-01-16T19:44:27Z) - Noisy Concurrent Training for Efficient Learning under Label Noise [13.041607703862724]
Deep neural networks (DNNs) fail to learn effectively under label noise and have been shown to memorize random labels which affect their performance.
We consider learning in isolation, using one-hot encoded labels as the sole source of supervision, and a lack of regularization to discourage memorization as the major shortcomings of the standard training procedure.
We propose Noisy Concurrent Training (NCT) which leverages collaborative learning to use the consensus between two models as an additional source of supervision.
arXiv Detail & Related papers (2020-09-17T14:22:17Z) - Early-Learning Regularization Prevents Memorization of Noisy Labels [29.04549895470588]
We propose a novel framework to perform classification via deep learning in the presence of noisy annotations.
Deep neural networks have been observed to first fit the training data with clean labels during an "early learning" phase.
We design a regularization term that steers the model towards these targets, implicitly preventing memorization of the false labels.
arXiv Detail & Related papers (2020-06-30T23:46:33Z) - MetricUNet: Synergistic Image- and Voxel-Level Learning for Precise CT
Prostate Segmentation via Online Sampling [66.01558025094333]
We propose a two-stage framework, with the first stage to quickly localize the prostate region and the second stage to precisely segment the prostate.
We introduce a novel online metric learning module through voxel-wise sampling in the multi-task network.
Our method can effectively learn more representative voxel-level features compared with the conventional learning methods with cross-entropy or Dice loss.
arXiv Detail & Related papers (2020-05-15T10:37:02Z) - DivideMix: Learning with Noisy Labels as Semi-supervised Learning [111.03364864022261]
We propose DivideMix, a framework for learning with noisy labels.
Experiments on multiple benchmark datasets demonstrate substantial improvements over state-of-the-art methods.
arXiv Detail & Related papers (2020-02-18T06:20:06Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.