Blind Knowledge Distillation for Robust Image Classification
- URL: http://arxiv.org/abs/2211.11355v1
- Date: Mon, 21 Nov 2022 11:17:07 GMT
- Title: Blind Knowledge Distillation for Robust Image Classification
- Authors: Timo Kaiser, Lukas Ehmann, Christoph Reinders and Bodo Rosenhahn
- Abstract summary: Blind Knowledge Distillation is a teacher-student approach for learning with noisy labels.
We use Otsus algorithm to estimate the tipping point from generalizing to overfitting.
We show in our experiments that Blind Knowledge Distillation detects overfitting effectively during training.
- Score: 19.668440671541546
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Optimizing neural networks with noisy labels is a challenging task,
especially if the label set contains real-world noise. Networks tend to
generalize to reasonable patterns in the early training stages and overfit to
specific details of noisy samples in the latter ones. We introduce Blind
Knowledge Distillation - a novel teacher-student approach for learning with
noisy labels by masking the ground truth related teacher output to filter out
potentially corrupted knowledge and to estimate the tipping point from
generalizing to overfitting. Based on this, we enable the estimation of noise
in the training data with Otsus algorithm. With this estimation, we train the
network with a modified weighted cross-entropy loss function. We show in our
experiments that Blind Knowledge Distillation detects overfitting effectively
during training and improves the detection of clean and noisy labels on the
recently published CIFAR-N dataset. Code is available at GitHub.
Related papers
- ERASE: Error-Resilient Representation Learning on Graphs for Label Noise
Tolerance [53.73316938815873]
We propose a method called ERASE (Error-Resilient representation learning on graphs for lAbel noiSe tolerancE) to learn representations with error tolerance.
ERASE combines prototype pseudo-labels with propagated denoised labels and updates representations with error resilience.
Our method can outperform multiple baselines with clear margins in broad noise levels and enjoy great scalability.
arXiv Detail & Related papers (2023-12-13T17:59:07Z) - Combating Label Noise With A General Surrogate Model For Sample
Selection [84.61367781175984]
We propose to leverage the vision-language surrogate model CLIP to filter noisy samples automatically.
We validate the effectiveness of our proposed method on both real-world and synthetic noisy datasets.
arXiv Detail & Related papers (2023-10-16T14:43:27Z) - Learning advisor networks for noisy image classification [22.77447144331876]
We introduce the novel concept of advisor network to address the problem of noisy labels in image classification.
We trained it with a meta-learning strategy so that it can adapt throughout the training of the main model.
We tested our method on CIFAR10 and CIFAR100 with synthetic noise, and on Clothing1M which contains real-world noise, reporting state-of-the-art results.
arXiv Detail & Related papers (2022-11-08T11:44:08Z) - Dynamic Adaptive Threshold based Learning for Noisy Annotations Robust
Facial Expression Recognition [3.823356975862006]
We propose a dynamic FER learning framework (DNFER) to handle noisy annotations.
Specifically, DNFER is based on supervised training using selected clean samples and unsupervised consistent training using all the samples.
We demonstrate the robustness of DNFER on both synthetic as well as on real noisy annotated FER datasets like RAFDB, FERPlus, SFEW and AffectNet.
arXiv Detail & Related papers (2022-08-22T12:02:41Z) - Context-based Virtual Adversarial Training for Text Classification with
Noisy Labels [1.9508698179748525]
We propose context-based virtual adversarial training (ConVAT) to prevent a text classifier from overfitting to noisy labels.
Unlike the previous works, the proposed method performs the adversarial training at the context level rather than the inputs.
We conduct extensive experiments on four text classification datasets with two types of label noises.
arXiv Detail & Related papers (2022-05-29T14:19:49Z) - Robust Training under Label Noise by Over-parameterization [41.03008228953627]
We propose a principled approach for robust training of over-parameterized deep networks in classification tasks where a proportion of training labels are corrupted.
The main idea is yet very simple: label noise is sparse and incoherent with the network learned from clean data, so we model the noise and learn to separate it from the data.
Remarkably, when trained using such a simple method in practice, we demonstrate state-of-the-art test accuracy against label noise on a variety of real datasets.
arXiv Detail & Related papers (2022-02-28T18:50:10Z) - Robust Long-Tailed Learning under Label Noise [50.00837134041317]
This work investigates the label noise problem under long-tailed label distribution.
We propose a robust framework,algo, that realizes noise detection for long-tailed learning.
Our framework can naturally leverage semi-supervised learning algorithms to further improve the generalisation.
arXiv Detail & Related papers (2021-08-26T03:45:00Z) - Attention-Aware Noisy Label Learning for Image Classification [97.26664962498887]
Deep convolutional neural networks (CNNs) learned on large-scale labeled samples have achieved remarkable progress in computer vision.
The cheapest way to obtain a large body of labeled visual data is to crawl from websites with user-supplied labels, such as Flickr.
This paper proposes the attention-aware noisy label learning approach to improve the discriminative capability of the network trained on datasets with potential label noise.
arXiv Detail & Related papers (2020-09-30T15:45:36Z) - Temporal Calibrated Regularization for Robust Noisy Label Learning [60.90967240168525]
Deep neural networks (DNNs) exhibit great success on many tasks with the help of large-scale well annotated datasets.
However, labeling large-scale data can be very costly and error-prone so that it is difficult to guarantee the annotation quality.
We propose a Temporal Calibrated Regularization (TCR) in which we utilize the original labels and the predictions in the previous epoch together.
arXiv Detail & Related papers (2020-07-01T04:48:49Z) - Learning with Out-of-Distribution Data for Audio Classification [60.48251022280506]
We show that detecting and relabelling certain OOD instances, rather than discarding them, can have a positive effect on learning.
The proposed method is shown to improve the performance of convolutional neural networks by a significant margin.
arXiv Detail & Related papers (2020-02-11T21:08:06Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.