Knowledge Distillation: Bad Models Can Be Good Role Models

• URL: http://arxiv.org/abs/2203.14649v1
• Date: Mon, 28 Mar 2022 11:13:53 GMT
• Title: Knowledge Distillation: Bad Models Can Be Good Role Models
• Abstract summary: We study conditional sampling behavior in the context of learning theory. We show that samplers, while being bad classifiers, can be good teachers.
• Score: 35.92225943440252
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Large neural networks trained in the overparameterized regime are able to fit noise to zero train error. Recent work \citep{nakkiran2020distributional} has empirically observed that such networks behave as "conditional samplers" from the noisy distribution. That is, they replicate the noise in the train data to unseen examples. We give a theoretical framework for studying this conditional sampling behavior in the context of learning theory. We relate the notion of such samplers to knowledge distillation, where a student network imitates the outputs of a teacher on unlabeled data. We show that samplers, while being bad classifiers, can be good teachers. Concretely, we prove that distillation from samplers is guaranteed to produce a student which approximates the Bayes optimal classifier. Finally, we show that some common learning algorithms (e.g., Nearest-Neighbours and Kernel Machines) can generate samplers when applied in the overparameterized regime.

Related papers

• Task-Specific Knowledge Distillation via Intermediate Probes [5.183958775837362]
  We introduce method, a framework that trains probes on frozen teacher hidden states and using the probe's predictions, rather than output logits, as supervision for student training.<n>method requires no architectural changes to student or teacher, is architecture-agnostic, and adds minimal compute since probe training is cheap and teacher representations can be cached.
  arXiv  Detail & Related papers  (2026-02-18T10:56:10Z)
• Combating Noisy Labels through Fostering Self- and Neighbor-Consistency [120.4394402099635]
  Label noise is pervasive in various real-world scenarios, posing challenges in supervised deep learning.<n>We propose a noise-robust method named Jo-SNC (textbfJoint sample selection and model regularization based on textbfSelf- and textbfNeighbor-textbfConsistency)<n>We design a self-adaptive, data-driven thresholding scheme to adjust per-class selection thresholds.
  arXiv  Detail & Related papers  (2026-01-19T07:55:29Z)
• Matching the Optimal Denoiser in Point Cloud Diffusion with (Improved) Rotational Alignment [5.8069334875117775]
  Training a diffusion model consists of learning how to denoise noisy samples at different noise levels.<n>We show that the optimal denoiser can be expressed in terms of a matrix Fisher distribution over $SO(3)$.<n>We build on this perspective to derive better approximators to the optimal denoiser in the limit of small noise.
  arXiv  Detail & Related papers  (2025-10-02T05:55:22Z)
• Wide Two-Layer Networks can Learn from Adversarial Perturbations [27.368408524000778]
  We theoretically explain the counterintuitive success of perturbation learning. We prove that adversarial perturbations contain sufficient class-specific features for networks to generalize from them.
  arXiv  Detail & Related papers  (2024-10-31T06:55:57Z)
• Learning Effective Representations for Retrieval Using Self-Distillation with Adaptive Relevance Margins [29.88235846291593]
  Bi-encoders estimate relevance of a document to a query by calculating the similarity of their respective embeddings.<n>Current state-of-the-art bi-encoders are trained using an expensive training regime involving knowledge distillation from a teacher model and batch-sampling.<n>We propose a novel parameter-free loss function for self-supervision that exploits the pre-trained language modeling capabilities of the encoder model as a training signal.
  arXiv  Detail & Related papers  (2024-07-31T10:33:32Z)
• Manifold DivideMix: A Semi-Supervised Contrastive Learning Framework for Severe Label Noise [4.90148689564172]
  Real-world datasets contain noisy label samples that have no semantic relevance to any class in the dataset. Most state-of-the-art methods leverage ID labeled noisy samples as unlabeled data for semi-supervised learning. We propose incorporating the information from all the training data by leveraging the benefits of self-supervised training.
  arXiv  Detail & Related papers  (2023-08-13T23:33:33Z)
• Reducing Training Sample Memorization in GANs by Training with Memorization Rejection [80.0916819303573]
  We propose rejection memorization, a training scheme that rejects generated samples that are near-duplicates of training samples during training. Our scheme is simple, generic and can be directly applied to any GAN architecture.
  arXiv  Detail & Related papers  (2022-10-21T20:17:50Z)
• BatchFormer: Learning to Explore Sample Relationships for Robust Representation Learning [93.38239238988719]
  We propose to enable deep neural networks with the ability to learn the sample relationships from each mini-batch. BatchFormer is applied into the batch dimension of each mini-batch to implicitly explore sample relationships during training. We perform extensive experiments on over ten datasets and the proposed method achieves significant improvements on different data scarcity applications.
  arXiv  Detail & Related papers  (2022-03-03T05:31:33Z)
• Robust Training under Label Noise by Over-parameterization [41.03008228953627]
  We propose a principled approach for robust training of over-parameterized deep networks in classification tasks where a proportion of training labels are corrupted. The main idea is yet very simple: label noise is sparse and incoherent with the network learned from clean data, so we model the noise and learn to separate it from the data. Remarkably, when trained using such a simple method in practice, we demonstrate state-of-the-art test accuracy against label noise on a variety of real datasets.
  arXiv  Detail & Related papers  (2022-02-28T18:50:10Z)
• CrowdTeacher: Robust Co-teaching with Noisy Answers & Sample-specific Perturbations for Tabular Data [8.276156981100364]
  Co-teaching methods have shown promising improvements for computer vision problems with noisy labels. Our model, CrowdTeacher, uses the idea that robustness in the input space model can improve the perturbation of the classifier for noisy labels. We showcase the boost in predictive power attained using CrowdTeacher for both synthetic and real datasets.
  arXiv  Detail & Related papers  (2021-03-31T15:09:38Z)
• Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective [63.87421152879726]
  We investigate the bias-variance tradeoff brought by distillation with soft labels. We propose the novel weighted soft labels to help the network adaptively handle the sample-wise bias-variance tradeoff.
  arXiv  Detail & Related papers  (2021-02-01T05:53:04Z)
• Understanding Classifier Mistakes with Generative Models [88.20470690631372]
  Deep neural networks are effective on supervised learning tasks, but have been shown to be brittle. In this paper, we leverage generative models to identify and characterize instances where classifiers fail to generalize. Our approach is agnostic to class labels from the training set which makes it applicable to models trained in a semi-supervised way.
  arXiv  Detail & Related papers  (2020-10-05T22:13:21Z)
• Automatic Recall Machines: Internal Replay, Continual Learning and the Brain [104.38824285741248]
  Replay in neural networks involves training on sequential data with memorized samples, which counteracts forgetting of previous behavior caused by non-stationarity. We present a method where these auxiliary samples are generated on the fly, given only the model that is being trained for the assessed objective. Instead the implicit memory of learned samples within the assessed model itself is exploited.
  arXiv  Detail & Related papers  (2020-06-22T15:07:06Z)
• Robust and On-the-fly Dataset Denoising for Image Classification [72.10311040730815]
  On-the-fly Data Denoising (ODD) is robust to mislabeled examples, while introducing almost zero computational overhead compared to standard training. ODD is able to achieve state-of-the-art results on a wide range of datasets including real-world ones such as WebVision and Clothing1M.
  arXiv  Detail & Related papers  (2020-03-24T03:59:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.