On the benefits of knowledge distillation for adversarial robustness
- URL: http://arxiv.org/abs/2203.07159v1
- Date: Mon, 14 Mar 2022 15:02:13 GMT
- Title: On the benefits of knowledge distillation for adversarial robustness
- Authors: Javier Maroto, Guillermo Ortiz-Jim\'enez and Pascal Frossard
- Abstract summary: We show that knowledge distillation can be used directly to boost the performance of state-of-the-art models in adversarial robustness.
We present Adversarial Knowledge Distillation (AKD), a new framework to improve a model's robust performance.
- Score: 53.41196727255314
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Knowledge distillation is normally used to compress a big network, or
teacher, onto a smaller one, the student, by training it to match its outputs.
Recently, some works have shown that robustness against adversarial attacks can
also be distilled effectively to achieve good rates of robustness on
mobile-friendly models. In this work, however, we take a different point of
view, and show that knowledge distillation can be used directly to boost the
performance of state-of-the-art models in adversarial robustness. In this
sense, we present a thorough analysis and provide general guidelines to distill
knowledge from a robust teacher and boost the clean and adversarial performance
of a student model even further. To that end, we present Adversarial Knowledge
Distillation (AKD), a new framework to improve a model's robust performance,
consisting on adversarially training a student on a mixture of the original
labels and the teacher outputs. Through carefully controlled ablation studies,
we show that using early-stopping, model ensembles and weak adversarial
training are key techniques to maximize performance of the student, and show
that these insights generalize across different robust distillation techniques.
Finally, we provide insights on the effect of robust knowledge distillation on
the dynamics of the student network, and show that AKD mostly improves the
calibration of the network and modify its training dynamics on samples that the
model finds difficult to learn, or even memorize.
Related papers
- Dynamic Guidance Adversarial Distillation with Enhanced Teacher Knowledge [17.382306203152943]
Dynamic Guidance Adversarial Distillation (DGAD) framework tackles the challenge of differential sample importance.
DGAD employs Misclassification-Aware Partitioning (MAP) to dynamically tailor the distillation focus.
Error-corrective Label Swapping (ELS) corrects misclassifications of the teacher on both clean and adversarially perturbed inputs.
arXiv Detail & Related papers (2024-09-03T05:52:37Z) - Knowledge Distillation via Token-level Relationship Graph [12.356770685214498]
We propose a novel method called Knowledge Distillation with Token-level Relationship Graph (TRG)
By employing TRG, the student model can effectively emulate higher-level semantic information from the teacher model.
We conduct experiments to evaluate the effectiveness of the proposed method against several state-of-the-art approaches.
arXiv Detail & Related papers (2023-06-20T08:16:37Z) - Tailoring Instructions to Student's Learning Levels Boosts Knowledge Distillation [52.53446712834569]
Learning Good Teacher Matters (LGTM) is an efficient training technique for incorporating distillation influence into the teacher's learning process.
Our LGTM outperforms 10 common knowledge distillation baselines on 6 text classification tasks in the GLUE benchmark.
arXiv Detail & Related papers (2023-05-16T17:50:09Z) - Distillation from Heterogeneous Models for Top-K Recommendation [43.83625440616829]
HetComp is a framework that guides the student model by transferring sequences of knowledge from teachers' trajectories.
HetComp significantly improves the distillation quality and the generalization of the student model.
arXiv Detail & Related papers (2023-03-02T10:23:50Z) - ARDIR: Improving Robustness using Knowledge Distillation of Internal
Representation [2.0875529088206553]
We propose Adversarial Robust Distillation with Internal Representation(ARDIR) to utilize knowledge distillation even more effectively.
ARDIR uses the internal representation of the teacher model as a label for adversarial training.
We show that ARDIR outperforms previous methods in our experiments.
arXiv Detail & Related papers (2022-11-01T03:11:59Z) - Dynamic Rectification Knowledge Distillation [0.0]
Dynamic Rectification Knowledge Distillation (DR-KD) is a knowledge distillation framework.
DR-KD transforms the student into its own teacher, and if the self-teacher makes wrong predictions while distilling information, the error is rectified prior to the knowledge being distilled.
Our proposed DR-KD performs remarkably well in the absence of a sophisticated cumbersome teacher model.
arXiv Detail & Related papers (2022-01-27T04:38:01Z) - How and When Adversarial Robustness Transfers in Knowledge Distillation? [137.11016173468457]
This paper studies how and when the adversarial robustness can be transferred from a teacher model to a student model in Knowledge distillation (KD)
We show that standard KD training fails to preserve adversarial robustness, and we propose KD with input gradient alignment (KDIGA) for remedy.
Under certain assumptions, we prove that the student model using our proposed KDIGA can achieve at least the same certified robustness as the teacher model.
arXiv Detail & Related papers (2021-10-22T21:30:53Z) - Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make
Student Better [66.69777970159558]
We propose a novel adversarial robustness distillation method called Robust Soft Label Adversarial Distillation (RSLAD)
RSLAD fully exploits the robust soft labels produced by a robust (adversarially-trained) large teacher model to guide the student's learning.
We empirically demonstrate the effectiveness of our RSLAD approach over existing adversarial training and distillation methods.
arXiv Detail & Related papers (2021-08-18T04:32:35Z) - Learning Student-Friendly Teacher Networks for Knowledge Distillation [50.11640959363315]
We propose a novel knowledge distillation approach to facilitate the transfer of dark knowledge from a teacher to a student.
Contrary to most of the existing methods that rely on effective training of student models given pretrained teachers, we aim to learn the teacher models that are friendly to students.
arXiv Detail & Related papers (2021-02-12T07:00:17Z) - Knowledge Distillation Meets Self-Supervision [109.6400639148393]
Knowledge distillation involves extracting "dark knowledge" from a teacher network to guide the learning of a student network.
We show that the seemingly different self-supervision task can serve as a simple yet powerful solution.
By exploiting the similarity between those self-supervision signals as an auxiliary task, one can effectively transfer the hidden information from the teacher to the student.
arXiv Detail & Related papers (2020-06-12T12:18:52Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.