Reliable Adversarial Distillation with Unreliable Teachers

• URL: http://arxiv.org/abs/2106.04928v1
• Date: Wed, 9 Jun 2021 09:22:39 GMT
• Title: Reliable Adversarial Distillation with Unreliable Teachers
• Abstract summary: We propose reliable introspective adversarial distillation (IAD) where students partially trust their teachers. IAD distinguishes between three cases given a query of a natural data (ND) and the corresponding adversarial data (AD): (a) if a teacher is good at AD, its SL is fully trusted; (b) if a teacher is good at ND but not AD, its SL is partially trusted and the student also takes its own SL into account.
• Score: 93.41159030567466
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In ordinary distillation, student networks are trained with soft labels (SLs) given by pretrained teacher networks, and students are expected to improve upon teachers since SLs are stronger supervision than the original hard labels. However, when considering adversarial robustness, teachers may become unreliable and adversarial distillation may not work: teachers are pretrained on their own adversarial data, and it is too demanding to require that teachers are also good at every adversarial data queried by students. Therefore, in this paper, we propose reliable introspective adversarial distillation (IAD) where students partially instead of fully trust their teachers. Specifically, IAD distinguishes between three cases given a query of a natural data (ND) and the corresponding adversarial data (AD): (a) if a teacher is good at AD, its SL is fully trusted; (b) if a teacher is good at ND but not AD, its SL is partially trusted and the student also takes its own SL into account; (c) otherwise, the student only relies on its own SL. Experiments demonstrate the effectiveness of IAD for improving upon teachers in terms of adversarial robustness.

Related papers

• Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall [79.59596652272923]
  We propose Switch Distillation, a simple mid-training objective that distills on tokens where the teacher is confident.<n>Relative to standard NTP, it achieves 1.61-1.71x the reasoning performance and 1.13-1.19x the knowledge and commonsense performance.
  arXiv  Detail & Related papers  (2026-09-01T17:00:30Z)
• The Strongest Teacher Is Not Always the Best Teacher: Student-Centric Answer Selection [46.38575272632866]
  Student-Centric Answer Sampling (SCAS) is a framework that selects from verified teacher-generated answers according to their estimated student-centric learning cost.<n> Experiments across 30 teacher models, 6 student base models, and 8 tasks show that SCAS consistently improves student performance.
  arXiv  Detail & Related papers  (2026-05-26T11:30:17Z)
• Toward Understanding Adversarial Distillation: Why Robust Teachers Fail [22.989324947501018]
  Adrial Distillation aims to enhance student robustness by guiding the student with a robust teacher's soft labels.<n>We identify a key mechanism of this teacher dependency: the misalignment between the teacher's supervisory confidence and the student's representational limitations.
  arXiv  Detail & Related papers  (2026-05-21T04:57:15Z)
• When Data-Free Knowledge Distillation Meets Non-Transferable Teacher: Escaping Out-of-Distribution Trap is All You Need [78.71039931096959]
  Data-free knowledge distillation (DFKD) transfers knowledge from a teacher to a student without access to the real in-distribution (ID) data.<n>Existing works typically assume teachers are trustworthy, leaving the robustness and security of DFKD from untrusted teachers largely unexplored.<n>We conduct the first investigation into distilling non-transferable learning (NTL) teachers using DFKD.<n>We find that NTL teachers fool DFKD through divert the generator's attention from the useful ID knowledge to the misleading OOD knowledge.
  arXiv  Detail & Related papers  (2025-07-05T18:03:52Z)
• Distilling Adversarial Robustness Using Heterogeneous Teachers [9.404102810698202]
  robustness can be transferred from an adversarially trained teacher to a student model using knowledge distillation. We develop a defense framework against adversarial attacks by distilling robustness using heterogeneous teachers. Experiments on classification tasks in both white-box and black-box scenarios demonstrate that DARHT achieves state-of-the-art clean and robust accuracies.
  arXiv  Detail & Related papers  (2024-02-23T19:55:13Z)
• Faithful Knowledge Distillation [75.59907631395849]
  We focus on two crucial questions with regard to a teacher-student pair: (i) do the teacher and student disagree at points close to correctly classified dataset examples, and (ii) is the distilled student as confident as the teacher around dataset examples? These are critical questions when considering the deployment of a smaller student network trained from a robust teacher within a safety-critical setting.
  arXiv  Detail & Related papers  (2023-06-07T13:41:55Z)
• On student-teacher deviations in distillation: does it pay to disobey? [54.908344098305804]
  Knowledge distillation has been widely used to improve the test accuracy of a "student" network. Despite being trained to fit the teacher's probabilities, the student may not only significantly deviate from the teacher probabilities, but may also outdo the teacher in performance.
  arXiv  Detail & Related papers  (2023-01-30T14:25:02Z)
• Sparse Teachers Can Be Dense with Knowledge [35.83646432932867]
  We propose a sparse teacher trick under the guidance of an overall knowledgable score for each teacher parameter. The aim is to ensure that the expressive parameters are retained while the student-unfriendly ones are removed. Experiments on the GLUE benchmark show that the proposed sparse teachers can be dense with knowledge and lead to students with compelling performance.
  arXiv  Detail & Related papers  (2022-10-08T05:25:34Z)
• Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillation [70.92135839545314]
  We propose the dynamic prior knowledge (DPK), which integrates part of teacher's features as the prior knowledge before the feature distillation. Our DPK makes the performance of the student model positively correlated with that of the teacher model, which means that we can further boost the accuracy of students by applying larger teachers.
  arXiv  Detail & Related papers  (2022-06-13T11:52:13Z)
• Generalized Knowledge Distillation via Relationship Matching [53.69235109551099]
  Knowledge of a well-trained deep neural network (a.k.a. the "teacher") is valuable for learning similar tasks. Knowledge distillation extracts knowledge from the teacher and integrates it with the target model. Instead of enforcing the teacher to work on the same task as the student, we borrow the knowledge from a teacher trained from a general label space.
  arXiv  Detail & Related papers  (2022-05-04T06:49:47Z)
• Does Knowledge Distillation Really Work? [106.38447017262183]
  We show that while knowledge distillation can improve student generalization, it does not typically work as it is commonly understood. We identify difficulties in optimization as a key reason for why the student is unable to match the teacher.
  arXiv  Detail & Related papers  (2021-06-10T17:44:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.