Achieving Adversarial Robustness Requires An Active Teacher

• URL: http://arxiv.org/abs/2012.07233v1
• Date: Mon, 14 Dec 2020 03:27:50 GMT
• Title: Achieving Adversarial Robustness Requires An Active Teacher
• Abstract summary: We claim that adversarial examples exist because the student cannot obtain sufficient information of the teacher from the training data. To efficiently attain robustness, a teacher that actively provides its information to the student may be necessary.
• Score: 8.193914488276468
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A new understanding of adversarial examples and adversarial robustness is proposed by decoupling the data generator and the label generator (which we call the teacher). In our framework, adversarial robustness is a conditional concept---the student model is not absolutely robust, but robust with respect to the teacher. Based on the new understanding, we claim that adversarial examples exist because the student cannot obtain sufficient information of the teacher from the training data. Various ways of achieving robustness is compared. Theoretical and numerical evidence shows that to efficiently attain robustness, a teacher that actively provides its information to the student may be necessary.

Related papers

• Toward Understanding Adversarial Distillation: Why Robust Teachers Fail [22.989324947501018]
  Adrial Distillation aims to enhance student robustness by guiding the student with a robust teacher's soft labels.<n>We identify a key mechanism of this teacher dependency: the misalignment between the teacher's supervisory confidence and the student's representational limitations.
  arXiv  Detail & Related papers  (2026-05-21T04:57:15Z)
• Sample-wise Adaptive Weighting for Transfer Consistency in Adversarial Distillation [22.989324947501018]
  Existing work often neglects to incorporate state-of-the-art robust teachers.<n>We identify adversarial transferability-the fraction of student-crafted adversarial examples that remain effective against the teacher.<n>We propose Sample-wise Adaptive Adversarial Distillation (SAAD), which reweights training examples by their measured transferability without incurring additional computational cost.
  arXiv  Detail & Related papers  (2025-12-11T04:31:04Z)
• Distilling Adversarial Robustness Using Heterogeneous Teachers [9.404102810698202]
  robustness can be transferred from an adversarially trained teacher to a student model using knowledge distillation. We develop a defense framework against adversarial attacks by distilling robustness using heterogeneous teachers. Experiments on classification tasks in both white-box and black-box scenarios demonstrate that DARHT achieves state-of-the-art clean and robust accuracies.
  arXiv  Detail & Related papers  (2024-02-23T19:55:13Z)
• Good Teachers Explain: Explanation-Enhanced Knowledge Distillation [52.498055901649025]
  Knowledge Distillation (KD) has proven effective for compressing large teacher models into smaller student models. In this work, we explore whether this can be achieved by not only optimizing the classic KD loss but also the similarity of the explanations generated by the teacher and the student. Despite the idea being simple and intuitive, we find that our proposed 'explanation-enhanced' KD consistently provides large gains in terms of accuracy and student-teacher agreement.
  arXiv  Detail & Related papers  (2024-02-05T15:47:54Z)
• Can Language Models Teach Weaker Agents? Teacher Explanations Improve Students via Personalization [84.86241161706911]
  We show that teacher LLMs can indeed intervene on student reasoning to improve their performance. We also demonstrate that in multi-turn interactions, teacher explanations generalize and learn from explained data. We verify that misaligned teachers can lower student performance to random chance by intentionally misleading them.
  arXiv  Detail & Related papers  (2023-06-15T17:27:20Z)
• Faithful Knowledge Distillation [75.59907631395849]
  We focus on two crucial questions with regard to a teacher-student pair: (i) do the teacher and student disagree at points close to correctly classified dataset examples, and (ii) is the distilled student as confident as the teacher around dataset examples? These are critical questions when considering the deployment of a smaller student network trained from a robust teacher within a safety-critical setting.
  arXiv  Detail & Related papers  (2023-06-07T13:41:55Z)
• Generating robust counterfactual explanations [60.32214822437734]
  The quality of a counterfactual depends on several criteria: realism, actionability, validity, robustness, etc. In this paper, we are interested in the notion of robustness of a counterfactual. More precisely, we focus on robustness to counterfactual input changes. We propose a new framework, CROCO, that generates robust counterfactuals while managing effectively this trade-off, and guarantees the user a minimal robustness.
  arXiv  Detail & Related papers  (2023-04-24T09:00:31Z)
• Generalized Knowledge Distillation via Relationship Matching [53.69235109551099]
  Knowledge of a well-trained deep neural network (a.k.a. the "teacher") is valuable for learning similar tasks. Knowledge distillation extracts knowledge from the teacher and integrates it with the target model. Instead of enforcing the teacher to work on the same task as the student, we borrow the knowledge from a teacher trained from a general label space.
  arXiv  Detail & Related papers  (2022-05-04T06:49:47Z)
• On the benefits of knowledge distillation for adversarial robustness [53.41196727255314]
  We show that knowledge distillation can be used directly to boost the performance of state-of-the-art models in adversarial robustness. We present Adversarial Knowledge Distillation (AKD), a new framework to improve a model's robust performance.
  arXiv  Detail & Related papers  (2022-03-14T15:02:13Z)
• Understanding Robustness in Teacher-Student Setting: A New Perspective [42.746182547068265]
  Adrial examples are machine learning models where bounded adversarial perturbation could mislead the models to make arbitrarily incorrect predictions. Extensive studies try to explain the existence of adversarial examples and provide ways to improve model robustness. Our studies could shed light on the future exploration about adversarial examples, and enhancing model robustness via principled data augmentation.
  arXiv  Detail & Related papers  (2021-02-25T20:54:24Z)
• Privacy-Preserving Teacher-Student Deep Reinforcement Learning [23.934121758649052]
  We develop a private mechanism that protects the privacy of the teacher's training dataset. We empirically show that the algorithm improves the student's learning upon convergence rate and utility.
  arXiv  Detail & Related papers  (2021-02-18T20:15:09Z)
• FaceLeaks: Inference Attacks against Transfer Learning Models via Black-box Queries [2.7564955518050693]
  We investigate if one can leak or infer private information without interacting with the teacher model directly. We propose novel strategies to infer from aggregate-level information. Our study indicates that information leakage is a real privacy threat to the transfer learning framework widely used in real-life situations.
  arXiv  Detail & Related papers  (2020-10-27T03:02:40Z)
• Feature Distillation With Guided Adversarial Contrastive Learning [41.28710294669751]
  We propose Guided Adversarial Contrastive Distillation (GACD) to transfer adversarial robustness from teacher to student with features. With a well-trained teacher model as an anchor, students are expected to extract features similar to the teacher. With GACD, the student not only learns to extract robust features, but also captures structural knowledge from the teacher.
  arXiv  Detail & Related papers  (2020-09-21T14:46:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.