PeerAiD: Improving Adversarial Distillation from a Specialized Peer Tutor

• URL: http://arxiv.org/abs/2403.06668v3
• Date: Fri, 17 May 2024 05:29:05 GMT
• Title: PeerAiD: Improving Adversarial Distillation from a Specialized Peer Tutor
• Authors: Jaewon Jung, Hongsun Jang, Jaeyong Song, Jinho Lee,
• Abstract summary: Adversarial robustness of the neural network is a significant concern when it is applied to security-critical domains. Previous works pretrain the teacher network to make it robust against the adversarial examples aimed at itself. We propose PeerAiD to make a peer network learn the adversarial examples of the student network instead of adversarial examples aimed at itself.
• Score: 6.089685202183291
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Adversarial robustness of the neural network is a significant concern when it is applied to security-critical domains. In this situation, adversarial distillation is a promising option which aims to distill the robustness of the teacher network to improve the robustness of a small student network. Previous works pretrain the teacher network to make it robust against the adversarial examples aimed at itself. However, the adversarial examples are dependent on the parameters of the target network. The fixed teacher network inevitably degrades its robustness against the unseen transferred adversarial examples which target the parameters of the student network in the adversarial distillation process. We propose PeerAiD to make a peer network learn the adversarial examples of the student network instead of adversarial examples aimed at itself. PeerAiD is an adversarial distillation that trains the peer network and the student network simultaneously in order to specialize the peer network for defending the student network. We observe that such peer networks surpass the robustness of the pretrained robust teacher model against adversarial examples aimed at the student network. With this peer network and adversarial distillation, PeerAiD achieves significantly higher robustness of the student network with AutoAttack (AA) accuracy by up to 1.66%p and improves the natural accuracy of the student network by up to 4.72%p with ResNet-18 on TinyImageNet dataset. Code is available at https://github.com/jaewonalive/PeerAiD.

Related papers

• Adaptive Teaching with Shared Classifier for Knowledge Distillation [6.03477652126575]
  Knowledge distillation (KD) is a technique used to transfer knowledge from a teacher network to a student network. We propose adaptive teaching with a shared classifier (ATSC) Our approach achieves state-of-the-art results on the CIFAR-100 and ImageNet datasets in both single-teacher and multiteacher scenarios.
  arXiv  Detail & Related papers  (2024-06-12T08:51:08Z)
• Distilling Adversarial Robustness Using Heterogeneous Teachers [9.404102810698202]
  robustness can be transferred from an adversarially trained teacher to a student model using knowledge distillation. We develop a defense framework against adversarial attacks by distilling robustness using heterogeneous teachers. Experiments on classification tasks in both white-box and black-box scenarios demonstrate that DARHT achieves state-of-the-art clean and robust accuracies.
  arXiv  Detail & Related papers  (2024-02-23T19:55:13Z)
• Opinion Control under Adversarial Network Perturbation: A Stackelberg Game Approach [12.916992671437017]
  adversarial network perturbation greatly influences the opinion formation of the public and threatens our societies. In this work, we model the adversarial network perturbation and analyze its impact on the networks' opinion. From the adversary's perspective, we formulate a Stackelberg game and aim to control the network's opinion even under such adversarial network perturbation.
  arXiv  Detail & Related papers  (2023-04-25T03:14:39Z)
• Excess Risk of Two-Layer ReLU Neural Networks in Teacher-Student Settings and its Superiority to Kernel Methods [58.44819696433327]
  We investigate the risk of two-layer ReLU neural networks in a teacher regression model. We find that the student network provably outperforms any solution methods.
  arXiv  Detail & Related papers  (2022-05-30T02:51:36Z)
• Improving Corruption and Adversarial Robustness by Enhancing Weak Subnets [91.9346332103637]
  We propose a novel robust training method which explicitly identifies and enhances weaks during training to improve robustness. Specifically, we develop a search algorithm to find particularly weaks and propose to explicitly strengthen them via knowledge distillation from the full network. We show that our EWS greatly improves the robustness against corrupted images as well as the accuracy on clean data.
  arXiv  Detail & Related papers  (2022-01-30T09:36:19Z)
• Faster Deep Reinforcement Learning with Slower Online Network [90.34900072689618]
  We endow two popular deep reinforcement learning algorithms, namely DQN and Rainbow, with updates that incentivize the online network to remain in the proximity of the target network. The resultant agents, called DQN Pro and Rainbow Pro, exhibit significant performance improvements over their original counterparts on the Atari benchmark.
  arXiv  Detail & Related papers  (2021-12-10T22:07:06Z)
• Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networks [98.21130211336964]
  Deep neural networks (DNNs) are known to be vulnerable to adversarial attacks. In this paper, we investigate the impact of network width and depth on the robustness of adversarially trained DNNs.
  arXiv  Detail & Related papers  (2021-10-07T23:13:33Z)
• Robust Ensembling Network for Unsupervised Domain Adaptation [20.152004296679138]
  We propose a Robust Ensembling Network (REN) for unsupervised domain adaptation (UDA) REN mainly includes a teacher network and a student network, which performs standard domain adaptation training and updates weights of the teacher network. For the purpose of improving the basic ability of the student network, we utilize the consistency constraint to balance the error between the student network and the teacher network.
  arXiv  Detail & Related papers  (2021-08-21T09:19:13Z)
• PQK: Model Compression via Pruning, Quantization, and Knowledge Distillation [43.45412122086056]
  We propose a novel model compression method called PQK consisting of pruning, quantization, and knowledge distillation processes. PQK makes use of unimportant weights pruned in the pruning process to make a teacher network for training a better student network without pre-training the teacher model. We apply our method to the recognition model and verify the effectiveness of PQK on keyword spotting (KWS) and image recognition.
  arXiv  Detail & Related papers  (2021-06-25T07:24:53Z)
• Rethinking Clustering for Robustness [56.14672993686335]
  ClusTR is a clustering-based and adversary-free training framework to learn robust models. textitClusTR outperforms adversarially-trained networks by up to $4%$ under strong PGD attacks.
  arXiv  Detail & Related papers  (2020-06-13T16:55:51Z)
• Towards Achieving Adversarial Robustness by Enforcing Feature Consistency Across Bit Planes [51.31334977346847]
  We train networks to form coarse impressions based on the information in higher bit planes, and use the lower bit planes only to refine their prediction. We demonstrate that, by imposing consistency on the representations learned across differently quantized images, the adversarial robustness of networks improves significantly.
  arXiv  Detail & Related papers  (2020-04-01T09:31:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.