Long-tailed Adversarial Training with Self-Distillation

• URL: http://arxiv.org/abs/2503.06461v1
• Date: Sun, 09 Mar 2025 05:39:36 GMT
• Title: Long-tailed Adversarial Training with Self-Distillation
• Authors: Seungju Cho, Hongsin Lee, Changick Kim,
• Abstract summary: We show that adversarial training struggles to achieve high performance on tail classes in long-tailed distributions.<n>We propose a novel self-distillation technique to advance adversarial robustness on long-tailed distributions.<n>Our experiments demonstrate state-of-the-art performance in both clean and robust accuracy for long-tailed adversarial robustness.
• Score: 15.184564265850382
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Adversarial training significantly enhances adversarial robustness, yet superior performance is predominantly achieved on balanced datasets. Addressing adversarial robustness in the context of unbalanced or long-tailed distributions is considerably more challenging, mainly due to the scarcity of tail data instances. Previous research on adversarial robustness within long-tailed distributions has primarily focused on combining traditional long-tailed natural training with existing adversarial robustness methods. In this study, we provide an in-depth analysis for the challenge that adversarial training struggles to achieve high performance on tail classes in long-tailed distributions. Furthermore, we propose a simple yet effective solution to advance adversarial robustness on long-tailed distributions through a novel self-distillation technique. Specifically, this approach leverages a balanced self-teacher model, which is trained using a balanced dataset sampled from the original long-tailed dataset. Our extensive experiments demonstrate state-of-the-art performance in both clean and robust accuracy for long-tailed adversarial robustness, with significant improvements in tail class performance on various datasets. We improve the accuracy against PGD attacks for tail classes by 20.3, 7.1, and 3.8 percentage points on CIFAR-10, CIFAR-100, and Tiny-ImageNet, respectively, while achieving the highest robust accuracy.

Related papers

• TAET: Two-Stage Adversarial Equalization Training on Long-Tailed Distributions [3.9635480458924994]
  Adversarial robustness is a critical challenge in deploying deep neural networks for real-world applications.<n>We propose a novel training framework, TAET, which integrates an initial stabilization phase followed by a stratified adversarial training phase.<n>Our method surpasses existing advanced defenses, achieving significant improvements in both memory and computational efficiency.
  arXiv  Detail & Related papers  (2025-03-02T12:07:00Z)
• Revisiting Adversarial Training under Long-Tailed Distributions [8.187045490998269]
  Adversarial training has been recognized as one of the most effective methods to counter such attacks. We show that Balanced Softmax Loss alone can achieve performance comparable to the complete RoBal approach while significantly reducing training overheads. To address this, we explore data augmentation as a solution and unexpectedly discover that, unlike results obtained with balanced data, data augmentation not only effectively alleviates robust overfitting but also significantly improves robustness.
  arXiv  Detail & Related papers  (2024-03-15T07:29:41Z)
• Orthogonal Uncertainty Representation of Data Manifold for Robust Long-Tailed Learning [52.021899899683675]
  In scenarios with long-tailed distributions, the model's ability to identify tail classes is limited due to the under-representation of tail samples. We propose an Orthogonal Uncertainty Representation (OUR) of feature embedding and an end-to-end training strategy to improve the long-tail phenomenon of model robustness.
  arXiv  Detail & Related papers  (2023-10-16T05:50:34Z)
• Alleviating the Effect of Data Imbalance on Adversarial Training [26.36714114672729]
  We study adversarial training on datasets that obey the long-tailed distribution. We propose a new adversarial training framework -- Re-balancing Adversarial Training (REAT)
  arXiv  Detail & Related papers  (2023-07-14T07:01:48Z)
• Enhancing Adversarial Training via Reweighting Optimization Trajectory [72.75558017802788]
  A number of approaches have been proposed to address drawbacks such as extra regularization, adversarial weights, and training with more data. We propose a new method named textbfWeighted Optimization Trajectories (WOT) that leverages the optimization trajectories of adversarial training in time. Our results show that WOT integrates seamlessly with the existing adversarial training methods and consistently overcomes the robust overfitting issue.
  arXiv  Detail & Related papers  (2023-06-25T15:53:31Z)
• Adversarial Robustness under Long-Tailed Distribution [93.50792075460336]
  Adversarial robustness has attracted extensive studies recently by revealing the vulnerability and intrinsic characteristics of deep networks. In this work we investigate the adversarial vulnerability as well as defense under long-tailed distributions. We propose a clean yet effective framework, RoBal, which consists of two dedicated modules, a scale-invariant and data re-balancing.
  arXiv  Detail & Related papers  (2021-04-06T17:53:08Z)
• Robust Pre-Training by Adversarial Contrastive Learning [120.33706897927391]
  Recent work has shown that, when integrated with adversarial training, self-supervised pre-training can lead to state-of-the-art robustness. We improve robustness-aware self-supervised pre-training by learning representations consistent under both data augmentations and adversarial perturbations.
  arXiv  Detail & Related papers  (2020-10-26T04:44:43Z)
• Long-tailed Recognition by Routing Diverse Distribution-Aware Experts [64.71102030006422]
  We propose a new long-tailed classifier called RoutIng Diverse Experts (RIDE) It reduces the model variance with multiple experts, reduces the model bias with a distribution-aware diversity loss, reduces the computational cost with a dynamic expert routing module. RIDE outperforms the state-of-the-art by 5% to 7% on CIFAR100-LT, ImageNet-LT and iNaturalist 2018 benchmarks.
  arXiv  Detail & Related papers  (2020-10-05T06:53:44Z)
• Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning [134.15174177472807]
  We introduce adversarial training into self-supervision, to provide general-purpose robust pre-trained models for the first time. We conduct extensive experiments to demonstrate that the proposed framework achieves large performance margins.
  arXiv  Detail & Related papers  (2020-03-28T18:28:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.