Improving Adversarial Robust Fairness via Anti-Bias Soft Label Distillation

• URL: http://arxiv.org/abs/2312.05508v3
• Date: Thu, 31 Oct 2024 09:04:41 GMT
• Title: Improving Adversarial Robust Fairness via Anti-Bias Soft Label Distillation
• Authors: Shiji Zhao, Ranjie Duan, Xizhe Wang, Xingxing Wei,
• Abstract summary: Adversarial Training (AT) has been widely proved to be an effective method to improve the adversarial robustness against adversarial examples for Deep Neural Networks (DNNs) As a variant of AT, Adversarial Robustness Distillation (ARD) has demonstrated its superior performance in improving the robustness of small student models. We propose an Anti-Bias Soft Label Distillation (ABSLD) method to mitigate the adversarial robust fairness problem within the framework of Knowledge Distillation (KD)
• Score: 14.163463596459064
• License:
• Abstract: Adversarial Training (AT) has been widely proved to be an effective method to improve the adversarial robustness against adversarial examples for Deep Neural Networks (DNNs). As a variant of AT, Adversarial Robustness Distillation (ARD) has demonstrated its superior performance in improving the robustness of small student models with the guidance of large teacher models. However, both AT and ARD encounter the robust fairness problem: these models exhibit strong robustness when facing part of classes (easy class), but weak robustness when facing others (hard class). In this paper, we give an in-depth analysis of the potential factors and argue that the smoothness degree of samples' soft labels for different classes (i.e., hard class or easy class) will affect the robust fairness of DNNs from both empirical observation and theoretical analysis. Based on the above finding, we propose an Anti-Bias Soft Label Distillation (ABSLD) method to mitigate the adversarial robust fairness problem within the framework of Knowledge Distillation (KD). Specifically, ABSLD adaptively reduces the student's error risk gap between different classes to achieve fairness by adjusting the class-wise smoothness degree of samples' soft labels during the training process, and the smoothness degree of soft labels is controlled by assigning different temperatures in KD to different classes. Extensive experiments demonstrate that ABSLD outperforms state-of-the-art AT, ARD, and robust fairness methods in the comprehensive metric (Normalized Standard Deviation) of robustness and fairness.

Related papers

• Soften to Defend: Towards Adversarial Robustness via Self-Guided Label Refinement [5.865750284677784]
  Adversarial training (AT) is one of the most effective ways to obtain the robustness of deep neural networks against adversarial attacks. AT methods suffer from robust overfitting, i.e., a significant generalization gap between the training and testing curves. We propose a label refinement approach for AT, which self-refines a more accurate and informative label distribution from over-confident hard labels.
  arXiv  Detail & Related papers  (2024-03-14T04:48:31Z)
• Towards Robust Federated Learning via Logits Calibration on Non-IID Data [49.286558007937856]
  Federated learning (FL) is a privacy-preserving distributed management framework based on collaborative model training of distributed devices in edge networks. Recent studies have shown that FL is vulnerable to adversarial examples, leading to a significant drop in its performance. In this work, we adopt the adversarial training (AT) framework to improve the robustness of FL models against adversarial example (AE) attacks.
  arXiv  Detail & Related papers  (2024-03-05T09:18:29Z)
• Towards Fairness-Aware Adversarial Learning [13.932705960012846]
  We propose a novel learning paradigm, named Fairness-Aware Adversarial Learning (FAAL) Our method aims to find the worst distribution among different categories, and the solution is guaranteed to obtain the upper bound performance with high probability. In particular, FAAL can fine-tune an unfair robust model to be fair within only two epochs, without compromising the overall clean and robust accuracies.
  arXiv  Detail & Related papers  (2024-02-27T18:01:59Z)
• DAFA: Distance-Aware Fair Adversarial Training [34.94780532071229]
  Under adversarial attacks, the majority of the model's predictions for samples from the worst class are biased towards classes similar to the worst class. We introduce the Distance-Aware Fair Adversarial training (DAFA) methodology, which addresses robust fairness by taking into account the similarities between classes.
  arXiv  Detail & Related papers  (2024-01-23T07:15:47Z)
• Doubly Robust Instance-Reweighted Adversarial Training [107.40683655362285]
  We propose a novel doubly-robust instance reweighted adversarial framework. Our importance weights are obtained by optimizing the KL-divergence regularized loss function. Our proposed approach outperforms related state-of-the-art baseline methods in terms of average robust performance.
  arXiv  Detail & Related papers  (2023-08-01T06:16:18Z)
• Adversarial Contrastive Distillation with Adaptive Denoising [15.119013995045192]
  We propose Contrastive Relationship DeNoise Distillation (CRDND) to boost the robustness of small models. We show CRDND can transfer robust knowledge efficiently and achieves state-of-the-art performances.
  arXiv  Detail & Related papers  (2023-02-17T09:00:18Z)
• SoftMatch: Addressing the Quantity-Quality Trade-off in Semi-supervised Learning [101.86916775218403]
  This paper revisits the popular pseudo-labeling methods via a unified sample weighting formulation. We propose SoftMatch to overcome the trade-off by maintaining both high quantity and high quality of pseudo-labels during training. In experiments, SoftMatch shows substantial improvements across a wide variety of benchmarks, including image, text, and imbalanced classification.
  arXiv  Detail & Related papers  (2023-01-26T03:53:25Z)
• Improving Robust Fairness via Balance Adversarial Training [51.67643171193376]
  Adversarial training (AT) methods are effective against adversarial attacks, yet they introduce severe disparity of accuracy and robustness between different classes. We propose Adversarial Training (BAT) to address the robust fairness problem.
  arXiv  Detail & Related papers  (2022-09-15T14:44:48Z)
• Scale-Equivalent Distillation for Semi-Supervised Object Detection [57.59525453301374]
  Recent Semi-Supervised Object Detection (SS-OD) methods are mainly based on self-training, generating hard pseudo-labels by a teacher model on unlabeled data as supervisory signals. We analyze the challenges these methods meet with the empirical experiment results. We introduce a novel approach, Scale-Equivalent Distillation (SED), which is a simple yet effective end-to-end knowledge distillation framework robust to large object size variance and class imbalance.
  arXiv  Detail & Related papers  (2022-03-23T07:33:37Z)
• Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Better [66.69777970159558]
  We propose a novel adversarial robustness distillation method called Robust Soft Label Adversarial Distillation (RSLAD) RSLAD fully exploits the robust soft labels produced by a robust (adversarially-trained) large teacher model to guide the student's learning. We empirically demonstrate the effectiveness of our RSLAD approach over existing adversarial training and distillation methods.
  arXiv  Detail & Related papers  (2021-08-18T04:32:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.