Soften to Defend: Towards Adversarial Robustness via Self-Guided Label Refinement

• URL: http://arxiv.org/abs/2403.09101v1
• Date: Thu, 14 Mar 2024 04:48:31 GMT
• Title: Soften to Defend: Towards Adversarial Robustness via Self-Guided Label Refinement
• Authors: Daiwei Yu, Zhuorong Li, Lina Wei, Canghong Jin, Yun Zhang, Sixian Chan,
• Abstract summary: Adversarial training (AT) is one of the most effective ways to obtain the robustness of deep neural networks against adversarial attacks. AT methods suffer from robust overfitting, i.e., a significant generalization gap between the training and testing curves. We propose a label refinement approach for AT, which self-refines a more accurate and informative label distribution from over-confident hard labels.
• Score: 5.865750284677784
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Adversarial training (AT) is currently one of the most effective ways to obtain the robustness of deep neural networks against adversarial attacks. However, most AT methods suffer from robust overfitting, i.e., a significant generalization gap in adversarial robustness between the training and testing curves. In this paper, we first identify a connection between robust overfitting and the excessive memorization of noisy labels in AT from a view of gradient norm. As such label noise is mainly caused by a distribution mismatch and improper label assignments, we are motivated to propose a label refinement approach for AT. Specifically, our Self-Guided Label Refinement first self-refines a more accurate and informative label distribution from over-confident hard labels, and then it calibrates the training by dynamically incorporating knowledge from self-distilled models into the current model and thus requiring no external teachers. Empirical results demonstrate that our method can simultaneously boost the standard accuracy and robust performance across multiple benchmark datasets, attack types, and architectures. In addition, we also provide a set of analyses from the perspectives of information theory to dive into our method and suggest the importance of soft labels for robust generalization.

Related papers

• Learning with Confidence: Training Better Classifiers from Soft Labels [0.0]
  In supervised machine learning, models are typically trained using data with hard labels, i.e., definite assignments of class membership. We investigate whether incorporating label uncertainty, represented as discrete probability distributions over the class labels, improves the predictive performance of classification models.
  arXiv  Detail & Related papers  (2024-09-24T13:12:29Z)
• Mitigating Label Noise through Data Ambiguation [9.51828574518325]
  Large models with high expressive power are prone to memorizing incorrect labels, thereby harming generalization performance. In this paper, we suggest to address the shortcomings of both methodologies by "ambiguating" the target information. More precisely, we leverage the framework of so-called superset learning to construct set-valued targets based on a confidence threshold.
  arXiv  Detail & Related papers  (2023-05-23T07:29:08Z)
• SoftMatch: Addressing the Quantity-Quality Trade-off in Semi-supervised Learning [101.86916775218403]
  This paper revisits the popular pseudo-labeling methods via a unified sample weighting formulation. We propose SoftMatch to overcome the trade-off by maintaining both high quantity and high quality of pseudo-labels during training. In experiments, SoftMatch shows substantial improvements across a wide variety of benchmarks, including image, text, and imbalanced classification.
  arXiv  Detail & Related papers  (2023-01-26T03:53:25Z)
• Exploiting Completeness and Uncertainty of Pseudo Labels for Weakly Supervised Video Anomaly Detection [149.23913018423022]
  Weakly supervised video anomaly detection aims to identify abnormal events in videos using only video-level labels. Two-stage self-training methods have achieved significant improvements by self-generating pseudo labels. We propose an enhancement framework by exploiting completeness and uncertainty properties for effective self-training.
  arXiv  Detail & Related papers  (2022-12-08T05:53:53Z)
• Unsupervised Domain Adaptive Salient Object Detection Through Uncertainty-Aware Pseudo-Label Learning [104.00026716576546]
  We propose to learn saliency from synthetic but clean labels, which naturally has higher pixel-labeling quality without the effort of manual annotations. We show that our proposed method outperforms the existing state-of-the-art deep unsupervised SOD methods on several benchmark datasets.
  arXiv  Detail & Related papers  (2022-02-26T16:03:55Z)
• Debiased Pseudo Labeling in Self-Training [77.83549261035277]
  Deep neural networks achieve remarkable performances on a wide range of tasks with the aid of large-scale labeled datasets. To mitigate the requirement for labeled data, self-training is widely used in both academia and industry by pseudo labeling on readily-available unlabeled data. We propose Debiased, in which the generation and utilization of pseudo labels are decoupled by two independent heads.
  arXiv  Detail & Related papers  (2022-02-15T02:14:33Z)
• Exploring Memorization in Adversarial Training [58.38336773082818]
  We investigate the memorization effect in adversarial training (AT) for promoting a deeper understanding of capacity, convergence, generalization, and especially robust overfitting. We propose a new mitigation algorithm motivated by detailed memorization analyses.
  arXiv  Detail & Related papers  (2021-06-03T05:39:57Z)
• STRUDEL: Self-Training with Uncertainty Dependent Label Refinement across Domains [4.812718493682454]
  We propose an unsupervised domain adaptation (UDA) approach for white matter hyperintensity (WMH) segmentation. We propose to predict the uncertainty of pseudo labels and integrate it in the training process with an uncertainty-guided loss function to highlight labels with high certainty. Our results on WMH segmentation across datasets demonstrate the significant improvement of STRUDEL with respect to standard self-training.
  arXiv  Detail & Related papers  (2021-04-23T13:46:26Z)
• Noisy Concurrent Training for Efficient Learning under Label Noise [13.041607703862724]
  Deep neural networks (DNNs) fail to learn effectively under label noise and have been shown to memorize random labels which affect their performance. We consider learning in isolation, using one-hot encoded labels as the sole source of supervision, and a lack of regularization to discourage memorization as the major shortcomings of the standard training procedure. We propose Noisy Concurrent Training (NCT) which leverages collaborative learning to use the consensus between two models as an additional source of supervision.
  arXiv  Detail & Related papers  (2020-09-17T14:22:17Z)
• Adversarial Self-Supervised Contrastive Learning [62.17538130778111]
  Existing adversarial learning approaches mostly use class labels to generate adversarial samples that lead to incorrect predictions. We propose a novel adversarial attack for unlabeled data, which makes the model confuse the instance-level identities of the perturbed data samples. We present a self-supervised contrastive learning framework to adversarially train a robust neural network without labeled data.
  arXiv  Detail & Related papers  (2020-06-13T08:24:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.