Conserve-Update-Revise to Cure Generalization and Robustness Trade-off in Adversarial Training

• URL: http://arxiv.org/abs/2401.14948v1
• Date: Fri, 26 Jan 2024 15:33:39 GMT
• Title: Conserve-Update-Revise to Cure Generalization and Robustness Trade-off in Adversarial Training
• Authors: Shruthi Gowda, Bahram Zonooz, Elahe Arani
• Abstract summary: Adrial training improves the robustness of neural networks against adversarial attacks. We show that selectively updating specific layers while preserving others can substantially enhance the network's learning capacity. We propose CURE, a novel training framework that leverages a gradient prominence criterion to perform selective conservation, updating, and revision of weights.
• Score: 21.163070161951868
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Adversarial training improves the robustness of neural networks against adversarial attacks, albeit at the expense of the trade-off between standard and robust generalization. To unveil the underlying factors driving this phenomenon, we examine the layer-wise learning capabilities of neural networks during the transition from a standard to an adversarial setting. Our empirical findings demonstrate that selectively updating specific layers while preserving others can substantially enhance the network's learning capacity. We therefore propose CURE, a novel training framework that leverages a gradient prominence criterion to perform selective conservation, updating, and revision of weights. Importantly, CURE is designed to be dataset- and architecture-agnostic, ensuring its applicability across various scenarios. It effectively tackles both memorization and overfitting issues, thus enhancing the trade-off between robustness and generalization and additionally, this training approach also aids in mitigating "robust overfitting". Furthermore, our study provides valuable insights into the mechanisms of selective adversarial training and offers a promising avenue for future research.

Related papers

• Adversarial Training Can Provably Improve Robustness: Theoretical Analysis of Feature Learning Process Under Structured Data [38.44734564565478]
  We provide a theoretical understanding of adversarial examples and adversarial training algorithms from the perspective of feature learning theory. We show that the adversarial training method can provably strengthen the robust feature learning and suppress the non-robust feature learning.
  arXiv  Detail & Related papers  (2024-10-11T03:59:49Z)
• Beyond Dropout: Robust Convolutional Neural Networks Based on Local Feature Masking [6.189613073024831]
  This study introduces an innovative Local Feature Masking (LFM) strategy aimed at fortifying the performance of Convolutional Neural Networks (CNNs) During the training phase, we strategically incorporate random feature masking in the shallow layers of CNNs. LFM compels the network to adapt by leveraging remaining features to compensate for the absence of certain semantic features.
  arXiv  Detail & Related papers  (2024-07-18T16:25:16Z)
• A Unified and General Framework for Continual Learning [58.72671755989431]
  Continual Learning (CL) focuses on learning from dynamic and changing data distributions while retaining previously acquired knowledge. Various methods have been developed to address the challenge of catastrophic forgetting, including regularization-based, Bayesian-based, and memory-replay-based techniques. This research aims to bridge this gap by introducing a comprehensive and overarching framework that encompasses and reconciles these existing methodologies.
  arXiv  Detail & Related papers  (2024-03-20T02:21:44Z)
• Towards Improving Robustness Against Common Corruptions using Mixture of Class Specific Experts [10.27974860479791]
  This paper introduces a novel paradigm known as the Mixture of Class-Specific Expert Architecture. The proposed architecture aims to mitigate vulnerabilities associated with common neural network structures.
  arXiv  Detail & Related papers  (2023-11-16T20:09:47Z)
• Adversarial Attacks and Defenses in Machine Learning-Powered Networks: A Contemporary Survey [114.17568992164303]
  Adrial attacks and defenses in machine learning and deep neural network have been gaining significant attention. This survey provides a comprehensive overview of the recent advancements in the field of adversarial attack and defense techniques. New avenues of attack are also explored, including search-based, decision-based, drop-based, and physical-world attacks.
  arXiv  Detail & Related papers  (2023-03-11T04:19:31Z)
• Critical Learning Periods for Multisensory Integration in Deep Networks [112.40005682521638]
  We show that the ability of a neural network to integrate information from diverse sources hinges critically on being exposed to properly correlated signals during the early phases of training. We show that critical periods arise from the complex and unstable early transient dynamics, which are decisive of final performance of the trained system and their learned representations.
  arXiv  Detail & Related papers  (2022-10-06T23:50:38Z)
• Learning Dynamics and Generalization in Reinforcement Learning [59.530058000689884]
  We show theoretically that temporal difference learning encourages agents to fit non-smooth components of the value function early in training. We show that neural networks trained using temporal difference algorithms on dense reward tasks exhibit weaker generalization between states than randomly networks and gradient networks trained with policy methods.
  arXiv  Detail & Related papers  (2022-06-05T08:49:16Z)
• Enhancing Adversarial Training with Feature Separability [52.39305978984573]
  We introduce a new concept of adversarial training graph (ATG) with which the proposed adversarial training with feature separability (ATFS) enables to boost the intra-class feature similarity and increase inter-class feature variance. Through comprehensive experiments, we demonstrate that the proposed ATFS framework significantly improves both clean and robust performance.
  arXiv  Detail & Related papers  (2022-05-02T04:04:23Z)
• Robustness through Cognitive Dissociation Mitigation in Contrastive Adversarial Training [2.538209532048867]
  We introduce a novel neural network training framework that increases model's adversarial robustness to adversarial attacks. We propose to improve model robustness to adversarial attacks by learning feature representations consistent under both data augmentations and adversarial perturbations. We validate our method on the CIFAR-10 dataset on which it outperforms both robust accuracy and clean accuracy over alternative supervised and self-supervised adversarial learning methods.
  arXiv  Detail & Related papers  (2022-03-16T21:41:27Z)
• Unsupervised Transfer Learning for Spatiotemporal Predictive Networks [90.67309545798224]
  We study how to transfer knowledge from a zoo of unsupervisedly learned models towards another network. Our motivation is that models are expected to understand complex dynamics from different sources. Our approach yields significant improvements on three benchmarks fortemporal prediction, and benefits the target even from less relevant ones.
  arXiv  Detail & Related papers  (2020-09-24T15:40:55Z)
• Exploiting the Full Capacity of Deep Neural Networks while Avoiding Overfitting by Targeted Sparsity Regularization [1.3764085113103217]
  Overfitting is one of the most common problems when training deep neural networks on comparatively small datasets. We propose novel targeted sparsity visualization and regularization strategies to counteract overfitting.
  arXiv  Detail & Related papers  (2020-02-21T11:38:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.