Beyond Pruning Criteria: The Dominant Role of Fine-Tuning and Adaptive Ratios in Neural Network Robustness

• URL: http://arxiv.org/abs/2410.15176v1
• Date: Sat, 19 Oct 2024 18:35:52 GMT
• Title: Beyond Pruning Criteria: The Dominant Role of Fine-Tuning and Adaptive Ratios in Neural Network Robustness
• Authors: Lincen Bai, Hedi Tabia, Raúl Santos-Rodríguez,
• Abstract summary: Deep neural networks (DNNs) excel in tasks like image recognition and natural language processing. Traditional pruning methods compromise the network's ability to withstand subtle perturbations. This paper challenges the conventional emphasis on weight importance scoring as the primary determinant of a pruned network's performance.
• Score: 7.742297876120561
• License:
• Abstract: Deep neural networks (DNNs) excel in tasks like image recognition and natural language processing, but their increasing complexity complicates deployment in resource-constrained environments and increases susceptibility to adversarial attacks. While traditional pruning methods reduce model size, they often compromise the network's ability to withstand subtle perturbations. This paper challenges the conventional emphasis on weight importance scoring as the primary determinant of a pruned network's performance. Through extensive analysis, including experiments conducted on CIFAR, Tiny-ImageNet, and various network architectures, we demonstrate that effective fine-tuning plays a dominant role in enhancing both performance and adversarial robustness, often surpassing the impact of the chosen pruning criteria. To address this issue, we introduce Module Robust Sensitivity, a novel metric that adaptively adjusts the pruning ratio for each network layer based on its sensitivity to adversarial perturbations. By integrating this metric into the pruning process, we develop a stable algorithm that maintains accuracy and robustness simultaneously. Experimental results show that our approach enables the practical deployment of more robust and efficient neural networks.

Related papers

• CCSRP: Robust Pruning of Spiking Neural Networks through Cooperative Coevolution [2.5388345537743056]
  Spiking neural networks (SNNs) have shown promise in various dynamic visual tasks, yet those ready for practical deployment often lack the compactness and robustness essential in resource-limited and safety-critical settings. We propose CCSRP, an innovative robust pruning method for SNNs, underpinned by cooperative co-evolution.
  arXiv  Detail & Related papers  (2024-07-18T04:28:16Z)
• Towards Improving Robustness Against Common Corruptions using Mixture of Class Specific Experts [10.27974860479791]
  This paper introduces a novel paradigm known as the Mixture of Class-Specific Expert Architecture. The proposed architecture aims to mitigate vulnerabilities associated with common neural network structures.
  arXiv  Detail & Related papers  (2023-11-16T20:09:47Z)
• Revisiting the Trade-off between Accuracy and Robustness via Weight Distribution of Filters [17.316537476091867]
  Adversarial attacks have been proven to be potential threats to Deep Neural Networks (DNNs) We propose a sample-wise dynamic network architecture named Adversarial Weight-Varied Network (AW-Net) AW-Net adaptively adjusts the network's weights based on regulation signals generated by an adversarial router.
  arXiv  Detail & Related papers  (2023-06-06T06:09:11Z)
• A Generic Shared Attention Mechanism for Various Backbone Neural Networks [53.36677373145012]
  Self-attention modules (SAMs) produce strongly correlated attention maps across different layers. Dense-and-Implicit Attention (DIA) shares SAMs across layers and employs a long short-term memory module. Our simple yet effective DIA can consistently enhance various network backbones.
  arXiv  Detail & Related papers  (2022-10-27T13:24:08Z)
• Comparative Analysis of Interval Reachability for Robust Implicit and Feedforward Neural Networks [64.23331120621118]
  We use interval reachability analysis to obtain robustness guarantees for implicit neural networks (INNs) INNs are a class of implicit learning models that use implicit equations as layers. We show that our approach performs at least as well as, and generally better than, applying state-of-the-art interval bound propagation methods to INNs.
  arXiv  Detail & Related papers  (2022-04-01T03:31:27Z)
• Pruning in the Face of Adversaries [0.0]
  We evaluate the impact of neural network pruning on the adversarial robustness against L-0, L-2 and L-infinity attacks. Our results confirm that neural network pruning and adversarial robustness are not mutually exclusive. We extend our analysis to situations that incorporate additional assumptions on the adversarial scenario and show that depending on the situation, different strategies are optimal.
  arXiv  Detail & Related papers  (2021-08-19T09:06:16Z)
• Residual Error: a New Performance Measure for Adversarial Robustness [85.0371352689919]
  A major challenge that limits the wide-spread adoption of deep learning has been their fragility to adversarial attacks. This study presents the concept of residual error, a new performance measure for assessing the adversarial robustness of a deep neural network. Experimental results using the case of image classification demonstrate the effectiveness and efficacy of the proposed residual error metric.
  arXiv  Detail & Related papers  (2021-06-18T16:34:23Z)
• Non-Singular Adversarial Robustness of Neural Networks [58.731070632586594]
  Adrial robustness has become an emerging challenge for neural network owing to its over-sensitivity to small input perturbations. We formalize the notion of non-singular adversarial robustness for neural networks through the lens of joint perturbations to data inputs as well as model weights.
  arXiv  Detail & Related papers  (2021-02-23T20:59:30Z)
• Improve Generalization and Robustness of Neural Networks via Weight Scale Shifting Invariant Regularizations [52.493315075385325]
  We show that a family of regularizers, including weight decay, is ineffective at penalizing the intrinsic norms of weights for networks with homogeneous activation functions. We propose an improved regularizer that is invariant to weight scale shifting and thus effectively constrains the intrinsic norm of a neural network.
  arXiv  Detail & Related papers  (2020-08-07T02:55:28Z)
• Network Diffusions via Neural Mean-Field Dynamics [52.091487866968286]
  We propose a novel learning framework for inference and estimation problems of diffusion on networks. Our framework is derived from the Mori-Zwanzig formalism to obtain an exact evolution of the node infection probabilities. Our approach is versatile and robust to variations of the underlying diffusion network models.
  arXiv  Detail & Related papers  (2020-06-16T18:45:20Z)
• Self-Supervised Dynamic Networks for Covariate Shift Robustness [9.542023122304098]
  Self-Supervised Dynamic Networks (SSDN) is an input-dependent mechanism that allows a self-supervised network to predict the weights of the main network. We present the conceptual and empirical advantages of the proposed method on the problem of image classification.
  arXiv  Detail & Related papers  (2020-06-06T19:37:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.