Beyond Pruning Criteria: The Dominant Role of Fine-Tuning and Adaptive Ratios in Neural Network Robustness

• URL: http://arxiv.org/abs/2410.15176v1
• Date: Sat, 19 Oct 2024 18:35:52 GMT
• Title: Beyond Pruning Criteria: The Dominant Role of Fine-Tuning and Adaptive Ratios in Neural Network Robustness
• Authors: Lincen Bai, Hedi Tabia, Raúl Santos-Rodríguez,
• Abstract summary: Deep neural networks (DNNs) excel in tasks like image recognition and natural language processing. Traditional pruning methods compromise the network's ability to withstand subtle perturbations. This paper challenges the conventional emphasis on weight importance scoring as the primary determinant of a pruned network's performance.
• Score: 7.742297876120561
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Deep neural networks (DNNs) excel in tasks like image recognition and natural language processing, but their increasing complexity complicates deployment in resource-constrained environments and increases susceptibility to adversarial attacks. While traditional pruning methods reduce model size, they often compromise the network's ability to withstand subtle perturbations. This paper challenges the conventional emphasis on weight importance scoring as the primary determinant of a pruned network's performance. Through extensive analysis, including experiments conducted on CIFAR, Tiny-ImageNet, and various network architectures, we demonstrate that effective fine-tuning plays a dominant role in enhancing both performance and adversarial robustness, often surpassing the impact of the chosen pruning criteria. To address this issue, we introduce Module Robust Sensitivity, a novel metric that adaptively adjusts the pruning ratio for each network layer based on its sensitivity to adversarial perturbations. By integrating this metric into the pruning process, we develop a stable algorithm that maintains accuracy and robustness simultaneously. Experimental results show that our approach enables the practical deployment of more robust and efficient neural networks.

Related papers

• Adaptive Dual-Weighting Framework for Federated Learning via Out-of-Distribution Detection [53.45696787935487]
  Federated Learning (FL) enables collaborative model training across large-scale distributed service nodes.<n>In real-world service-oriented deployments, data generated by heterogeneous users, devices, and application scenarios are inherently non-IID.<n>We propose FLood, a novel FL framework inspired by out-of-distribution (OOD) detection.
  arXiv  Detail & Related papers  (2026-02-01T05:54:59Z)
• Towards Compact and Robust DNNs via Compression-aware Sharpness Minimization [7.641622965415444]
  Compression-aware ShArpness Minimization (C-SAM) is a framework that shifts sharpness-aware learning from parameter perturbations to mask perturbations.<n>C-SAM consistently achieves higher certified robustness than strong baselines, with improvements of up to 42%, while maintaining task accuracy comparable to the corresponding unpruned models.
  arXiv  Detail & Related papers  (2026-01-28T06:49:32Z)
• Robust-R1: Degradation-Aware Reasoning for Robust Visual Understanding [54.05243949024302]
  Existing robust MLLMs rely on implicit training/adaptation that focuses solely on visual encoder generalization.<n>We propose Robust-R1, a novel framework that explicitly models visual degradations through structured reasoning chains.<n>Our approach integrates: (i) supervised fine-tuning for degradation-aware reasoning foundations, (ii) reward-driven alignment for accurately perceiving degradation parameters, and (iii) dynamic reasoning depth scaling adapted to degradation intensity.
  arXiv  Detail & Related papers  (2025-12-19T12:56:17Z)
• Bridging Symmetry and Robustness: On the Role of Equivariance in Enhancing Adversarial Robustness [9.013874391203453]
  Adversarial examples reveal critical vulnerabilities in deep neural networks by exploiting their sensitivity to imperceptible input perturbations.<n>In this work, we investigate an architectural approach to adversarial robustness by embedding group-equivariant convolutions.<n>These layers encode symmetry priors that align model behavior with structured transformations in the input space, promoting smoother decision boundaries.
  arXiv  Detail & Related papers  (2025-10-17T19:26:58Z)
• Sycophancy Mitigation Through Reinforcement Learning with Uncertainty-Aware Adaptive Reasoning Trajectories [58.988535279557546]
  We introduce textbf sycophancy Mitigation through Adaptive Reasoning Trajectories.<n>We show that SMART significantly reduces sycophantic behavior while preserving strong performance on out-of-distribution inputs.
  arXiv  Detail & Related papers  (2025-09-20T17:09:14Z)
• COMponent-Aware Pruning for Accelerated Control Tasks in Latent Space Models [1.6874375111244326]
  The rapid growth of resource-constrained mobile platforms has increased the demand for computationally efficient neural network controllers (NNCs)<n>Deep neural networks (DNNs) demonstrate superior performance in control applications, their substantial computational complexity and memory requirements present significant barriers to practical deployment on edge devices.<n>This paper introduces a comprehensive model compression methodology that leverages component-aware structured pruning to determine the optimal pruning magnitude for each pruning group.
  arXiv  Detail & Related papers  (2025-08-11T16:16:51Z)
• LyAm: Robust Non-Convex Optimization for Stable Learning in Noisy Environments [0.0]
  Training deep neural networks, particularly in computer vision tasks, often suffers from noisy gradients.<n>We propose LyAm, a novel that integrates Adam's adaptive moment estimation with Lyapunov-based stability mechanisms.<n>LyAm consistently outperforms state-of-the-art settings in terms of accuracy, convergence, speed, and stability.
  arXiv  Detail & Related papers  (2025-07-15T12:35:13Z)
• Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approach [14.639659415276533]
  Mixture of Experts (MoE) have shown remarkable success in leveraging specialized expert networks for complex machine learning tasks. Their susceptibility to adversarial attacks presents a critical challenge for deployment in robust applications. This paper addresses the question of how to incorporate robustness into MoEs while maintaining high natural accuracy.
  arXiv  Detail & Related papers  (2025-02-05T20:45:52Z)
• Adversarial Robustness through Dynamic Ensemble Learning [0.0]
  Adversarial attacks pose a significant threat to the reliability of pre-trained language models (PLMs) This paper presents Adversarial Robustness through Dynamic Ensemble Learning (ARDEL), a novel scheme designed to enhance the robustness of PLMs against such attacks.
  arXiv  Detail & Related papers  (2024-12-20T05:36:19Z)
• A Tunable Despeckling Neural Network Stabilized via Diffusion Equation [15.996302571895045]
  Multiplicative Gamma noise remove is a critical research area in the application of synthetic aperture radar (SAR) imaging. We propose a tunable, regularized neural network that unrolls a denoising unit and a regularization unit into a single network for end-to-end training.
  arXiv  Detail & Related papers  (2024-11-24T17:08:43Z)
• CCSRP: Robust Pruning of Spiking Neural Networks through Cooperative Coevolution [2.5388345537743056]
  Spiking neural networks (SNNs) have shown promise in various dynamic visual tasks, yet those ready for practical deployment often lack the compactness and robustness essential in resource-limited and safety-critical settings. We propose CCSRP, an innovative robust pruning method for SNNs, underpinned by cooperative co-evolution.
  arXiv  Detail & Related papers  (2024-07-18T04:28:16Z)
• Data-Driven Lipschitz Continuity: A Cost-Effective Approach to Improve Adversarial Robustness [47.9744734181236]
  We explore the concept of Lipschitz continuity to certify the robustness of deep neural networks (DNNs) against adversarial attacks. We propose a novel algorithm that remaps the input domain into a constrained range, reducing the Lipschitz constant and potentially enhancing robustness. Our method achieves the best robust accuracy for CIFAR10, CIFAR100, and ImageNet datasets on the RobustBench leaderboard.
  arXiv  Detail & Related papers  (2024-06-28T03:10:36Z)
• Enhancing Reliability of Neural Networks at the Edge: Inverted Normalization with Stochastic Affine Transformations [0.22499166814992438]
  We propose a method to inherently enhance the robustness and inference accuracy of BayNNs deployed in in-memory computing architectures. Empirical results show a graceful degradation in inference accuracy, with an improvement of up to $58.11%$.
  arXiv  Detail & Related papers  (2024-01-23T00:27:31Z)
• Towards Improving Robustness Against Common Corruptions using Mixture of Class Specific Experts [10.27974860479791]
  This paper introduces a novel paradigm known as the Mixture of Class-Specific Expert Architecture. The proposed architecture aims to mitigate vulnerabilities associated with common neural network structures.
  arXiv  Detail & Related papers  (2023-11-16T20:09:47Z)
• Improving Generalization of Adversarial Training via Robust Critical Fine-Tuning [19.91117174405902]
  Deep neural networks are susceptible to adversarial examples, posing a significant security risk in critical applications. This paper proposes Robustness Critical FineTuning (RiFT), a novel approach to enhance generalization without compromising adversarial robustness.
  arXiv  Detail & Related papers  (2023-08-01T09:02:34Z)
• A Generic Shared Attention Mechanism for Various Backbone Neural Networks [53.36677373145012]
  Self-attention modules (SAMs) produce strongly correlated attention maps across different layers. Dense-and-Implicit Attention (DIA) shares SAMs across layers and employs a long short-term memory module. Our simple yet effective DIA can consistently enhance various network backbones.
  arXiv  Detail & Related papers  (2022-10-27T13:24:08Z)
• Comparative Analysis of Interval Reachability for Robust Implicit and Feedforward Neural Networks [64.23331120621118]
  We use interval reachability analysis to obtain robustness guarantees for implicit neural networks (INNs) INNs are a class of implicit learning models that use implicit equations as layers. We show that our approach performs at least as well as, and generally better than, applying state-of-the-art interval bound propagation methods to INNs.
  arXiv  Detail & Related papers  (2022-04-01T03:31:27Z)
• Robustness and Accuracy Could Be Reconcilable by (Proper) Definition [109.62614226793833]
  The trade-off between robustness and accuracy has been widely studied in the adversarial literature. We find that it may stem from the improperly defined robust error, which imposes an inductive bias of local invariance. By definition, SCORE facilitates the reconciliation between robustness and accuracy, while still handling the worst-case uncertainty.
  arXiv  Detail & Related papers  (2022-02-21T10:36:09Z)
• Pruning in the Face of Adversaries [0.0]
  We evaluate the impact of neural network pruning on the adversarial robustness against L-0, L-2 and L-infinity attacks. Our results confirm that neural network pruning and adversarial robustness are not mutually exclusive. We extend our analysis to situations that incorporate additional assumptions on the adversarial scenario and show that depending on the situation, different strategies are optimal.
  arXiv  Detail & Related papers  (2021-08-19T09:06:16Z)
• Residual Error: a New Performance Measure for Adversarial Robustness [85.0371352689919]
  A major challenge that limits the wide-spread adoption of deep learning has been their fragility to adversarial attacks. This study presents the concept of residual error, a new performance measure for assessing the adversarial robustness of a deep neural network. Experimental results using the case of image classification demonstrate the effectiveness and efficacy of the proposed residual error metric.
  arXiv  Detail & Related papers  (2021-06-18T16:34:23Z)
• Efficient Micro-Structured Weight Unification and Pruning for Neural Network Compression [56.83861738731913]
  Deep Neural Network (DNN) models are essential for practical applications, especially for resource limited devices. Previous unstructured or structured weight pruning methods can hardly truly accelerate inference. We propose a generalized weight unification framework at a hardware compatible micro-structured level to achieve high amount of compression and acceleration.
  arXiv  Detail & Related papers  (2021-06-15T17:22:59Z)
• Non-Singular Adversarial Robustness of Neural Networks [58.731070632586594]
  Adrial robustness has become an emerging challenge for neural network owing to its over-sensitivity to small input perturbations. We formalize the notion of non-singular adversarial robustness for neural networks through the lens of joint perturbations to data inputs as well as model weights.
  arXiv  Detail & Related papers  (2021-02-23T20:59:30Z)
• Improve Generalization and Robustness of Neural Networks via Weight Scale Shifting Invariant Regularizations [52.493315075385325]
  We show that a family of regularizers, including weight decay, is ineffective at penalizing the intrinsic norms of weights for networks with homogeneous activation functions. We propose an improved regularizer that is invariant to weight scale shifting and thus effectively constrains the intrinsic norm of a neural network.
  arXiv  Detail & Related papers  (2020-08-07T02:55:28Z)
• A general framework for defining and optimizing robustness [74.67016173858497]
  We propose a rigorous and flexible framework for defining different types of robustness properties for classifiers. Our concept is based on postulates that robustness of a classifier should be considered as a property that is independent of accuracy. We develop a very general robustness framework that is applicable to any type of classification model.
  arXiv  Detail & Related papers  (2020-06-19T13:24:20Z)
• Network Diffusions via Neural Mean-Field Dynamics [52.091487866968286]
  We propose a novel learning framework for inference and estimation problems of diffusion on networks. Our framework is derived from the Mori-Zwanzig formalism to obtain an exact evolution of the node infection probabilities. Our approach is versatile and robust to variations of the underlying diffusion network models.
  arXiv  Detail & Related papers  (2020-06-16T18:45:20Z)
• Self-Supervised Dynamic Networks for Covariate Shift Robustness [9.542023122304098]
  Self-Supervised Dynamic Networks (SSDN) is an input-dependent mechanism that allows a self-supervised network to predict the weights of the main network. We present the conceptual and empirical advantages of the proposed method on the problem of image classification.
  arXiv  Detail & Related papers  (2020-06-06T19:37:20Z)
• Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustness [97.67477497115163]
  We use mode connectivity to study the adversarial robustness of deep neural networks. Our experiments cover various types of adversarial attacks applied to different network architectures and datasets. Our results suggest that mode connectivity offers a holistic tool and practical means for evaluating and improving adversarial robustness.
  arXiv  Detail & Related papers  (2020-04-30T19:12:50Z)
• Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inference [119.19779637025444]
  Deep networks were recently suggested to face the odds between accuracy (on clean natural images) and robustness (on adversarially perturbed images) This paper studies multi-exit networks associated with input-adaptive inference, showing their strong promise in achieving a "sweet point" in cooptimizing model accuracy, robustness and efficiency.
  arXiv  Detail & Related papers  (2020-02-24T00:40:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.