Sparsity Meets Robustness: Channel Pruning for the Feynman-Kac Formalism Principled Robust Deep Neural Nets

• URL: http://arxiv.org/abs/2003.00631v1
• Date: Mon, 2 Mar 2020 02:18:43 GMT
• Title: Sparsity Meets Robustness: Channel Pruning for the Feynman-Kac Formalism Principled Robust Deep Neural Nets
• Authors: Thu Dinh, Bao Wang, Andrea L. Bertozzi, and Stanley J. Osher
• Abstract summary: This paper focuses on a co-design of efficient compression algorithms and sparse neural architectures for robust and accurate deep learning. We leverage the relaxed augmented Lagrangian based algorithms to prune the weights of adversarially trained DNNs. Using a Feynman-Kac formalism principled robust and sparse DNNs, we can at least double the channel sparsity of the adversarially trained ResNet20 for CIFAR10 classification.
• Score: 13.102014808597264
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep neural nets (DNNs) compression is crucial for adaptation to mobile devices. Though many successful algorithms exist to compress naturally trained DNNs, developing efficient and stable compression algorithms for robustly trained DNNs remains widely open. In this paper, we focus on a co-design of efficient DNN compression algorithms and sparse neural architectures for robust and accurate deep learning. Such a co-design enables us to advance the goal of accommodating both sparsity and robustness. With this objective in mind, we leverage the relaxed augmented Lagrangian based algorithms to prune the weights of adversarially trained DNNs, at both structured and unstructured levels. Using a Feynman-Kac formalism principled robust and sparse DNNs, we can at least double the channel sparsity of the adversarially trained ResNet20 for CIFAR10 classification, meanwhile, improve the natural accuracy by $8.69$\% and the robust accuracy under the benchmark $20$ iterations of IFGSM attack by $5.42$\%. The code is available at \url{https://github.com/BaoWangMath/rvsm-rgsm-admm}.

Related papers

• RSC-SNN: Exploring the Trade-off Between Adversarial Robustness and Accuracy in Spiking Neural Networks via Randomized Smoothing Coding [17.342181435229573]
  Spiking Neural Networks (SNNs) have received widespread attention due to their unique neuronal dynamics and low-power nature. Previous research empirically shows that SNNs with Poisson coding are more robust than Artificial Neural Networks (ANNs) on small-scale datasets. This work theoretically demonstrates that SNN's inherent adversarial robustness stems from its Poisson coding.
  arXiv  Detail & Related papers  (2024-07-29T15:26:15Z)
• DepthShrinker: A New Compression Paradigm Towards Boosting Real-Hardware Efficiency of Compact Neural Networks [29.46621102184345]
  We propose a framework dubbed DepthShrinker to develop hardware-friendly compact networks. Our framework delivers hardware-friendly compact networks that outperform both state-of-the-art efficient DNNs and compression techniques.
  arXiv  Detail & Related papers  (2022-06-02T02:32:47Z)
• Comparative Analysis of Interval Reachability for Robust Implicit and Feedforward Neural Networks [64.23331120621118]
  We use interval reachability analysis to obtain robustness guarantees for implicit neural networks (INNs) INNs are a class of implicit learning models that use implicit equations as layers. We show that our approach performs at least as well as, and generally better than, applying state-of-the-art interval bound propagation methods to INNs.
  arXiv  Detail & Related papers  (2022-04-01T03:31:27Z)
• Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networks [98.21130211336964]
  Deep neural networks (DNNs) are known to be vulnerable to adversarial attacks. In this paper, we investigate the impact of network width and depth on the robustness of adversarially trained DNNs.
  arXiv  Detail & Related papers  (2021-10-07T23:13:33Z)
• Quantized Neural Networks via {-1, +1} Encoding Decomposition and Acceleration [83.84684675841167]
  We propose a novel encoding scheme using -1, +1 to decompose quantized neural networks (QNNs) into multi-branch binary networks. We validate the effectiveness of our method on large-scale image classification, object detection, and semantic segmentation tasks.
  arXiv  Detail & Related papers  (2021-06-18T03:11:15Z)
• Training Graph Neural Networks with 1000 Layers [133.84813995275988]
  We study reversible connections, group convolutions, weight tying, and equilibrium models to advance the memory and parameter efficiency of GNNs. To the best of our knowledge, RevGNN-Deep is the deepest GNN in the literature by one order of magnitude.
  arXiv  Detail & Related papers  (2021-06-14T15:03:00Z)
• Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch [75.69506249886622]
  Sparsity in Deep Neural Networks (DNNs) has been widely studied to compress and accelerate the models on resource-constrained environments. In this paper, we are the first to study training from scratch an N:M fine-grained structured sparse network.
  arXiv  Detail & Related papers  (2021-02-08T05:55:47Z)
• Auto Graph Encoder-Decoder for Neural Network Pruning [0.8164433158925593]
  We propose an automatic graph encoder-decoder model compression (AGMC) method combined with graph neural networks (GNN) and reinforcement learning (RL) Results show that our learning-based DNN embedding achieves better performance and a higher compression ratio with fewer search steps.
  arXiv  Detail & Related papers  (2020-11-25T11:05:21Z)
• TaxoNN: A Light-Weight Accelerator for Deep Neural Network Training [2.5025363034899732]
  We present a novel approach to add the training ability to a baseline DNN accelerator (inference only) by splitting the SGD algorithm into simple computational elements. Based on this approach we propose TaxoNN, a light-weight accelerator for DNN training. Our experimental results show that TaxoNN delivers, on average, 0.97% higher misclassification rate compared to a full-precision implementation.
  arXiv  Detail & Related papers  (2020-10-11T09:04:19Z)
• An Integrated Approach to Produce Robust Models with High Efficiency [9.476463361600828]
  Quantization and structure simplification are promising ways to adapt Deep Neural Networks (DNNs) to mobile devices. In this work, we try to obtain both features by applying a convergent relaxation quantization algorithm, Binary-Relax (BR), to a robust adversarial-trained model, ResNets Ensemble. We design a trade-off loss function that helps DNNs preserve their natural accuracy and improve the channel sparsity.
  arXiv  Detail & Related papers  (2020-08-31T00:44:59Z)
• Approximation and Non-parametric Estimation of ResNet-type Convolutional Neural Networks [52.972605601174955]
  We show a ResNet-type CNN can attain the minimax optimal error rates in important function classes. We derive approximation and estimation error rates of the aformentioned type of CNNs for the Barron and H"older classes.
  arXiv  Detail & Related papers  (2019-03-24T19:42:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.