Robustness to distribution shifts of compressed networks for edge devices

• URL: http://arxiv.org/abs/2401.12014v1
• Date: Mon, 22 Jan 2024 15:00:32 GMT
• Title: Robustness to distribution shifts of compressed networks for edge devices
• Authors: Lulan Shen, Ali Edalati, Brett Meyer, Warren Gross, James J. Clark
• Abstract summary: It is important to investigate the robustness of compressed networks in two types of data distribution shifts: domain shifts and adversarial perturbations. In this study, we discover that compressed models are less robust to distribution shifts than their original networks. compact networks obtained by knowledge distillation are much more robust to distribution shifts than pruned networks.
• Score: 6.606005367624169
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: It is necessary to develop efficient DNNs deployed on edge devices with limited computation resources. However, the compressed networks often execute new tasks in the target domain, which is different from the source domain where the original network is trained. It is important to investigate the robustness of compressed networks in two types of data distribution shifts: domain shifts and adversarial perturbations. In this study, we discover that compressed models are less robust to distribution shifts than their original networks. Interestingly, larger networks are more vulnerable to losing robustness than smaller ones, even when they are compressed to a similar size as the smaller networks. Furthermore, compact networks obtained by knowledge distillation are much more robust to distribution shifts than pruned networks. Finally, post-training quantization is a reliable method for achieving significant robustness to distribution shifts, and it outperforms both pruned and distilled models in terms of robustness.

Related papers

• DenseShift: Towards Accurate and Efficient Low-Bit Power-of-Two Quantization [27.231327287238102]
  We propose the DenseShift network, which significantly improves the accuracy of Shift networks. Our experiments on various computer vision and speech tasks demonstrate that DenseShift outperforms existing low-bit multiplication-free networks.
  arXiv  Detail & Related papers  (2022-08-20T15:17:40Z)
• Understanding the effect of sparsity on neural networks robustness [32.15505923976003]
  This paper examines the impact of static sparsity on the robustness of a trained network to weight perturbations, data corruption, and adversarial examples. We show that, up to a certain sparsity achieved by increasing network width and depth while keeping the network capacity fixed, sparsified networks consistently match and often outperform their initially dense versions.
  arXiv  Detail & Related papers  (2022-06-22T08:51:40Z)
• Fast Conditional Network Compression Using Bayesian HyperNetworks [54.06346724244786]
  We introduce a conditional compression problem and propose a fast framework for tackling it. The problem is how to quickly compress a pretrained large neural network into optimal smaller networks given target contexts. Our methods can quickly generate compressed networks with significantly smaller sizes than baseline methods.
  arXiv  Detail & Related papers  (2022-05-13T00:28:35Z)
• Heavy Tails in SGD and Compressibility of Overparametrized Neural Networks [9.554646174100123]
  We show that the dynamics of the gradient descent training algorithm has a key role in obtaining compressible networks. We prove that the networks are guaranteed to be '$ell_p$-compressible', and the compression errors of different pruning techniques become arbitrarily small as the network size increases.
  arXiv  Detail & Related papers  (2021-06-07T17:02:59Z)
• Attribution Preservation in Network Compression for Reliable Network Interpretation [81.84564694303397]
  Neural networks embedded in safety-sensitive applications rely on input attribution for hindsight analysis and network compression to reduce its size for edge-computing. We show that these seemingly unrelated techniques conflict with each other as network compression deforms the produced attributions. This phenomenon arises due to the fact that conventional network compression methods only preserve the predictions of the network while ignoring the quality of the attributions.
  arXiv  Detail & Related papers  (2020-10-28T16:02:31Z)
• ShiftAddNet: A Hardware-Inspired Deep Network [87.18216601210763]
  ShiftAddNet is an energy-efficient multiplication-less deep neural network. It leads to both energy-efficient inference and training, without compromising expressive capacity. ShiftAddNet aggressively reduces over 80% hardware-quantified energy cost of DNNs training and inference, while offering comparable or better accuracies.
  arXiv  Detail & Related papers  (2020-10-24T05:09:14Z)
• Towards Accurate Quantization and Pruning via Data-free Knowledge Transfer [61.85316480370141]
  We study data-free quantization and pruning by transferring knowledge from trained large networks to compact networks. Our data-free compact networks achieve competitive accuracy to networks trained and fine-tuned with training data.
  arXiv  Detail & Related papers  (2020-10-14T18:02:55Z)
• Achieving Adversarial Robustness via Sparsity [33.11581532788394]
  We prove that the sparsity of network weights is closely associated with model robustness. We propose a novel adversarial training method called inverse weights inheritance.
  arXiv  Detail & Related papers  (2020-09-11T13:15:43Z)
• Resolution Adaptive Networks for Efficient Inference [53.04907454606711]
  We propose a novel Resolution Adaptive Network (RANet), which is inspired by the intuition that low-resolution representations are sufficient for classifying "easy" inputs. In RANet, the input images are first routed to a lightweight sub-network that efficiently extracts low-resolution representations. High-resolution paths in the network maintain the capability to recognize the "hard" samples.
  arXiv  Detail & Related papers  (2020-03-16T16:54:36Z)
• ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions [76.05981545084738]
  We propose several ideas for enhancing a binary network to close its accuracy gap from real-valued networks without incurring any additional computational cost. We first construct a baseline network by modifying and binarizing a compact real-valued network with parameter-free shortcuts. We show that the proposed ReActNet outperforms all the state-of-the-arts by a large margin.
  arXiv  Detail & Related papers  (2020-03-07T02:12:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.