Investigating Calibration and Corruption Robustness of Post-hoc Pruned Perception CNNs: An Image Classification Benchmark Study

• URL: http://arxiv.org/abs/2405.20876v1
• Date: Fri, 31 May 2024 14:52:49 GMT
• Title: Investigating Calibration and Corruption Robustness of Post-hoc Pruned Perception CNNs: An Image Classification Benchmark Study
• Authors: Pallavi Mitra, Gesina Schwalbe, Nadja Klein,
• Abstract summary: Convolutional Neural Networks (CNNs) have achieved state-of-the-art performance in many computer vision tasks. High computational and storage demands hinder their deployment into resource-constrained environments. Model pruning helps to meet these restrictions by reducing the model size, while maintaining superior performance.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Convolutional Neural Networks (CNNs) have achieved state-of-the-art performance in many computer vision tasks. However, high computational and storage demands hinder their deployment into resource-constrained environments, such as embedded devices. Model pruning helps to meet these restrictions by reducing the model size, while maintaining superior performance. Meanwhile, safety-critical applications pose more than just resource and performance constraints. In particular, predictions must not be overly confident, i.e., provide properly calibrated uncertainty estimations (proper uncertainty calibration), and CNNs must be robust against corruptions like naturally occurring input perturbations (natural corruption robustness). This work investigates the important trade-off between uncertainty calibration, natural corruption robustness, and performance for current state-of-research post-hoc CNN pruning techniques in the context of image classification tasks. Our study reveals that post-hoc pruning substantially improves the model's uncertainty calibration, performance, and natural corruption robustness, sparking hope for safe and robust embedded CNNs.Furthermore, uncertainty calibration and natural corruption robustness are not mutually exclusive targets under pruning, as evidenced by the improved safety aspects obtained by post-hoc unstructured pruning with increasing compression.

Related papers

• CLUE: Neural Networks Calibration via Learning Uncertainty-Error alignment [7.702016079410588]
  We introduce CLUE (Calibration via Learning Uncertainty-Error Alignment), a novel approach that aligns predicted uncertainty with observed error during training.<n>We show that CLUE achieves superior calibration quality and competitive predictive performance with respect to state-of-the-art approaches.
  arXiv  Detail & Related papers  (2025-05-28T19:23:47Z)
• Benchmarking the Spatial Robustness of DNNs via Natural and Adversarial Localized Corruptions [49.546479320670464]
  This paper introduces specialized metrics for benchmarking the spatial robustness of segmentation models. We propose region-aware multi-attack adversarial analysis, a method that enables a deeper understanding of model robustness. The results reveal that models respond to these two types of threats differently.
  arXiv  Detail & Related papers  (2025-04-02T11:37:39Z)
• OCCUQ: Exploring Efficient Uncertainty Quantification for 3D Occupancy Prediction [9.742801351723482]
  Current methods often overlook uncertainties arising from adversarial conditions or distributional shifts. We propose an efficient adaptation of an uncertainty estimation technique for 3D occupancy prediction. Our approach consistently demonstrates reliable uncertainty measures, indicating its potential for enhancing the robustness of autonomous driving systems.
  arXiv  Detail & Related papers  (2025-03-13T17:50:07Z)
• Enhancing Reliability of Neural Networks at the Edge: Inverted Normalization with Stochastic Affine Transformations [0.22499166814992438]
  We propose a method to inherently enhance the robustness and inference accuracy of BayNNs deployed in in-memory computing architectures. Empirical results show a graceful degradation in inference accuracy, with an improvement of up to $58.11%$.
  arXiv  Detail & Related papers  (2024-01-23T00:27:31Z)
• Dynamic Batch Norm Statistics Update for Natural Robustness [5.366500153474747]
  We propose a unified framework consisting of a corruption-detection model and BN statistics update. Our results demonstrate about 8% and 4% accuracy improvement on CIFAR10-C and ImageNet-C.
  arXiv  Detail & Related papers  (2023-10-31T17:20:30Z)
• Investigating the Corruption Robustness of Image Classifiers with Random Lp-norm Corruptions [3.1337872355726084]
  This study investigates the use of random p-norm corruptions to augment the training and test data of image classifiers. We find that training data augmentation with a combination of p-norm corruptions significantly improves corruption robustness, even on top of state-of-the-art data augmentation schemes.
  arXiv  Detail & Related papers  (2023-05-09T12:45:43Z)
• Bridging Precision and Confidence: A Train-Time Loss for Calibrating Object Detection [58.789823426981044]
  We propose a novel auxiliary loss formulation that aims to align the class confidence of bounding boxes with the accurateness of predictions. Our results reveal that our train-time loss surpasses strong calibration baselines in reducing calibration error for both in and out-domain scenarios.
  arXiv  Detail & Related papers  (2023-03-25T08:56:21Z)
• Can pruning improve certified robustness of neural networks? [106.03070538582222]
  We show that neural network pruning can improve empirical robustness of deep neural networks (NNs) Our experiments show that by appropriately pruning an NN, its certified accuracy can be boosted up to 8.2% under standard training. We additionally observe the existence of certified lottery tickets that can match both standard and certified robust accuracies of the original dense models.
  arXiv  Detail & Related papers  (2022-06-15T05:48:51Z)
• Benchmarking the Robustness of Spatial-Temporal Models Against Corruptions [32.821121530785504]
  We establish a corruption robustness benchmark, Mini Kinetics-C and Mini SSV2-C, which considers temporal corruptions beyond spatial corruptions in images. We make the first attempt to conduct an exhaustive study on the corruption robustness of established CNN-based and Transformer-based spatial-temporal models.
  arXiv  Detail & Related papers  (2021-10-13T05:59:39Z)
• Improving robustness against common corruptions with frequency biased models [112.65717928060195]
  unseen image corruptions can cause a surprisingly large drop in performance. Image corruption types have different characteristics in the frequency spectrum and would benefit from a targeted type of data augmentation. We propose a new regularization scheme that minimizes the total variation (TV) of convolution feature-maps to increase high-frequency robustness.
  arXiv  Detail & Related papers  (2021-03-30T10:44:50Z)
• Approaching Neural Network Uncertainty Realism [53.308409014122816]
  Quantifying or at least upper-bounding uncertainties is vital for safety-critical systems such as autonomous vehicles. We evaluate uncertainty realism -- a strict quality criterion -- with a Mahalanobis distance-based statistical test. We adopt it to the automotive domain and show that it significantly improves uncertainty realism compared to a plain encoder-decoder model.
  arXiv  Detail & Related papers  (2021-01-08T11:56:12Z)
• Attribute-Guided Adversarial Training for Robustness to Natural Perturbations [64.35805267250682]
  We propose an adversarial training approach which learns to generate new samples so as to maximize exposure of the classifier to the attributes-space. Our approach enables deep neural networks to be robust against a wide range of naturally occurring perturbations.
  arXiv  Detail & Related papers  (2020-12-03T10:17:30Z)
• A Simple Framework to Quantify Different Types of Uncertainty in Deep Neural Networks for Image Classification [0.0]
  Quantifying uncertainty in a model's predictions is important as it enables the safety of an AI system to be increased. This is crucial for applications where the cost of an error is high, such as in autonomous vehicle control, medical image analysis, financial estimations or legal fields. We propose a complete framework to capture and quantify three known types of uncertainty in Deep Neural Networks for the task of image classification.
  arXiv  Detail & Related papers  (2020-11-17T15:36:42Z)
• Adversarial Robustness on In- and Out-Distribution Improves Explainability [109.68938066821246]
  RATIO is a training procedure for robustness via Adversarial Training on In- and Out-distribution. RATIO achieves state-of-the-art $l$-adrial on CIFAR10 and maintains better clean accuracy.
  arXiv  Detail & Related papers  (2020-03-20T18:57:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.