On Calibration of Modern Quantized Efficient Neural Networks

• URL: http://arxiv.org/abs/2309.13866v2
• Date: Tue, 26 Sep 2023 05:33:41 GMT
• Title: On Calibration of Modern Quantized Efficient Neural Networks
• Authors: Joey Kuang, Alexander Wong
• Abstract summary: Quality of calibration is observed to track the quantization quality. GhostNet-VGG is shown to be the most robust to overall performance drop at lower precision.
• Score: 79.06893963657335
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We explore calibration properties at various precisions for three architectures: ShuffleNetv2, GhostNet-VGG, and MobileOne; and two datasets: CIFAR-100 and PathMNIST. The quality of calibration is observed to track the quantization quality; it is well-documented that performance worsens with lower precision, and we observe a similar correlation with poorer calibration. This becomes especially egregious at 4-bit activation regime. GhostNet-VGG is shown to be the most robust to overall performance drop at lower precision. We find that temperature scaling can improve calibration error for quantized networks, with some caveats. We hope that these preliminary insights can lead to more opportunities for explainable and reliable EdgeML.

Related papers

• GETS: Ensemble Temperature Scaling for Calibration in Graph Neural Networks [8.505932176266368]
  Graph Neural Networks deliver strong classification results but often suffer from poor calibration performance, leading to overconfidence or underconfidence. Existing post hoc methods, such as temperature scaling, fail to effectively utilize graph structures, while current GNN calibration methods often overlook the potential of leveraging diverse input information and model ensembles jointly. In the paper, we propose Graph Ensemble TemperatureScaling, a novel calibration framework that combines input and model ensemble strategies within a Graph Mixture of Experts archi SOTA calibration techniques, reducing expected calibration error by 25 percent across 10 GNN benchmark datasets.
  arXiv  Detail & Related papers  (2024-10-12T15:34:41Z)
• Cal-DETR: Calibrated Detection Transformer [67.75361289429013]
  We propose a mechanism for calibrated detection transformers (Cal-DETR), particularly for Deformable-DETR, UP-DETR and DINO. We develop an uncertainty-guided logit modulation mechanism that leverages the uncertainty to modulate the class logits. Results corroborate the effectiveness of Cal-DETR against the competing train-time methods in calibrating both in-domain and out-domain detections.
  arXiv  Detail & Related papers  (2023-11-06T22:13:10Z)
• Towards Unbiased Calibration using Meta-Regularization [6.440598446802981]
  We propose to learn better-calibrated models via meta-regularization, which has two components. We evaluate the effectiveness of the proposed approach in regularizing neural networks towards improved and unbiased calibration on three computer vision datasets.
  arXiv  Detail & Related papers  (2023-03-27T10:00:50Z)
• Bridging Precision and Confidence: A Train-Time Loss for Calibrating Object Detection [58.789823426981044]
  We propose a novel auxiliary loss formulation that aims to align the class confidence of bounding boxes with the accurateness of predictions. Our results reveal that our train-time loss surpasses strong calibration baselines in reducing calibration error for both in and out-domain scenarios.
  arXiv  Detail & Related papers  (2023-03-25T08:56:21Z)
• Sample-dependent Adaptive Temperature Scaling for Improved Calibration [95.7477042886242]
  Post-hoc approach to compensate for neural networks being wrong is to perform temperature scaling. We propose to predict a different temperature value for each input, allowing us to adjust the mismatch between confidence and accuracy. We test our method on the ResNet50 and WideResNet28-10 architectures using the CIFAR10/100 and Tiny-ImageNet datasets.
  arXiv  Detail & Related papers  (2022-07-13T14:13:49Z)
• Meta-Calibration: Learning of Model Calibration Using Differentiable Expected Calibration Error [46.12703434199988]
  We introduce a new differentiable surrogate for expected calibration error (DECE) that allows calibration quality to be directly optimised. We also propose a meta-learning framework that uses DECE to optimise for validation set calibration.
  arXiv  Detail & Related papers  (2021-06-17T15:47:50Z)
• On the Dark Side of Calibration for Modern Neural Networks [65.83956184145477]
  We show the breakdown of expected calibration error (ECE) into predicted confidence and refinement. We highlight that regularisation based calibration only focuses on naively reducing a model's confidence. We find that many calibration approaches with the likes of label smoothing, mixup etc. lower the utility of a DNN by degrading its refinement.
  arXiv  Detail & Related papers  (2021-06-17T11:04:14Z)
• On Calibration of Mixup Training for Deep Neural Networks [1.6242924916178283]
  We argue and provide empirical evidence that, due to its fundamentals, Mixup does not necessarily improve calibration. Our loss is inspired by Bayes decision theory and introduces a new training framework for designing losses for probabilistic modelling. We provide state-of-the-art accuracy with consistent improvements in calibration performance.
  arXiv  Detail & Related papers  (2020-03-22T16:54:31Z)
• Calibrating Deep Neural Networks using Focal Loss [77.92765139898906]
  Miscalibration is a mismatch between a model's confidence and its correctness. We show that focal loss allows us to learn models that are already very well calibrated. We show that our approach achieves state-of-the-art calibration without compromising on accuracy in almost all cases.
  arXiv  Detail & Related papers  (2020-02-21T17:35:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.