Fixing Overconfidence in Dynamic Neural Networks
- URL: http://arxiv.org/abs/2302.06359v4
- Date: Fri, 8 Dec 2023 12:56:51 GMT
- Title: Fixing Overconfidence in Dynamic Neural Networks
- Authors: Lassi Meronen, Martin Trapp, Andrea Pilzer, Le Yang, Arno Solin
- Abstract summary: We present an efficient approach for quantifying uncertainty in dynamic neural networks.
We show improvements on CIFAR-100, ImageNet, and Caltech-256 in terms of accuracy, capturing uncertainty, and calibration error.
- Score: 21.148621590039582
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Dynamic neural networks are a recent technique that promises a remedy for the
increasing size of modern deep learning models by dynamically adapting their
computational cost to the difficulty of the inputs. In this way, the model can
adjust to a limited computational budget. However, the poor quality of
uncertainty estimates in deep learning models makes it difficult to distinguish
between hard and easy samples. To address this challenge, we present a
computationally efficient approach for post-hoc uncertainty quantification in
dynamic neural networks. We show that adequately quantifying and accounting for
both aleatoric and epistemic uncertainty through a probabilistic treatment of
the last layers improves the predictive performance and aids decision-making
when determining the computational budget. In the experiments, we show
improvements on CIFAR-100, ImageNet, and Caltech-256 in terms of accuracy,
capturing uncertainty, and calibration error.
Related papers
- Randomized Forward Mode Gradient for Spiking Neural Networks in Scientific Machine Learning [4.178826560825283]
Spiking neural networks (SNNs) represent a promising approach in machine learning, combining the hierarchical learning capabilities of deep neural networks with the energy efficiency of spike-based computations.
Traditional end-to-end training of SNNs is often based on back-propagation, where weight updates are derived from gradients computed through the chain rule.
This method encounters challenges due to its limited biological plausibility and inefficiencies on neuromorphic hardware.
In this study, we introduce an alternative training approach for SNNs. Instead of using back-propagation, we leverage weight perturbation methods within a forward-mode
arXiv Detail & Related papers (2024-11-11T15:20:54Z) - Predicting Probabilities of Error to Combine Quantization and Early Exiting: QuEE [68.6018458996143]
We propose a more general dynamic network that can combine both quantization and early exit dynamic network: QuEE.
Our algorithm can be seen as a form of soft early exiting or input-dependent compression.
The crucial factor of our approach is accurate prediction of the potential accuracy improvement achievable through further computation.
arXiv Detail & Related papers (2024-06-20T15:25:13Z) - Learning Sample Difficulty from Pre-trained Models for Reliable
Prediction [55.77136037458667]
We propose to utilize large-scale pre-trained models to guide downstream model training with sample difficulty-aware entropy regularization.
We simultaneously improve accuracy and uncertainty calibration across challenging benchmarks.
arXiv Detail & Related papers (2023-04-20T07:29:23Z) - Neural Networks with Quantization Constraints [111.42313650830248]
We present a constrained learning approach to quantization training.
We show that the resulting problem is strongly dual and does away with gradient estimations.
We demonstrate that the proposed approach exhibits competitive performance in image classification tasks.
arXiv Detail & Related papers (2022-10-27T17:12:48Z) - Scalable computation of prediction intervals for neural networks via
matrix sketching [79.44177623781043]
Existing algorithms for uncertainty estimation require modifying the model architecture and training procedure.
This work proposes a new algorithm that can be applied to a given trained neural network and produces approximate prediction intervals.
arXiv Detail & Related papers (2022-05-06T13:18:31Z) - On Efficient Uncertainty Estimation for Resource-Constrained Mobile
Applications [0.0]
Predictive uncertainty supplements model predictions and enables improved functionality of downstream tasks.
We tackle this problem by building upon Monte Carlo Dropout (MCDO) models using the Axolotl framework.
We conduct experiments on (1) a multi-class classification task using the CIFAR10 dataset, and (2) a more complex human body segmentation task.
arXiv Detail & Related papers (2021-11-11T22:24:15Z) - Accurate and Reliable Forecasting using Stochastic Differential
Equations [48.21369419647511]
It is critical yet challenging for deep learning models to properly characterize uncertainty that is pervasive in real-world environments.
This paper develops SDE-HNN to characterize the interaction between the predictive mean and variance of HNNs for accurate and reliable regression.
Experiments on the challenging datasets show that our method significantly outperforms the state-of-the-art baselines in terms of both predictive performance and uncertainty quantification.
arXiv Detail & Related papers (2021-03-28T04:18:11Z) - Simple and Principled Uncertainty Estimation with Deterministic Deep
Learning via Distance Awareness [24.473250414880454]
We study principled approaches to high-quality uncertainty estimation that require only a single deep neural network (DNN)
By formalizing the uncertainty quantification as a minimax learning problem, we first identify input distance awareness, i.e., the model's ability to quantify the distance of a testing example from the training data in the input space.
We then propose Spectral-normalized Neural Gaussian Process (SNGP), a simple method that improves the distance-awareness ability of modern DNNs.
arXiv Detail & Related papers (2020-06-17T19:18:22Z) - Functional Space Variational Inference for Uncertainty Estimation in
Computer Aided Diagnosis [2.1940032945704817]
Deep neural networks have revolutionized medical image analysis and disease diagnosis.
It is difficult to generate well-calibrated probabilistic outputs for such networks, which makes them uninterpretable black boxes.
We show that by shifting Bayesian inference to the functional space we can craft meaningful priors that give better calibrated uncertainty estimates at a much lower computational cost.
arXiv Detail & Related papers (2020-05-24T16:42:11Z) - Uncertainty Estimation Using a Single Deep Deterministic Neural Network [66.26231423824089]
We propose a method for training a deterministic deep model that can find and reject out of distribution data points at test time with a single forward pass.
We scale training in these with a novel loss function and centroid updating scheme and match the accuracy of softmax models.
arXiv Detail & Related papers (2020-03-04T12:27:36Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.