Fast and reliable uncertainty quantification with neural network ensembles for industrial image classification

• URL: http://arxiv.org/abs/2403.10182v5
• Date: Mon, 13 Jan 2025 06:51:13 GMT
• Title: Fast and reliable uncertainty quantification with neural network ensembles for industrial image classification
• Authors: Arthur Thuy, Dries F. Benoit,
• Abstract summary: Image classification with neural networks (NNs) is widely used in industrial processes. NNs tend to make confident yet incorrect predictions when confronted with out-of-distribution (OOD) data. Deep ensembles, composed of multiple independent NNs, have been shown to perform strongly but are computationally expensive. This study investigates the predictive and uncertainty performance of efficient NN ensembles in the context of image classification for industrial processes.
• Score: 1.104960878651584
• License:
• Abstract: Image classification with neural networks (NNs) is widely used in industrial processes, situations where the model likely encounters unknown objects during deployment, i.e., out-of-distribution (OOD) data. Worryingly, NNs tend to make confident yet incorrect predictions when confronted with OOD data. To increase the models' reliability, they should quantify the uncertainty in their own predictions, communicating when the output should (not) be trusted. Deep ensembles, composed of multiple independent NNs, have been shown to perform strongly but are computationally expensive. Recent research has proposed more efficient NN ensembles, namely the snapshot, batch, and multi-input multi-output ensemble. This study investigates the predictive and uncertainty performance of efficient NN ensembles in the context of image classification for industrial processes. It is the first to provide a comprehensive comparison and it proposes a novel Diversity Quality metric to quantify the ensembles' performance on the in-distribution and OOD sets in one single metric. The results highlight the batch ensemble as a cost-effective and competitive alternative to the deep ensemble. It matches the deep ensemble in both uncertainty and accuracy while exhibiting considerable savings in training time, test time, and memory storage.

Related papers

• CreINNs: Credal-Set Interval Neural Networks for Uncertainty Estimation in Classification Tasks [4.904199965391026]
  This work presents a novel approach, termed Credal-Set Interval Neural Networks (CreINNs) for classification. CreINNs are designed to predict an upper and a lower probability bound for each class, rather than a single probability value. Experiments on standard multiclass and binary classification tasks demonstrate that the proposed CreINNs can achieve superior or comparable quality of uncertainty estimation.
  arXiv  Detail & Related papers  (2024-01-10T10:04:49Z)
• Deep Neural Networks Tend To Extrapolate Predictably [51.303814412294514]
  neural network predictions tend to be unpredictable and overconfident when faced with out-of-distribution (OOD) inputs. We observe that neural network predictions often tend towards a constant value as input data becomes increasingly OOD. We show how one can leverage our insights in practice to enable risk-sensitive decision-making in the presence of OOD inputs.
  arXiv  Detail & Related papers  (2023-10-02T03:25:32Z)
• Improving Uncertainty Quantification of Variance Networks by Tree-Structured Learning [10.566352737844369]
  We propose a novel tree-structured local neural network model that partitions the feature space into multiple regions based on uncertainty heterogeneity. The proposed Uncertainty-Splitting Neural Regression Tree (USNRT) employs novel splitting criteria. USNRT or its ensemble shows superior performance compared to some recent popular methods for quantifying uncertainty with variances.
  arXiv  Detail & Related papers  (2022-12-24T05:25:09Z)
• Batch-Ensemble Stochastic Neural Networks for Out-of-Distribution Detection [55.028065567756066]
  Out-of-distribution (OOD) detection has recently received much attention from the machine learning community due to its importance in deploying machine learning models in real-world applications. In this paper we propose an uncertainty quantification approach by modelling the distribution of features. We incorporate an efficient ensemble mechanism, namely batch-ensemble, to construct the batch-ensemble neural networks (BE-SNNs) and overcome the feature collapse problem. We show that BE-SNNs yield superior performance on several OOD benchmarks, such as the Two-Moons dataset, the FashionMNIST vs MNIST dataset, FashionM
  arXiv  Detail & Related papers  (2022-06-26T16:00:22Z)
• Comparative Analysis of Interval Reachability for Robust Implicit and Feedforward Neural Networks [64.23331120621118]
  We use interval reachability analysis to obtain robustness guarantees for implicit neural networks (INNs) INNs are a class of implicit learning models that use implicit equations as layers. We show that our approach performs at least as well as, and generally better than, applying state-of-the-art interval bound propagation methods to INNs.
  arXiv  Detail & Related papers  (2022-04-01T03:31:27Z)
• Multidimensional Uncertainty-Aware Evidential Neural Networks [21.716045815385268]
  We propose a novel uncertainty-aware evidential NN called WGAN-ENN (WENN) for solving an out-of-versa (OOD) detection problem. We took a hybrid approach that combines Wasserstein Generative Adrial Network (WGAN) with ENNs to jointly train a model with prior knowledge of a certain class. We demonstrated that the estimation of uncertainty by WENN can significantly help distinguish OOD samples from boundary samples.
  arXiv  Detail & Related papers  (2020-12-26T04:28:56Z)
• Learn what you can't learn: Regularized Ensembles for Transductive Out-of-distribution Detection [76.39067237772286]
  We show that current out-of-distribution (OOD) detection algorithms for neural networks produce unsatisfactory results in a variety of OOD detection scenarios. This paper studies how such "hard" OOD scenarios can benefit from adjusting the detection method after observing a batch of the test data. We propose a novel method that uses an artificial labeling scheme for the test data and regularization to obtain ensembles of models that produce contradictory predictions only on the OOD samples in a test batch.
  arXiv  Detail & Related papers  (2020-12-10T16:55:13Z)
• Probing Predictions on OOD Images via Nearest Categories [97.055916832257]
  We study out-of-distribution (OOD) prediction behavior of neural networks when they classify images from unseen classes or corrupted images. We introduce a new measure, nearest category generalization (NCG), where we compute the fraction of OOD inputs that are classified with the same label as their nearest neighbor in the training set. We find that robust networks have consistently higher NCG accuracy than natural training, even when the OOD data is much farther away than the robustness radius.
  arXiv  Detail & Related papers  (2020-11-17T07:42:27Z)
• Multi-Loss Sub-Ensembles for Accurate Classification with Uncertainty Estimation [1.2891210250935146]
  We propose an efficient method for uncertainty estimation in deep neural networks (DNNs) achieving high accuracy. We keep our inference time relatively low by leveraging the advantage proposed by the Deep-Sub-Ensembles method. Our results show improved accuracy on the classification task and competitive results on several uncertainty measures.
  arXiv  Detail & Related papers  (2020-10-05T10:59:11Z)
• Diversity inducing Information Bottleneck in Model Ensembles [73.80615604822435]
  In this paper, we target the problem of generating effective ensembles of neural networks by encouraging diversity in prediction. We explicitly optimize a diversity inducing adversarial loss for learning latent variables and thereby obtain diversity in the output predictions necessary for modeling multi-modal data. Compared to the most competitive baselines, we show significant improvements in classification accuracy, under a shift in the data distribution.
  arXiv  Detail & Related papers  (2020-03-10T03:10:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.