BayCANN: Streamlining Bayesian Calibration with Artificial Neural Network Metamodeling

• URL: http://arxiv.org/abs/2010.13452v1
• Date: Mon, 26 Oct 2020 09:47:39 GMT
• Title: BayCANN: Streamlining Bayesian Calibration with Artificial Neural Network Metamodeling
• Abstract summary: We propose to use artificial neural networks (ANN) as one solution to limitations of Bayesian calibration. We demonstrate BayCANN by calibrating a natural history model of colorectal cancer to adenoma prevalence and cancer incidence data. BayCANN was generally more accurate than IMIS in recovering the "true" values.
• Score: 77.34726150561087
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Purpose: Bayesian calibration is theoretically superior to standard direct-search algorithm because it can reveal the full joint posterior distribution of the calibrated parameters. However, to date, Bayesian calibration has not been used often in health decision sciences due to practical and computational burdens. In this paper we propose to use artificial neural networks (ANN) as one solution to these limitations. Methods: Bayesian Calibration using Artificial Neural Networks (BayCANN) involves (1) training an ANN metamodel on a sample of model inputs and outputs, and (2) then calibrating the trained ANN metamodel instead of the full model in a probabilistic programming language to obtain the posterior joint distribution of the calibrated parameters. We demonstrate BayCANN by calibrating a natural history model of colorectal cancer to adenoma prevalence and cancer incidence data. In addition, we compare the efficiency and accuracy of BayCANN against performing a Bayesian calibration directly on the simulation model using an incremental mixture importance sampling (IMIS) algorithm. Results: BayCANN was generally more accurate than IMIS in recovering the "true" parameter values. The ratio of the absolute ANN deviation from the truth compared to IMIS for eight out of the nine calibrated parameters were less than one indicating that BayCANN was more accurate than IMIS. In addition, BayCANN took about 15 minutes total compared to the IMIS method which took 80 minutes. Conclusions: In our case study, BayCANN was more accurate than IMIS and was five-folds faster. Because BayCANN does not depend on the structure of the simulation model, it can be adapted to models of various levels of complexity with minor changes to its structure. We provide BayCANN's open-source implementation in R.

Related papers

• Neural Networks with (Low-Precision) Polynomial Approximations: New Insights and Techniques for Accuracy Improvement [13.406378419824003]
  replacing non-polynomial functions with their approximations is a standard practice in privacy-preserving machine learning. The resulting neural network, called approximation of neural network (PANN), is compatible with advanced cryptosystems. We provide an explanation on the effect of approximate error in PANN. We propose solutions to increase inference accuracy for PANN.
  arXiv  Detail & Related papers  (2024-02-17T08:54:25Z)
• Bridging Precision and Confidence: A Train-Time Loss for Calibrating Object Detection [58.789823426981044]
  We propose a novel auxiliary loss formulation that aims to align the class confidence of bounding boxes with the accurateness of predictions. Our results reveal that our train-time loss surpasses strong calibration baselines in reducing calibration error for both in and out-domain scenarios.
  arXiv  Detail & Related papers  (2023-03-25T08:56:21Z)
• On the Calibration of Pre-trained Language Models using Mixup Guided by Area Under the Margin and Saliency [47.90235939359225]
  We propose a novel mixup strategy for pre-trained language models that improves model calibration further. Our method achieves the lowest expected calibration error compared to strong baselines on both in-domain and out-of-domain test samples.
  arXiv  Detail & Related papers  (2022-03-14T23:45:08Z)
• Model Architecture Adaption for Bayesian Neural Networks [9.978961706999833]
  We show a novel network architecture search (NAS) that optimize BNNs for both accuracy and uncertainty. In our experiments, the searched models show comparable uncertainty ability and accuracy compared to the state-of-the-art (deep ensemble)
  arXiv  Detail & Related papers  (2022-02-09T10:58:50Z)
• Calibrated and Sharp Uncertainties in Deep Learning via Density Estimation [10.209143402485406]
  This paper argues that calibration is important in practice and is easy to maintain.<n>We introduce a simple training procedure based on recalibration that yields calibrated models without sacrificing overall performance.
  arXiv  Detail & Related papers  (2021-12-14T06:19:05Z)
• Learning to Learn to Demodulate with Uncertainty Quantification via Bayesian Meta-Learning [59.014197664747165]
  We introduce the use of Bayesian meta-learning via variational inference for the purpose of obtaining well-calibrated few-pilot demodulators. The resulting Bayesian ensembles offer better calibrated soft decisions, at the computational cost of running multiple instances of the neural network for demodulation.
  arXiv  Detail & Related papers  (2021-08-02T11:07:46Z)
• A Free Lunch From ANN: Towards Efficient, Accurate Spiking Neural Networks Calibration [11.014383784032084]
  Spiking Neural Network (SNN) has been recognized as one of the next generation of neural networks. We show that a proper way to calibrate the parameters during the conversion of ANN to SNN can bring significant improvements.
  arXiv  Detail & Related papers  (2021-06-13T13:20:12Z)
• The Right Tool for the Job: Matching Model and Instance Complexities [62.95183777679024]
  As NLP models become larger, executing a trained model requires significant computational resources incurring monetary and environmental costs. We propose a modification to contextual representation fine-tuning which, during inference, allows for an early (and fast) "exit" We test our proposed modification on five different datasets in two tasks: three text classification datasets and two natural language inference benchmarks.
  arXiv  Detail & Related papers  (2020-04-16T04:28:08Z)
• On Calibration of Mixup Training for Deep Neural Networks [1.6242924916178283]
  We argue and provide empirical evidence that, due to its fundamentals, Mixup does not necessarily improve calibration. Our loss is inspired by Bayes decision theory and introduces a new training framework for designing losses for probabilistic modelling. We provide state-of-the-art accuracy with consistent improvements in calibration performance.
  arXiv  Detail & Related papers  (2020-03-22T16:54:31Z)
• Calibrating Deep Neural Networks using Focal Loss [77.92765139898906]
  Miscalibration is a mismatch between a model's confidence and its correctness. We show that focal loss allows us to learn models that are already very well calibrated. We show that our approach achieves state-of-the-art calibration without compromising on accuracy in almost all cases.
  arXiv  Detail & Related papers  (2020-02-21T17:35:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.