The fine print on tempered posteriors

• URL: http://arxiv.org/abs/2309.05292v1
• Date: Mon, 11 Sep 2023 08:21:42 GMT
• Title: The fine print on tempered posteriors
• Authors: Konstantinos Pitas, Julyan Arbel
• Abstract summary: We conduct a detailed investigation of tempered posteriors and uncover a number of crucial and previously unspecified points. Contrary to previous works, we finally show through a PAC-Bayesian analysis that the temperature $lambda$ cannot be seen as simply fixing a misdiscussed prior or likelihood.
• Score: 4.503508912578133
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We conduct a detailed investigation of tempered posteriors and uncover a number of crucial and previously undiscussed points. Contrary to previous results, we first show that for realistic models and datasets and the tightly controlled case of the Laplace approximation to the posterior, stochasticity does not in general improve test accuracy. The coldest temperature is often optimal. One might think that Bayesian models with some stochasticity can at least obtain improvements in terms of calibration. However, we show empirically that when gains are obtained this comes at the cost of degradation in test accuracy. We then discuss how targeting Frequentist metrics using Bayesian models provides a simple explanation of the need for a temperature parameter $\lambda$ in the optimization objective. Contrary to prior works, we finally show through a PAC-Bayesian analysis that the temperature $\lambda$ cannot be seen as simply fixing a misspecified prior or likelihood.

Related papers

• Temperature Optimization for Bayesian Deep Learning [9.610060788662972]
  We propose a data-driven approach to select the temperature that maximizes test log-predictive density. We empirically demonstrate that our method performs comparably to grid search, at a fraction of the cost.
  arXiv  Detail & Related papers  (2024-10-08T07:32:22Z)
• Can a Confident Prior Replace a Cold Posterior? [20.018444020989712]
  We introduce a "DirClip" prior that is practical to sample and nearly matches the performance of a cold posterior. Second, we introduce a "confidence prior" that directly approximates a cold likelihood in the limit of decreasing temperature but cannot be easily sampled.
  arXiv  Detail & Related papers  (2024-03-02T17:28:55Z)
• Extracting or Guessing? Improving Faithfulness of Event Temporal Relation Extraction [87.04153383938969]
  We improve the faithfulness of TempRel extraction models from two perspectives. The first perspective is to extract genuinely based on contextual description. The second perspective is to provide proper uncertainty estimation.
  arXiv  Detail & Related papers  (2022-10-10T19:53:13Z)
• Sample-dependent Adaptive Temperature Scaling for Improved Calibration [95.7477042886242]
  Post-hoc approach to compensate for neural networks being wrong is to perform temperature scaling. We propose to predict a different temperature value for each input, allowing us to adjust the mismatch between confidence and accuracy. We test our method on the ResNet50 and WideResNet28-10 architectures using the CIFAR10/100 and Tiny-ImageNet datasets.
  arXiv  Detail & Related papers  (2022-07-13T14:13:49Z)
• Boost Test-Time Performance with Closed-Loop Inference [85.43516360332646]
  We propose to predict hard-classified test samples in a looped manner to boost the model performance. We first devise a filtering criterion to identify those hard-classified test samples that need additional inference loops. For each hard sample, we construct an additional auxiliary learning task based on its original top-$K$ predictions to calibrate the model.
  arXiv  Detail & Related papers  (2022-03-21T10:20:21Z)
• Posterior temperature optimized Bayesian models for inverse problems in medical imaging [59.82184400837329]
  We present an unsupervised Bayesian approach to inverse problems in medical imaging using mean-field variational inference with a fully tempered posterior. We show that an optimized posterior temperature leads to improved accuracy and uncertainty estimation. Our source code is publicly available at calibrated.com/Cardio-AI/mfvi-dip-mia.
  arXiv  Detail & Related papers  (2022-02-02T12:16:33Z)
• Posterior Temperature Optimization in Variational Inference [69.50862982117127]
  Cold posteriors have been reported to perform better in practice in the context of deep learning. In this work, we first derive the ELBO for a fully tempered posterior in mean-field variational inference. We then use Bayesian optimization to automatically find the optimal posterior temperature.
  arXiv  Detail & Related papers  (2021-06-11T13:01:28Z)
• Cold Posteriors and Aleatoric Uncertainty [32.341379426923105]
  Recent work has observed that one can outperform exact inference in Bayesian neural networks by tuning the "temperature" of the posterior on a validation set. We argue that commonly used priors can significantly overestimate the aleatoric uncertainty in the labels on many classification datasets.
  arXiv  Detail & Related papers  (2020-07-31T18:37:31Z)
• Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift [81.74795324629712]
  We call prediction-time batch normalization, which significantly improves model accuracy and calibration under covariate shift. We show that prediction-time batch normalization provides complementary benefits to existing state-of-the-art approaches for improving robustness. The method has mixed results when used alongside pre-training, and does not seem to perform as well under more natural types of dataset shift.
  arXiv  Detail & Related papers  (2020-06-19T05:08:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.