Look beyond labels: Incorporating functional summary information in Bayesian neural networks

• URL: http://arxiv.org/abs/2207.01234v1
• Date: Mon, 4 Jul 2022 07:06:45 GMT
• Title: Look beyond labels: Incorporating functional summary information in Bayesian neural networks
• Authors: Vishnu Raj, Tianyu Cui, Markus Heinonen and Pekka Marttinen
• Abstract summary: We present a simple approach to incorporate summary information about the predicted probability. The available summary information is incorporated as augmented data and modeled with a Dirichlet process. We show how the method can inform the model about task difficulty or class imbalance.
• Score: 11.874130244353253
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Bayesian deep learning offers a principled approach to train neural networks that accounts for both aleatoric and epistemic uncertainty. In variational inference, priors are often specified over the weight parameters, but they do not capture the true prior knowledge in large and complex neural network architectures. We present a simple approach to incorporate summary information about the predicted probability (such as sigmoid or softmax score) outputs in Bayesian neural networks (BNNs). The available summary information is incorporated as augmented data and modeled with a Dirichlet process, and we derive the corresponding \emph{Summary Evidence Lower BOund}. We show how the method can inform the model about task difficulty or class imbalance. Extensive empirical experiments show that, with negligible computational overhead, the proposed method yields a BNN with a better calibration of uncertainty.

Related papers

• A Rate-Distortion View of Uncertainty Quantification [36.85921945174863]
  In supervised learning, understanding an input's proximity to the training data can help a model decide whether it has sufficient evidence for reaching a reliable prediction. We introduce Distance Aware Bottleneck (DAB), a new method for enriching deep neural networks with this property.
  arXiv  Detail & Related papers  (2024-06-16T01:33:22Z)
• Bayesian Neural Networks with Domain Knowledge Priors [52.80929437592308]
  We propose a framework for integrating general forms of domain knowledge into a BNN prior. We show that BNNs using our proposed domain knowledge priors outperform those with standard priors.
  arXiv  Detail & Related papers  (2024-02-20T22:34:53Z)
• Deep Neural Networks Tend To Extrapolate Predictably [51.303814412294514]
  neural network predictions tend to be unpredictable and overconfident when faced with out-of-distribution (OOD) inputs. We observe that neural network predictions often tend towards a constant value as input data becomes increasingly OOD. We show how one can leverage our insights in practice to enable risk-sensitive decision-making in the presence of OOD inputs.
  arXiv  Detail & Related papers  (2023-10-02T03:25:32Z)
• Sparsifying Bayesian neural networks with latent binary variables and normalizing flows [10.865434331546126]
  We will consider two extensions to the latent binary Bayesian neural networks (LBBNN) method. Firstly, by using the local reparametrization trick (LRT) to sample the hidden units directly, we get a more computationally efficient algorithm. More importantly, by using normalizing flows on the variational posterior distribution of the LBBNN parameters, the network learns a more flexible variational posterior distribution than the mean field Gaussian.
  arXiv  Detail & Related papers  (2023-05-05T09:40:28Z)
• Neural networks trained with SGD learn distributions of increasing complexity [78.30235086565388]
  We show that neural networks trained using gradient descent initially classify their inputs using lower-order input statistics. We then exploit higher-order statistics only later during training. We discuss the relation of DSB to other simplicity biases and consider its implications for the principle of universality in learning.
  arXiv  Detail & Related papers  (2022-11-21T15:27:22Z)
• Augmenting Neural Networks with Priors on Function Values [22.776982718042962]
  Prior knowledge of function values is often available in the natural sciences. BNNs enable the user to specify prior information only on the neural network weights, not directly on the function values. We develop an approach to augment BNNs with prior information on the function values themselves.
  arXiv  Detail & Related papers  (2022-02-10T02:24:15Z)
• NUQ: Nonparametric Uncertainty Quantification for Deterministic Neural Networks [151.03112356092575]
  We show the principled way to measure the uncertainty of predictions for a classifier based on Nadaraya-Watson's nonparametric estimate of the conditional label distribution. We demonstrate the strong performance of the method in uncertainty estimation tasks on a variety of real-world image datasets.
  arXiv  Detail & Related papers  (2022-02-07T12:30:45Z)
• Analytic Mutual Information in Bayesian Neural Networks [0.8122270502556371]
  Mutual information is an example of an uncertainty measure in a Bayesian neural network to quantify uncertainty. We derive the analytical formula of the mutual information between model parameters and the predictive output by leveraging the notion of the point process entropy. As an application, we discuss the estimation of the Dirichlet parameters and show its practical application in the active learning uncertainty measures.
  arXiv  Detail & Related papers  (2022-01-24T17:30:54Z)
• Transformers Can Do Bayesian Inference [56.99390658880008]
  We present Prior-Data Fitted Networks (PFNs) PFNs leverage in-context learning in large-scale machine learning techniques to approximate a large set of posteriors. We demonstrate that PFNs can near-perfectly mimic Gaussian processes and also enable efficient Bayesian inference for intractable problems.
  arXiv  Detail & Related papers  (2021-12-20T13:07:39Z)
• Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium State [66.2457134675891]
  Spiking neural networks (SNNs) are brain-inspired models that enable energy-efficient implementation on neuromorphic hardware. Most existing methods imitate the backpropagation framework and feedforward architectures for artificial neural networks. We propose a novel training method that does not rely on the exact reverse of the forward computation.
  arXiv  Detail & Related papers  (2021-09-29T07:46:54Z)
• Bayesian Neural Networks [0.0]
  We show how errors in prediction by neural networks can be obtained in principle, and provide the two favoured methods for characterising these errors. We will also describe how both of these methods have substantial pitfalls when put into practice.
  arXiv  Detail & Related papers  (2020-06-02T09:43:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.