Function Space Diversity for Uncertainty Prediction via Repulsive Last-Layer Ensembles

• URL: http://arxiv.org/abs/2412.15758v1
• Date: Fri, 20 Dec 2024 10:24:08 GMT
• Title: Function Space Diversity for Uncertainty Prediction via Repulsive Last-Layer Ensembles
• Abstract summary: We discuss function space inference via particle optimization and present practical modifications that improve uncertainty estimation.<n>In this work, we demonstrate that the input samples, where particle predictions are enforced to be diverse, are detrimental to the model performance.<n>While diversity on training data itself can lead to underfitting, the use of label-destroying data augmentation, or unlabeled out-of-distribution data can improve prediction diversity and uncertainty estimates.
• Score: 11.551956337460982
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Bayesian inference in function space has gained attention due to its robustness against overparameterization in neural networks. However, approximating the infinite-dimensional function space introduces several challenges. In this work, we discuss function space inference via particle optimization and present practical modifications that improve uncertainty estimation and, most importantly, make it applicable for large and pretrained networks. First, we demonstrate that the input samples, where particle predictions are enforced to be diverse, are detrimental to the model performance. While diversity on training data itself can lead to underfitting, the use of label-destroying data augmentation, or unlabeled out-of-distribution data can improve prediction diversity and uncertainty estimates. Furthermore, we take advantage of the function space formulation, which imposes no restrictions on network parameterization other than sufficient flexibility. Instead of using full deep ensembles to represent particles, we propose a single multi-headed network that introduces a minimal increase in parameters and computation. This allows seamless integration to pretrained networks, where this repulsive last-layer ensemble can be used for uncertainty aware fine-tuning at minimal additional cost. We achieve competitive results in disentangling aleatoric and epistemic uncertainty for active learning, detecting out-of-domain data, and providing calibrated uncertainty estimates under distribution shifts with minimal compute and memory.

Related papers

• From Overfitting to Reliability: Introducing the Hierarchical Approximate Bayesian Neural Network [3.632251954989679]
  HABNN is a novel approach that uses a Gaussian-inverse-Wishart distribution as a hyperprior of the network's weights.<n>Results indicate that HABNN not only matches but often outperforms state-of-the-art models.
  arXiv  Detail & Related papers  (2025-12-15T09:08:42Z)
• Multidimensional Distributional Neural Network Output Demonstrated in Super-Resolution of Surface Wind Speed [0.0]
  We present a framework for training neural networks with a multidimensional Gaussian loss.<n>This framework generates closed-form predictive distributions over outputs with non-identically distributed and heteroscedastic structure.<n>We discuss its broader applicability to uncertainty-aware prediction in scientific models.
  arXiv  Detail & Related papers  (2025-08-21T18:22:44Z)
• FIVA: Federated Inverse Variance Averaging for Universal CT Segmentation with Uncertainty Estimation [4.544160712377809]
  This work presents a novel federated learning approach to achieve universal segmentation across diverse abdominal CT datasets.<n>The proposed method quantifies prediction uncertainty by propagating the uncertainty from the model weights.<n> Experimental evaluations demonstrate the effectiveness of this approach in improving the quality of federated aggregation and uncertainty-weighted inference.
  arXiv  Detail & Related papers  (2025-08-08T11:34:01Z)
• Tractable Function-Space Variational Inference in Bayesian Neural Networks [72.97620734290139]
  A popular approach for estimating the predictive uncertainty of neural networks is to define a prior distribution over the network parameters. We propose a scalable function-space variational inference method that allows incorporating prior information. We show that the proposed method leads to state-of-the-art uncertainty estimation and predictive performance on a range of prediction tasks.
  arXiv  Detail & Related papers  (2023-12-28T18:33:26Z)
• Function-Space Regularization for Deep Bayesian Classification [33.63495888167032]
  We apply a Dirichlet prior in predictive space and perform approximate function-space variational inference. By adapting the inference, the same function-space prior can be combined with different models without affecting model architecture or size.
  arXiv  Detail & Related papers  (2023-07-12T10:17:54Z)
• Toward Robust Uncertainty Estimation with Random Activation Functions [3.0586855806896045]
  We propose a novel approach for uncertainty quantification via ensembles, called Random Activation Functions (RAFs) Ensemble. RAFs Ensemble outperforms state-of-the-art ensemble uncertainty quantification methods on both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2023-02-28T13:17:56Z)
• Improved uncertainty quantification for neural networks with Bayesian last layer [0.0]
  Uncertainty quantification is an important task in machine learning. We present a reformulation of the log-marginal likelihood of a NN with BLL which allows for efficient training using backpropagation.
  arXiv  Detail & Related papers  (2023-02-21T20:23:56Z)
• Feature Space Particle Inference for Neural Network Ensembles [13.392254060510666]
  Particle-based inference methods offer a promising approach from a Bayesian perspective. We propose optimizing particles in the feature space where the activation of a specific intermediate layer lies. Our method encourages each member to capture distinct features, which is expected to improve ensemble prediction robustness.
  arXiv  Detail & Related papers  (2022-06-02T09:16:26Z)
• A Free Lunch with Influence Functions? Improving Neural Network Estimates with Concepts from Semiparametric Statistics [41.99023989695363]
  We explore the potential for semiparametric theory to be used to improve neural networks and machine learning algorithms. We propose a new neural network method MultiNet, which seeks the flexibility and diversity of an ensemble using a single architecture.
  arXiv  Detail & Related papers  (2022-02-18T09:35:51Z)
• Uncertainty Modeling for Out-of-Distribution Generalization [56.957731893992495]
  We argue that the feature statistics can be properly manipulated to improve the generalization ability of deep learning models. Common methods often consider the feature statistics as deterministic values measured from the learned features. We improve the network generalization ability by modeling the uncertainty of domain shifts with synthesized feature statistics during training.
  arXiv  Detail & Related papers  (2022-02-08T16:09:12Z)
• NUQ: Nonparametric Uncertainty Quantification for Deterministic Neural Networks [151.03112356092575]
  We show the principled way to measure the uncertainty of predictions for a classifier based on Nadaraya-Watson's nonparametric estimate of the conditional label distribution. We demonstrate the strong performance of the method in uncertainty estimation tasks on a variety of real-world image datasets.
  arXiv  Detail & Related papers  (2022-02-07T12:30:45Z)
• Sampling-free Variational Inference for Neural Networks with Multiplicative Activation Noise [51.080620762639434]
  We propose a more efficient parameterization of the posterior approximation for sampling-free variational inference. Our approach yields competitive results for standard regression problems and scales well to large-scale image classification tasks.
  arXiv  Detail & Related papers  (2021-03-15T16:16:18Z)
• Improving Uncertainty Calibration via Prior Augmented Data [56.88185136509654]
  Neural networks have proven successful at learning from complex data distributions by acting as universal function approximators. They are often overconfident in their predictions, which leads to inaccurate and miscalibrated probabilistic predictions. We propose a solution by seeking out regions of feature space where the model is unjustifiably overconfident, and conditionally raising the entropy of those predictions towards that of the prior distribution of the labels.
  arXiv  Detail & Related papers  (2021-02-22T07:02:37Z)
• Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift [100.52588638477862]
  We develop an approximate Bayesian inference scheme based on posterior regularisation. We demonstrate the utility of our method in the context of transferring prognostic models of prostate cancer across globally diverse populations.
  arXiv  Detail & Related papers  (2020-06-26T13:50:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.