On the Epistemic Uncertainty of Overparametrized Neural Networks

• URL: http://arxiv.org/abs/2605.25234v1
• Date: Sun, 24 May 2026 19:39:29 GMT
• Title: On the Epistemic Uncertainty of Overparametrized Neural Networks
• Abstract summary: Epistemic uncertainty is often viewed as a reducible uncertainty that vanishes with increasing data.<n>In overparametrized neural networks, model parameters are typically non-identifiable due to symmetries and redundant representations.
• Score: 16.62208593810495
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Epistemic uncertainty is often viewed as a reducible uncertainty that vanishes with increasing data. This perspective implicitly assumes parameter identifiability and equates epistemic uncertainty with predictive variability. In overparametrized neural networks, however, model parameters are typically non-identifiable due to symmetries and redundant representations. As a consequence, substantial parameter uncertainty can persist even when the underlying function is fully identified. In this work, we analyze epistemic uncertainty through the lens of non-identifiability and characterize both discrete and continuous sources of residual uncertainty. Focusing on one-hidden-layer ReLU networks, we thoroughly analyze the resulting posterior structure and validate our theoretical insights through empirical studies.

Related papers

• Why Machine Learning Models Fail to Fully Capture Epistemic Uncertainty [3.4970971805884474]
  We make use of a more fine-grained taxonomy of epistemic uncertainty sources in machine learning models.<n>We show that high model bias can lead to misleadingly low estimates of epistemic uncertainty.<n>Common second-order uncertainty methods systematically blur bias-induced errors into aleatoric estimates.
  arXiv  Detail & Related papers  (2025-05-29T14:50:46Z)
• Probabilistic Modeling of Disparity Uncertainty for Robust and Efficient Stereo Matching [61.73532883992135]
  We propose a new uncertainty-aware stereo matching framework.<n>We adopt Bayes risk as the measurement of uncertainty and use it to separately estimate data and model uncertainty.
  arXiv  Detail & Related papers  (2024-12-24T23:28:20Z)
• Information-theoretic Analysis of Test Data Sensitivity in Uncertainty [32.27260899727789]
  A recent analysis by Xu and Raginsky 2022 rigorously decomposed the predictive uncertainty in Bayesian inference into two uncertainties. They analyzed those uncertainties in an information-theoretic way, assuming that the model is well-specified and treating the model's parameters as latent variables. In this work, we study such uncertainty sensitivity using our novel decomposition method for the predictive uncertainty.
  arXiv  Detail & Related papers  (2023-07-23T23:42:06Z)
• Posterior Collapse and Latent Variable Non-identifiability [54.842098835445]
  We propose a class of latent-identifiable variational autoencoders, deep generative models which enforce identifiability without sacrificing flexibility. Across synthetic and real datasets, latent-identifiable variational autoencoders outperform existing methods in mitigating posterior collapse and providing meaningful representations of the data.
  arXiv  Detail & Related papers  (2023-01-02T06:16:56Z)
• Looking at the posterior: accuracy and uncertainty of neural-network predictions [0.0]
  We show that prediction accuracy depends on both epistemic and aleatoric uncertainty. We introduce a novel acquisition function that outperforms common uncertainty-based methods.
  arXiv  Detail & Related papers  (2022-11-26T16:13:32Z)
• The Unreasonable Effectiveness of Deep Evidential Regression [72.30888739450343]
  A new approach with uncertainty-aware regression-based neural networks (NNs) shows promise over traditional deterministic methods and typical Bayesian NNs. We detail the theoretical shortcomings and analyze the performance on synthetic and real-world data sets, showing that Deep Evidential Regression is a quantification rather than an exact uncertainty.
  arXiv  Detail & Related papers  (2022-05-20T10:10:32Z)
• Dense Uncertainty Estimation via an Ensemble-based Conditional Latent Variable Model [68.34559610536614]
  We argue that the aleatoric uncertainty is an inherent attribute of the data and can only be correctly estimated with an unbiased oracle model. We propose a new sampling and selection strategy at train time to approximate the oracle model for aleatoric uncertainty estimation. Our results show that our solution achieves both accurate deterministic results and reliable uncertainty estimation.
  arXiv  Detail & Related papers  (2021-11-22T08:54:10Z)
• Joint Dermatological Lesion Classification and Confidence Modeling with Uncertainty Estimation [23.817227116949958]
  We propose an overall framework that jointly considers dermatological classification and uncertainty estimation together. The estimated confidence of each feature to avoid uncertain feature and undesirable shift is pooled from confidence network. We demonstrate the potential of the proposed approach in two state-of-the-art dermoscopic datasets.
  arXiv  Detail & Related papers  (2021-07-19T11:54:37Z)
• Bayesian Uncertainty Estimation of Learned Variational MRI Reconstruction [63.202627467245584]
  We introduce a Bayesian variational framework to quantify the model-immanent (epistemic) uncertainty. We demonstrate that our approach yields competitive results for undersampled MRI reconstruction.
  arXiv  Detail & Related papers  (2021-02-12T18:08:14Z)
• The Hidden Uncertainty in a Neural Networks Activations [105.4223982696279]
  The distribution of a neural network's latent representations has been successfully used to detect out-of-distribution (OOD) data. This work investigates whether this distribution correlates with a model's epistemic uncertainty, thus indicating its ability to generalise to novel inputs.
  arXiv  Detail & Related papers  (2020-12-05T17:30:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.