(Implicit) Ensembles of Ensembles: Epistemic Uncertainty Collapse in Large Models

• URL: http://arxiv.org/abs/2409.02628v1
• Date: Wed, 4 Sep 2024 11:45:55 GMT
• Title: (Implicit) Ensembles of Ensembles: Epistemic Uncertainty Collapse in Large Models
• Authors: Andreas Kirsch,
• Abstract summary: Epistemic uncertainty is crucial for safety-critical applications and out-of-distribution detection tasks. We uncover a paradoxical phenomenon in deep learning models: an epistemic uncertainty collapse as model complexity increases.
• Score: 3.0539022029583953
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Epistemic uncertainty is crucial for safety-critical applications and out-of-distribution detection tasks. Yet, we uncover a paradoxical phenomenon in deep learning models: an epistemic uncertainty collapse as model complexity increases, challenging the assumption that larger models invariably offer better uncertainty quantification. We propose that this stems from implicit ensembling within large models. To support this hypothesis, we demonstrate epistemic uncertainty collapse empirically across various architectures, from explicit ensembles of ensembles and simple MLPs to state-of-the-art vision models, including ResNets and Vision Transformers -- for the latter, we examine implicit ensemble extraction and decompose larger models into diverse sub-models, recovering epistemic uncertainty. We provide theoretical justification for these phenomena and explore their implications for uncertainty estimation.

Related papers

• On the Calibration of Epistemic Uncertainty: Principles, Paradoxes and Conflictual Loss [3.8248583585487155]
  Evidential uncertainty is produced by Deep Ensembles, Bayesian Deep Networks, or Evidential Deep Networks. Although measurable, this form of uncertainty is difficult to calibrate on an objective basis. We propose a regularization function for deep ensembles, called conflictual loss in line with the above requirements.
  arXiv  Detail & Related papers  (2024-07-16T23:21:28Z)
• Revealing Multimodal Contrastive Representation Learning through Latent Partial Causal Models [85.67870425656368]
  We introduce a unified causal model specifically designed for multimodal data. We show that multimodal contrastive representation learning excels at identifying latent coupled variables. Experiments demonstrate the robustness of our findings, even when the assumptions are violated.
  arXiv  Detail & Related papers  (2024-02-09T07:18:06Z)
• Identifiable Latent Polynomial Causal Models Through the Lens of Change [82.14087963690561]
  Causal representation learning aims to unveil latent high-level causal representations from observed low-level data. One of its primary tasks is to provide reliable assurance of identifying these latent causal models, known as identifiability.
  arXiv  Detail & Related papers  (2023-10-24T07:46:10Z)
• On the Embedding Collapse when Scaling up Recommendation Models [53.66285358088788]
  We identify the embedding collapse phenomenon as the inhibition of scalability, wherein the embedding matrix tends to occupy a low-dimensional subspace. We propose a simple yet effective multi-embedding design incorporating embedding-set-specific interaction modules to learn embedding sets with large diversity.
  arXiv  Detail & Related papers  (2023-10-06T17:50:38Z)
• Measuring and Modeling Uncertainty Degree for Monocular Depth Estimation [50.920911532133154]
  The intrinsic ill-posedness and ordinal-sensitive nature of monocular depth estimation (MDE) models pose major challenges to the estimation of uncertainty degree. We propose to model the uncertainty of MDE models from the perspective of the inherent probability distributions. By simply introducing additional training regularization terms, our model, with surprisingly simple formations and without requiring extra modules or multiple inferences, can provide uncertainty estimations with state-of-the-art reliability.
  arXiv  Detail & Related papers  (2023-07-19T12:11:15Z)
• Quantification of Uncertainty with Adversarial Models [6.772632213236167]
  Quantifying uncertainty is important for actionable predictions in real-world applications. We suggest Quantification of Uncertainty with Adversarial Models (QUAM) QUAM identifies regions where the whole product under the integral is large, not just the posterior.
  arXiv  Detail & Related papers  (2023-07-06T17:56:10Z)
• High Fidelity Image Counterfactuals with Probabilistic Causal Models [25.87025672100077]
  We present a causal generative modelling framework for accurate estimation of high fidelity image counterfactuals with deep structural causal models. We leverage ideas from causal mediation analysis and advances in generative modelling to design new deep causal mechanisms for structured variables in causal models.
  arXiv  Detail & Related papers  (2023-06-27T19:28:41Z)
• Decision-Making Under Uncertainty: Beyond Probabilities [5.358161704743754]
  A classical assumption is that probabilities can sufficiently capture all uncertainty in a system. In this paper, the focus is on the uncertainty that goes beyond this classical interpretation. We show several solution techniques for both discrete and continuous models.
  arXiv  Detail & Related papers  (2023-03-10T10:53:33Z)
• Fairness Increases Adversarial Vulnerability [50.90773979394264]
  This paper shows the existence of a dichotomy between fairness and robustness, and analyzes when achieving fairness decreases the model robustness to adversarial samples. Experiments on non-linear models and different architectures validate the theoretical findings in multiple vision domains. The paper proposes a simple, yet effective, solution to construct models achieving good tradeoffs between fairness and robustness.
  arXiv  Detail & Related papers  (2022-11-21T19:55:35Z)
• A General Framework for Survival Analysis and Multi-State Modelling [70.31153478610229]
  We use neural ordinary differential equations as a flexible and general method for estimating multi-state survival models. We show that our model exhibits state-of-the-art performance on popular survival data sets and demonstrate its efficacy in a multi-state setting.
  arXiv  Detail & Related papers  (2020-06-08T19:24:54Z)
• Model Uncertainty Quantification for Reliable Deep Vision Structural Health Monitoring [2.5126058470073263]
  This paper proposes Bayesian inference for deep vision structural health monitoring models. Uncertainty can be quantified using the Monte Carlo dropout sampling. Three independent case studies for cracks, local damage identification, and bridge component detection are investigated.
  arXiv  Detail & Related papers  (2020-04-10T17:54:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.