On the Unknowable Limits to Prediction

• URL: http://arxiv.org/abs/2411.19223v5
• Date: Mon, 10 Feb 2025 22:34:34 GMT
• Title: On the Unknowable Limits to Prediction
• Authors: Jiani Yan, Charles Rahal,
• Abstract summary: Many domains stand to benefit from iterative enhancements in measurement, construct validity, and modeling.<n>Our approach demonstrates how apparently 'unpredictable' outcomes can become more tractable with improved data.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose a rigorous decomposition of predictive error, highlighting that not all 'irreducible' error is genuinely immutable. Many domains stand to benefit from iterative enhancements in measurement, construct validity, and modeling. Our approach demonstrates how apparently 'unpredictable' outcomes can become more tractable with improved data (across both target and features) and refined algorithms. By distinguishing aleatoric from epistemic error, we delineate how accuracy may asymptotically improve--though inherent stochasticity may remain--and offer a robust framework for advancing computational research.

Related papers

• Bin-Conditional Conformal Prediction of Fatalities from Armed Conflict [0.5312303275762104]
  We introduce a novel extension to conformal prediction algorithm which we call bin-conditional conformal prediction. This method allows users to obtain individual-level prediction intervals for any arbitrary prediction model. We apply the bin-conditional conformal prediction algorithm to forecast fatalities from armed conflict.
  arXiv  Detail & Related papers  (2024-10-18T14:41:42Z)
• Ensured: Explanations for Decreasing the Epistemic Uncertainty in Predictions [1.2289361708127877]
  Epistem uncertainty adds a crucial dimension to explanation quality. We introduce new types of explanations that specifically target this uncertainty. We introduce a new metric, ensured ranking, designed to help users identify the most reliable explanations.
  arXiv  Detail & Related papers  (2024-10-07T20:21:51Z)
• Calibrated Probabilistic Forecasts for Arbitrary Sequences [58.54729945445505]
  Real-world data streams can change unpredictably due to distribution shifts, feedback loops and adversarial actors. We present a forecasting framework ensuring valid uncertainty estimates regardless of how data evolves.
  arXiv  Detail & Related papers  (2024-09-27T21:46:42Z)
• Generalized Gaussian Temporal Difference Error for Uncertainty-aware Reinforcement Learning [0.19418036471925312]
  We introduce a novel framework for generalized Gaussian error modeling in deep reinforcement learning. We improve the estimation and mitigation of data-dependent aleatoric uncertainty. Experiments with policy gradient algorithms demonstrate significant performance gains.
  arXiv  Detail & Related papers  (2024-08-05T08:12:25Z)
• Quantification of Predictive Uncertainty via Inference-Time Sampling [57.749601811982096]
  We propose a post-hoc sampling strategy for estimating predictive uncertainty accounting for data ambiguity. The method can generate different plausible outputs for a given input and does not assume parametric forms of predictive distributions.
  arXiv  Detail & Related papers  (2023-08-03T12:43:21Z)
• Model-Based Uncertainty in Value Functions [89.31922008981735]
  We focus on characterizing the variance over values induced by a distribution over MDPs. Previous work upper bounds the posterior variance over values by solving a so-called uncertainty Bellman equation. We propose a new uncertainty Bellman equation whose solution converges to the true posterior variance over values.
  arXiv  Detail & Related papers  (2023-02-24T09:18:27Z)
• On Calibrated Model Uncertainty in Deep Learning [0.0]
  We extend the approximate inference for the loss-calibrated Bayesian framework to dropweights based Bayesian neural networks. We show that decisions informed by loss-calibrated uncertainty can improve diagnostic performance to a greater extent than straightforward alternatives.
  arXiv  Detail & Related papers  (2022-06-15T20:16:32Z)
• On the Difficulty of Epistemic Uncertainty Quantification in Machine Learning: The Case of Direct Uncertainty Estimation through Loss Minimisation [8.298716599039501]
  Uncertainty quantification has received increasing attention in machine learning. The latter refers to the learner's (lack of) knowledge and appears to be especially difficult to measure and quantify. We show that loss minimisation does not work for second-order predictors.
  arXiv  Detail & Related papers  (2022-03-11T17:26:05Z)
• Dense Uncertainty Estimation via an Ensemble-based Conditional Latent Variable Model [68.34559610536614]
  We argue that the aleatoric uncertainty is an inherent attribute of the data and can only be correctly estimated with an unbiased oracle model. We propose a new sampling and selection strategy at train time to approximate the oracle model for aleatoric uncertainty estimation. Our results show that our solution achieves both accurate deterministic results and reliable uncertainty estimation.
  arXiv  Detail & Related papers  (2021-11-22T08:54:10Z)
• Aleatoric uncertainty for Errors-in-Variables models in deep regression [0.48733623015338234]
  We show how the concept of Errors-in-Variables can be used in Bayesian deep regression. We discuss the approach along various simulated and real examples.
  arXiv  Detail & Related papers  (2021-05-19T12:37:02Z)
• DEUP: Direct Epistemic Uncertainty Prediction [56.087230230128185]
  Epistemic uncertainty is part of out-of-sample prediction error due to the lack of knowledge of the learner. We propose a principled approach for directly estimating epistemic uncertainty by learning to predict generalization error and subtracting an estimate of aleatoric uncertainty.
  arXiv  Detail & Related papers  (2021-02-16T23:50:35Z)
• The Aleatoric Uncertainty Estimation Using a Separate Formulation with Virtual Residuals [51.71066839337174]
  Existing methods can quantify the error in the target estimation, but they tend to underestimate it. We propose a new separable formulation for the estimation of a signal and of its uncertainty, avoiding the effect of overfitting. We demonstrate that the proposed method outperforms a state-of-the-art technique for signal and uncertainty estimation.
  arXiv  Detail & Related papers  (2020-11-03T12:11:27Z)
• Revisiting One-vs-All Classifiers for Predictive Uncertainty and Out-of-Distribution Detection in Neural Networks [22.34227625637843]
  We investigate how the parametrization of the probabilities in discriminative classifiers affects the uncertainty estimates. We show that one-vs-all formulations can improve calibration on image classification tasks.
  arXiv  Detail & Related papers  (2020-07-10T01:55:02Z)
• Efficient Ensemble Model Generation for Uncertainty Estimation with Bayesian Approximation in Segmentation [74.06904875527556]
  We propose a generic and efficient segmentation framework to construct ensemble segmentation models. In the proposed method, ensemble models can be efficiently generated by using the layer selection method. We also devise a new pixel-wise uncertainty loss, which improves the predictive performance.
  arXiv  Detail & Related papers  (2020-05-21T16:08:38Z)
• Learning to Predict Error for MRI Reconstruction [67.76632988696943]
  We demonstrate that predictive uncertainty estimated by the current methods does not highly correlate with prediction error. We propose a novel method that estimates the target labels and magnitude of the prediction error in two steps.
  arXiv  Detail & Related papers  (2020-02-13T15:55:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.