The $s$-value: evaluating stability with respect to distributional shifts

• URL: http://arxiv.org/abs/2105.03067v4
• Date: Thu, 4 Jul 2024 22:56:06 GMT
• Title: The $s$-value: evaluating stability with respect to distributional shifts
• Authors: Suyash Gupta, Dominik Rothenhäusler,
• Abstract summary: In practice, distributions change between locations and across time. This makes it difficult to gather knowledge that transfers across data sets. We propose a measure of instability that quantifies the distributional instability of a statistical parameter with respect to Kullback-Leibler divergence. We evaluate the performance of the proposed measure on real data and show that it can elucidate the distributional instability of a parameter with respect to certain shifts.
• Score: 3.330229314824913
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Common statistical measures of uncertainty such as $p$-values and confidence intervals quantify the uncertainty due to sampling, that is, the uncertainty due to not observing the full population. However, sampling is not the only source of uncertainty. In practice, distributions change between locations and across time. This makes it difficult to gather knowledge that transfers across data sets. We propose a measure of instability that quantifies the distributional instability of a statistical parameter with respect to Kullback-Leibler divergence, that is, the sensitivity of the parameter under general distributional perturbations within a Kullback-Leibler divergence ball. In addition, we quantify the instability of parameters with respect to directional or variable-specific shifts. Measuring instability with respect to directional shifts can be used to detect the type of shifts a parameter is sensitive to. We discuss how such knowledge can inform data collection for improved estimation of statistical parameters under shifted distributions. We evaluate the performance of the proposed measure on real data and show that it can elucidate the distributional instability of a parameter with respect to certain shifts and can be used to improve estimation accuracy under shifted distributions.

Related papers

• Uncertainty Decomposition and Error Margin Detection of Homodyned-K Distribution in Quantitative Ultrasound [1.912429179274357]
  Homodyned K-distribution (HK-distribution) parameter estimation in quantitative ultrasound (QUS) has been recently addressed using Bayesian Neural Networks (BNNs) BNNs have been shown to significantly reduce computational time in speckle statistics-based QUS without compromising accuracy and precision.
  arXiv  Detail & Related papers  (2024-09-17T22:16:49Z)
• Probabilistic Contrastive Learning with Explicit Concentration on the Hypersphere [3.572499139455308]
  This paper introduces a new perspective on incorporating uncertainty into contrastive learning by embedding representations within a spherical space. We leverage the concentration parameter, kappa, as a direct, interpretable measure to quantify uncertainty explicitly.
  arXiv  Detail & Related papers  (2024-05-26T07:08:13Z)
• Quantification of Predictive Uncertainty via Inference-Time Sampling [57.749601811982096]
  We propose a post-hoc sampling strategy for estimating predictive uncertainty accounting for data ambiguity. The method can generate different plausible outputs for a given input and does not assume parametric forms of predictive distributions.
  arXiv  Detail & Related papers  (2023-08-03T12:43:21Z)
• Reliable amortized variational inference with physics-based latent distribution correction [0.4588028371034407]
  A neural network is trained to approximate the posterior distribution over existing pairs of model and data. The accuracy of this approach relies on the availability of high-fidelity training data. We show that our correction step improves the robustness of amortized variational inference with respect to changes in number of source experiments, noise variance, and shifts in the prior distribution.
  arXiv  Detail & Related papers  (2022-07-24T02:38:54Z)
• Certifying Model Accuracy under Distribution Shifts [151.67113334248464]
  We present provable robustness guarantees on the accuracy of a model under bounded Wasserstein shifts of the data distribution. We show that a simple procedure that randomizes the input of the model within a transformation space is provably robust to distributional shifts under the transformation.
  arXiv  Detail & Related papers  (2022-01-28T22:03:50Z)
• Decomposing Representations for Deterministic Uncertainty Estimation [34.11413246048065]
  We show that current feature density based uncertainty estimators cannot perform well consistently across different OoD detection settings. We propose to decompose the learned representations and integrate the uncertainties estimated on them separately.
  arXiv  Detail & Related papers  (2021-12-01T22:12:01Z)
• Dense Uncertainty Estimation via an Ensemble-based Conditional Latent Variable Model [68.34559610536614]
  We argue that the aleatoric uncertainty is an inherent attribute of the data and can only be correctly estimated with an unbiased oracle model. We propose a new sampling and selection strategy at train time to approximate the oracle model for aleatoric uncertainty estimation. Our results show that our solution achieves both accurate deterministic results and reliable uncertainty estimation.
  arXiv  Detail & Related papers  (2021-11-22T08:54:10Z)
• Training on Test Data with Bayesian Adaptation for Covariate Shift [96.3250517412545]
  Deep neural networks often make inaccurate predictions with unreliable uncertainty estimates. We derive a Bayesian model that provides for a well-defined relationship between unlabeled inputs under distributional shift and model parameters. We show that our method improves both accuracy and uncertainty estimation.
  arXiv  Detail & Related papers  (2021-09-27T01:09:08Z)
• The Hidden Uncertainty in a Neural Networks Activations [105.4223982696279]
  The distribution of a neural network's latent representations has been successfully used to detect out-of-distribution (OOD) data. This work investigates whether this distribution correlates with a model's epistemic uncertainty, thus indicating its ability to generalise to novel inputs.
  arXiv  Detail & Related papers  (2020-12-05T17:30:35Z)
• Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift [100.52588638477862]
  We develop an approximate Bayesian inference scheme based on posterior regularisation. We demonstrate the utility of our method in the context of transferring prognostic models of prostate cancer across globally diverse populations.
  arXiv  Detail & Related papers  (2020-06-26T13:50:19Z)
• Nonparametric Estimation of Uncertainty Sets for Robust Optimization [2.741266294612776]
  We investigate a data-driven approach to constructing uncertainty sets for robust optimization problems. We provide a nonparametric method to estimate uncertainty sets whose probability mass is guaranteed to approximate a given target mass.
  arXiv  Detail & Related papers  (2020-04-07T01:47:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.