Beyond the Norms: Detecting Prediction Errors in Regression Models

• URL: http://arxiv.org/abs/2406.06968v1
• Date: Tue, 11 Jun 2024 05:51:44 GMT
• Title: Beyond the Norms: Detecting Prediction Errors in Regression Models
• Authors: Andres Altieri, Marco Romanelli, Georg Pichler, Florence Alberge, Pablo Piantanida,
• Abstract summary: This paper tackles the challenge of detecting unreliable behavior in regression algorithms. We introduce the notion of unreliability in regression, when the output of the regressor exceeds a specified discrepancy (or error) We show empirical improvements in error detection for multiple regression tasks, consistently outperforming popular baseline approaches.
• Score: 26.178065248948773
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper tackles the challenge of detecting unreliable behavior in regression algorithms, which may arise from intrinsic variability (e.g., aleatoric uncertainty) or modeling errors (e.g., model uncertainty). First, we formally introduce the notion of unreliability in regression, i.e., when the output of the regressor exceeds a specified discrepancy (or error). Then, using powerful tools for probabilistic modeling, we estimate the discrepancy density, and we measure its statistical diversity using our proposed metric for statistical dissimilarity. In turn, this allows us to derive a data-driven score that expresses the uncertainty of the regression outcome. We show empirical improvements in error detection for multiple regression tasks, consistently outperforming popular baseline approaches, and contributing to the broader field of uncertainty quantification and safe machine learning systems. Our code is available at https://zenodo.org/records/11281964.

Related papers

• Selective Nonparametric Regression via Testing [54.20569354303575]
  We develop an abstention procedure via testing the hypothesis on the value of the conditional variance at a given point. Unlike existing methods, the proposed one allows to account not only for the value of the variance itself but also for the uncertainty of the corresponding variance predictor.
  arXiv  Detail & Related papers  (2023-09-28T13:04:11Z)
• Beta quantile regression for robust estimation of uncertainty in the presence of outliers [1.6377726761463862]
  Quantile Regression can be used to estimate aleatoric uncertainty in deep neural networks. We propose a robust solution for quantile regression that incorporates concepts from robust divergence.
  arXiv  Detail & Related papers  (2023-09-14T01:18:57Z)
• How Reliable is Your Regression Model's Uncertainty Under Real-World Distribution Shifts? [46.05502630457458]
  We propose a benchmark of 8 image-based regression datasets with different types of challenging distribution shifts. We find that while methods are well calibrated when there is no distribution shift, they all become highly overconfident on many of the benchmark datasets.
  arXiv  Detail & Related papers  (2023-02-07T18:54:39Z)
• The Implicit Delta Method [61.36121543728134]
  In this paper, we propose an alternative, the implicit delta method, which works by infinitesimally regularizing the training loss of uncertainty. We show that the change in the evaluation due to regularization is consistent for the variance of the evaluation estimator, even when the infinitesimal change is approximated by a finite difference.
  arXiv  Detail & Related papers  (2022-11-11T19:34:17Z)
• Dense Uncertainty Estimation via an Ensemble-based Conditional Latent Variable Model [68.34559610536614]
  We argue that the aleatoric uncertainty is an inherent attribute of the data and can only be correctly estimated with an unbiased oracle model. We propose a new sampling and selection strategy at train time to approximate the oracle model for aleatoric uncertainty estimation. Our results show that our solution achieves both accurate deterministic results and reliable uncertainty estimation.
  arXiv  Detail & Related papers  (2021-11-22T08:54:10Z)
• Aleatoric uncertainty for Errors-in-Variables models in deep regression [0.48733623015338234]
  We show how the concept of Errors-in-Variables can be used in Bayesian deep regression. We discuss the approach along various simulated and real examples.
  arXiv  Detail & Related papers  (2021-05-19T12:37:02Z)
• Regression Bugs Are In Your Model! Measuring, Reducing and Analyzing Regressions In NLP Model Updates [68.09049111171862]
  This work focuses on quantifying, reducing and analyzing regression errors in the NLP model updates. We formulate the regression-free model updates into a constrained optimization problem. We empirically analyze how model ensemble reduces regression.
  arXiv  Detail & Related papers  (2021-05-07T03:33:00Z)
• Recalibration of Aleatoric and Epistemic Regression Uncertainty in Medical Imaging [2.126171264016785]
  Well-calibrated uncertainty in regression allows robust rejection of unreliable predictions or detection of out-of-distribution samples. $ sigma $ scaling is able to reliably recalibrate predictive uncertainty.
  arXiv  Detail & Related papers  (2021-04-26T07:18:58Z)
• Learning Probabilistic Ordinal Embeddings for Uncertainty-Aware Regression [91.3373131262391]
  Uncertainty is the only certainty there is. Traditionally, the direct regression formulation is considered and the uncertainty is modeled by modifying the output space to a certain family of probabilistic distributions. How to model the uncertainty within the present-day technologies for regression remains an open issue.
  arXiv  Detail & Related papers  (2021-03-25T06:56:09Z)
• Improving Uncertainty Calibration via Prior Augmented Data [56.88185136509654]
  Neural networks have proven successful at learning from complex data distributions by acting as universal function approximators. They are often overconfident in their predictions, which leads to inaccurate and miscalibrated probabilistic predictions. We propose a solution by seeking out regions of feature space where the model is unjustifiably overconfident, and conditionally raising the entropy of those predictions towards that of the prior distribution of the labels.
  arXiv  Detail & Related papers  (2021-02-22T07:02:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.