Conditional independence testing under misspecified inductive biases
- URL: http://arxiv.org/abs/2307.02520v2
- Date: Fri, 27 Oct 2023 22:33:18 GMT
- Title: Conditional independence testing under misspecified inductive biases
- Authors: Felipe Maia Polo, Yuekai Sun, Moulinath Banerjee
- Abstract summary: We study the performance of regression-based CI tests under misspecified inductive biases.
Namely, we propose new approximations or upper bounds for the testing errors of three regression-based tests.
We introduce the Rao-Blackwellized Predictor Test (RBPT), a regression-based CI test robust against misspecified inductive biases.
- Score: 27.34558936393097
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Conditional independence (CI) testing is a fundamental and challenging task
in modern statistics and machine learning. Many modern methods for CI testing
rely on powerful supervised learning methods to learn regression functions or
Bayes predictors as an intermediate step; we refer to this class of tests as
regression-based tests. Although these methods are guaranteed to control Type-I
error when the supervised learning methods accurately estimate the regression
functions or Bayes predictors of interest, their behavior is less understood
when they fail due to misspecified inductive biases; in other words, when the
employed models are not flexible enough or when the training algorithm does not
induce the desired predictors. Then, we study the performance of
regression-based CI tests under misspecified inductive biases. Namely, we
propose new approximations or upper bounds for the testing errors of three
regression-based tests that depend on misspecification errors. Moreover, we
introduce the Rao-Blackwellized Predictor Test (RBPT), a regression-based CI
test robust against misspecified inductive biases. Finally, we conduct
experiments with artificial and real data, showcasing the usefulness of our
theory and methods.
Related papers
- Detecting Model Misspecification in Amortized Bayesian Inference with Neural Networks: An Extended Investigation [9.950524371154394]
We propose a new misspecification measure that can be trained in an unsupervised fashion and reliably detects model misspecification at test time.
We show how the proposed misspecification test warns users about suspicious outputs, raises an alarm when predictions are not trustworthy, and guides model designers in their search for better simulators.
arXiv Detail & Related papers (2024-06-05T11:30:16Z) - Uncertainty-Calibrated Test-Time Model Adaptation without Forgetting [55.17761802332469]
Test-time adaptation (TTA) seeks to tackle potential distribution shifts between training and test data by adapting a given model w.r.t. any test sample.
Prior methods perform backpropagation for each test sample, resulting in unbearable optimization costs to many applications.
We propose an Efficient Anti-Forgetting Test-Time Adaptation (EATA) method which develops an active sample selection criterion to identify reliable and non-redundant samples.
arXiv Detail & Related papers (2024-03-18T05:49:45Z) - Selective Learning: Towards Robust Calibration with Dynamic Regularization [79.92633587914659]
Miscalibration in deep learning refers to there is a discrepancy between the predicted confidence and performance.
We introduce Dynamic Regularization (DReg) which aims to learn what should be learned during training thereby circumventing the confidence adjusting trade-off.
arXiv Detail & Related papers (2024-02-13T11:25:20Z) - Test Generation Strategies for Building Failure Models and Explaining
Spurious Failures [4.995172162560306]
Test inputs fail not only when the system under test is faulty but also when the inputs are invalid or unrealistic.
We propose to build failure models for inferring interpretable rules on test inputs that cause spurious failures.
We show that our proposed surrogate-assisted approach generates failure models with an average accuracy of 83%.
arXiv Detail & Related papers (2023-12-09T18:36:15Z) - Precise Error Rates for Computationally Efficient Testing [75.63895690909241]
We revisit the question of simple-versus-simple hypothesis testing with an eye towards computational complexity.
An existing test based on linear spectral statistics achieves the best possible tradeoff curve between type I and type II error rates.
arXiv Detail & Related papers (2023-11-01T04:41:16Z) - Selective Nonparametric Regression via Testing [54.20569354303575]
We develop an abstention procedure via testing the hypothesis on the value of the conditional variance at a given point.
Unlike existing methods, the proposed one allows to account not only for the value of the variance itself but also for the uncertainty of the corresponding variance predictor.
arXiv Detail & Related papers (2023-09-28T13:04:11Z) - Testing for Overfitting [0.0]
We discuss the overfitting problem and explain why standard and concentration results do not hold for evaluation with training data.
We introduce and argue for a hypothesis test by means of which both model performance may be evaluated using training data.
arXiv Detail & Related papers (2023-05-09T22:49:55Z) - Sequential Kernelized Independence Testing [101.22966794822084]
We design sequential kernelized independence tests inspired by kernelized dependence measures.
We demonstrate the power of our approaches on both simulated and real data.
arXiv Detail & Related papers (2022-12-14T18:08:42Z) - Learning to Increase the Power of Conditional Randomization Tests [8.883733362171032]
The model-X conditional randomization test is a generic framework for conditional independence testing.
We introduce novel model-fitting schemes that are designed to explicitly improve the power of model-X tests.
arXiv Detail & Related papers (2022-07-03T12:29:25Z) - Certifying Data-Bias Robustness in Linear Regression [12.00314910031517]
We present a technique for certifying whether linear regression models are pointwise-robust to label bias in a training dataset.
We show how to solve this problem exactly for individual test points, and provide an approximate but more scalable method.
We also unearth gaps in bias-robustness, such as high levels of non-robustness for certain bias assumptions on some datasets.
arXiv Detail & Related papers (2022-06-07T20:47:07Z) - Conformal prediction for the design problem [72.14982816083297]
In many real-world deployments of machine learning, we use a prediction algorithm to choose what data to test next.
In such settings, there is a distinct type of distribution shift between the training and test data.
We introduce a method to quantify predictive uncertainty in such settings.
arXiv Detail & Related papers (2022-02-08T02:59:12Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.