Learning to Increase the Power of Conditional Randomization Tests
- URL: http://arxiv.org/abs/2207.01022v1
- Date: Sun, 3 Jul 2022 12:29:25 GMT
- Title: Learning to Increase the Power of Conditional Randomization Tests
- Authors: Shalev Shaer and Yaniv Romano
- Abstract summary: The model-X conditional randomization test is a generic framework for conditional independence testing.
We introduce novel model-fitting schemes that are designed to explicitly improve the power of model-X tests.
- Score: 8.883733362171032
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: The model-X conditional randomization test is a generic framework for
conditional independence testing, unlocking new possibilities to discover
features that are conditionally associated with a response of interest while
controlling type-I error rates. An appealing advantage of this test is that it
can work with any machine learning model to design powerful test statistics. In
turn, the common practice in the model-X literature is to form a test statistic
using machine learning models, trained to maximize predictive accuracy with the
hope to attain a test with good power. However, the ideal goal here is to drive
the model (during training) to maximize the power of the test, not merely the
predictive accuracy. In this paper, we bridge this gap by introducing, for the
first time, novel model-fitting schemes that are designed to explicitly improve
the power of model-X tests. This is done by introducing a new cost function
that aims at maximizing the test statistic used to measure violations of
conditional independence. Using synthetic and real data sets, we demonstrate
that the combination of our proposed loss function with various base predictive
models (lasso, elastic net, and deep neural networks) consistently increases
the number of correct discoveries obtained, while maintaining type-I error
rates under control.
Related papers
- Precise Error Rates for Computationally Efficient Testing [75.63895690909241]
We revisit the question of simple-versus-simple hypothesis testing with an eye towards computational complexity.
An existing test based on linear spectral statistics achieves the best possible tradeoff curve between type I and type II error rates.
arXiv Detail & Related papers (2023-11-01T04:41:16Z) - Model-Free Sequential Testing for Conditional Independence via Testing
by Betting [8.293345261434943]
The proposed test allows researchers to analyze an incoming i.i.d. data stream with any arbitrary dependency structure.
We allow the processing of data points online as soon as they arrive and stop data acquisition once significant results are detected.
arXiv Detail & Related papers (2022-10-01T20:05:33Z) - Model-agnostic out-of-distribution detection using combined statistical
tests [15.27980070479021]
We present simple methods for out-of-distribution detection using a trained generative model.
We combine a classical parametric test (Rao's score test) with the recently introduced typicality test.
Despite their simplicity and generality, these methods can be competitive with model-specific out-of-distribution detection algorithms.
arXiv Detail & Related papers (2022-03-02T13:32:09Z) - Conformal prediction for the design problem [72.14982816083297]
In many real-world deployments of machine learning, we use a prediction algorithm to choose what data to test next.
In such settings, there is a distinct type of distribution shift between the training and test data.
We introduce a method to quantify predictive uncertainty in such settings.
arXiv Detail & Related papers (2022-02-08T02:59:12Z) - MEMO: Test Time Robustness via Adaptation and Augmentation [131.28104376280197]
We study the problem of test time robustification, i.e., using the test input to improve model robustness.
Recent prior works have proposed methods for test time adaptation, however, they each introduce additional assumptions.
We propose a simple approach that can be used in any test setting where the model is probabilistic and adaptable.
arXiv Detail & Related papers (2021-10-18T17:55:11Z) - ALT-MAS: A Data-Efficient Framework for Active Testing of Machine
Learning Algorithms [58.684954492439424]
We propose a novel framework to efficiently test a machine learning model using only a small amount of labeled test data.
The idea is to estimate the metrics of interest for a model-under-test using Bayesian neural network (BNN)
arXiv Detail & Related papers (2021-04-11T12:14:04Z) - Significance tests of feature relevance for a blackbox learner [6.72450543613463]
We derive two consistent tests for the feature relevance of a blackbox learner.
The first evaluates a loss difference with perturbation on an inference sample.
The second splits the inference sample into two but does not require data perturbation.
arXiv Detail & Related papers (2021-03-02T00:59:19Z) - Goal-directed Generation of Discrete Structures with Conditional
Generative Models [85.51463588099556]
We introduce a novel approach to directly optimize a reinforcement learning objective, maximizing an expected reward.
We test our methodology on two tasks: generating molecules with user-defined properties and identifying short python expressions which evaluate to a given target value.
arXiv Detail & Related papers (2020-10-05T20:03:13Z) - Monotonicity in practice of adaptive testing [0.0]
This article evaluates Bayesian network models used for computerized adaptive testing and learned with a recently proposed monotonicity gradient algorithm.
The quality of methods is empirically evaluated on a large data set of the Czech National Mathematics exam.
arXiv Detail & Related papers (2020-09-15T10:55:41Z) - Good Classifiers are Abundant in the Interpolating Regime [64.72044662855612]
We develop a methodology to compute precisely the full distribution of test errors among interpolating classifiers.
We find that test errors tend to concentrate around a small typical value $varepsilon*$, which deviates substantially from the test error of worst-case interpolating model.
Our results show that the usual style of analysis in statistical learning theory may not be fine-grained enough to capture the good generalization performance observed in practice.
arXiv Detail & Related papers (2020-06-22T21:12:31Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.