Advanced Tutorial: Label-Efficient Two-Sample Tests

• URL: http://arxiv.org/abs/2501.03568v1
• Date: Tue, 07 Jan 2025 06:43:18 GMT
• Title: Advanced Tutorial: Label-Efficient Two-Sample Tests
• Authors: Weizhi Li, Visar Berisha, Gautam Dasarathy,
• Abstract summary: This tutorial explores two-sample testing in a context where an analyst has many features from two samples. In machine learning, a similar scenario is studied in active learning. This tutorial extends active learning concepts to two-sample testing within this textitlabel-costly setting.
• Score: 15.574402626262053
• License:
• Abstract: Hypothesis testing is a statistical inference approach used to determine whether data supports a specific hypothesis. An important type is the two-sample test, which evaluates whether two sets of data points are from identical distributions. This test is widely used, such as by clinical researchers comparing treatment effectiveness. This tutorial explores two-sample testing in a context where an analyst has many features from two samples, but determining the sample membership (or labels) of these features is costly. In machine learning, a similar scenario is studied in active learning. This tutorial extends active learning concepts to two-sample testing within this \textit{label-costly} setting while maintaining statistical validity and high testing power. Additionally, the tutorial discusses practical applications of these label-efficient two-sample tests.

Related papers

• Statistical Verification of Linear Classifiers [76.95660509846216]
  We propose a homogeneity test closely related to the concept of linear separability between two samples. We focus on establishing upper bounds for the test's emphp-value when applied to two-dimensional samples.
  arXiv  Detail & Related papers  (2025-01-24T11:56:45Z)
• Downstream-Pretext Domain Knowledge Traceback for Active Learning [138.02530777915362]
  We propose a downstream-pretext domain knowledge traceback (DOKT) method that traces the data interactions of downstream knowledge and pre-training guidance. DOKT consists of a traceback diversity indicator and a domain-based uncertainty estimator. Experiments conducted on ten datasets show that our model outperforms other state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-07-20T01:34:13Z)
• Training Guarantees of Neural Network Classification Two-Sample Tests by Kernel Analysis [58.435336033383145]
  We construct and analyze a neural network two-sample test to determine whether two datasets came from the same distribution. We derive the theoretical minimum training time needed to ensure the NTK two-sample test detects a deviation-level between the datasets. We show that the statistical power associated with the neural network two-sample test goes to 1 as the neural network training samples and test evaluation samples go to infinity.
  arXiv  Detail & Related papers  (2024-07-05T18:41:16Z)
• Active Sequential Two-Sample Testing [18.99517340397671]
  We consider the two-sample testing problem in a new scenario where sample measurements are inexpensive to access. We devise the first emphactiveNIST-sample testing framework that not only sequentially but also emphactively queries. In practice, we introduce an instantiation of our framework and evaluate it using several experiments.
  arXiv  Detail & Related papers  (2023-01-30T02:23:49Z)
• Two-sample test based on Self-Organizing Maps [68.8204255655161]
  Machine-learning classifiers can be leveraged as a two-sample statistical test. Self-Organizing Maps are a dimensionality reduction initially devised as a data visualization tool. But since their original purpose is visualization, they can also offer insights.
  arXiv  Detail & Related papers  (2022-12-17T21:35:47Z)
• E-Valuating Classifier Two-Sample Tests [11.248868528186332]
  Our test combines ideas from existing work-valid split likelihood ratio tests and predictive independence tests. The resulting E-values are suitable for anytime sequential two-sample tests.
  arXiv  Detail & Related papers  (2022-10-24T08:18:36Z)
• AutoML Two-Sample Test [13.468660785510945]
  We use a simple test that takes the mean discrepancy of a witness function as the test statistic and prove that minimizing a squared loss leads to a witness with optimal testing power. We provide an implementation of the AutoML two-sample test in the Python package autotst.
  arXiv  Detail & Related papers  (2022-06-17T15:41:07Z)
• Label efficient two-sample test [39.0914588747459]
  Two-sample tests evaluate whether two samples are realizations of the same distribution (the null hypothesis) or two different distributions (the alternative hypothesis) In this paper, we consider this important variation on the classical two-sample test problem and pose it as a problem of obtaining the labels of only a small number of samples in service of performing a two-sample test. We devise a label efficient three-stage framework: firstly, a classifier is trained with samples uniformly labeled to model the posterior probabilities of the labels; secondly, an innovative query scheme dubbed emphbimodal query is used to query labels
  arXiv  Detail & Related papers  (2021-11-17T01:55:01Z)
• An Optimal Witness Function for Two-Sample Testing [13.159512679346685]
  We propose data-dependent test statistics based on a one-dimensional witness function, which we call witness two-sample tests (WiTS) We show that the WiTS test based on a characteristic kernel is consistent against any fixed alternative.
  arXiv  Detail & Related papers  (2021-02-10T17:13:21Z)
• Two-Sample Testing on Ranked Preference Data and the Role of Modeling Assumptions [57.77347280992548]
  In this paper, we design two-sample tests for pairwise comparison data and ranking data. Our test requires essentially no assumptions on the distributions. By applying our two-sample test on real-world pairwise comparison data, we conclude that ratings and rankings provided by people are indeed distributed differently.
  arXiv  Detail & Related papers  (2020-06-21T20:51:09Z)
• Learning Deep Kernels for Non-Parametric Two-Sample Tests [50.92621794426821]
  We propose a class of kernel-based two-sample tests, which aim to determine whether two sets of samples are drawn from the same distribution. Our tests are constructed from kernels parameterized by deep neural nets, trained to maximize test power.
  arXiv  Detail & Related papers  (2020-02-21T03:54:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.