Post-ADC Inference: Valid Inference After Active Data Collection

• URL: http://arxiv.org/abs/2605.11511v1
• Date: Tue, 12 May 2026 04:27:51 GMT
• Title: Post-ADC Inference: Valid Inference After Active Data Collection
• Abstract summary: We study statistical inference on actively collected data when the inferential target is constructed in a data-dependent manner after data collection.<n>We propose post-ADC inference, a framework that accounts for the biases arising from both the active data collection process and the subsequent data-driven target construction.<n>Our method builds on selective inference and provides valid $p$-values and confidence intervals that correct for both sources of bias.
• Score: 9.892633657809785
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The validity of statistical inference depends critically on how data are collected. When data gathered through active data collection (ADC) are reused for a post-hoc inferential task, conventional inference can fail because the sampling is adaptively biased toward regions favored by the collection strategy. This issue is especially pronounced in black-box optimization, where sequential model-based optimization (SMBO) methods such as the tree-structured Parzen estimator (TPE) and Gaussian process upper confidence bound (GP-UCB) preferentially concentrate evaluations in promising regions. We study statistical inference on actively collected data when the inferential target is constructed in a data-dependent manner after data collection. To enable valid inference in this setting, we propose post-ADC inference, a framework that accounts for the biases arising from both the active data collection process and the subsequent data-driven target construction. Our method builds on selective inference and provides valid $p$-values and confidence intervals that correct for both sources of bias. The framework applies to a broad class of ADC processes by imposing only assumptions on the observation noise, without requiring any assumptions on the underlying black-box function or the surrogate model used by the SMBO algorithm. Empirical results also show that post-ADC inference provides valid inference for data collected by GP-UCB and TPE.

Related papers

• Learnable Chernoff Baselines for Inference-Time Alignment [64.81256817158851]
  We introduce Learnable Chernoff Baselines as a method for efficiently and approximately sampling from exponentially tilted kernels.<n>We establish total-variation guarantees to the ideal aligned model, and demonstrate in both continuous and discrete diffusion settings that LCB sampling closely matches ideal rejection sampling.
  arXiv  Detail & Related papers  (2026-02-08T00:09:40Z)
• Robust Sampling for Active Statistical Inference [11.929391566298841]
  Active statistical inference is a new method for inference with AI-assisted data collection.<n>We present robust sampling strategies for active statistical inference.<n>We demonstrate the utility of the method on a series of real datasets.
  arXiv  Detail & Related papers  (2025-11-12T05:18:36Z)
• Measuring Model Performance in the Presence of an Intervention [11.381587523287495]
  In many AI for social impact applications, the presence of an intervention that affects the outcome can bias the evaluation.<n>RCTs randomly assign interventions, allowing data from the control group to be used for unbiased model evaluation.<n>We propose nuisance parameter weighting (NPW), an unbiased model evaluation approach that reweights data from the treatment group to mimic the distributions of samples that would or would not experience the outcome.
  arXiv  Detail & Related papers  (2025-11-08T02:24:16Z)
• Bayesian Semi-supervised Inference via a Debiased Modeling Approach [1.2833734915643464]
  Inference in semi-supervised (SS) settings has gained substantial attention in recent years due to increased relevance in modern big-data problems.<n>We propose a novel Bayesian method for estimating the population mean in SS settings.
  arXiv  Detail & Related papers  (2025-09-22T06:49:10Z)
• Model-free Methods for Event History Analysis and Efficient Adjustment (PhD Thesis) [55.2480439325792]
  This thesis is a series of independent contributions to statistics unified by a model-free perspective.<n>The first chapter elaborates on how a model-free perspective can be used to formulate flexible methods that leverage prediction techniques from machine learning.<n>The second chapter studies the concept of local independence, which describes whether the evolution of one process is directly influenced by another.
  arXiv  Detail & Related papers  (2025-02-11T19:24:09Z)
• Stratified Prediction-Powered Inference for Hybrid Language Model Evaluation [62.2436697657307]
  Prediction-powered inference (PPI) is a method that improves statistical estimates based on limited human-labeled data.<n>We propose a method called Stratified Prediction-Powered Inference (StratPPI)<n>We show that the basic PPI estimates can be considerably improved by employing simple data stratification strategies.
  arXiv  Detail & Related papers  (2024-06-06T17:37:39Z)
• Prediction De-Correlated Inference: A safe approach for post-prediction inference [0.8192907805418583]
  Inference in this setting is often called post-prediction inference. We propose a novel assumption-lean framework for statistical inference under post-prediction setting, called Prediction De-Correlated Inference (PDC) Our approach is safe, in the sense that PDC can automatically adapt to any black-box machine-learning model and consistently outperform the supervised counterparts.
  arXiv  Detail & Related papers  (2023-12-11T16:04:48Z)
• Optimal Transport-Guided Conditional Score-Based Diffusion Models [63.14903268958398]
  Conditional score-based diffusion model (SBDM) is for conditional generation of target data with paired data as condition, and has achieved great success in image translation. To tackle the applications with partially paired or even unpaired dataset, we propose a novel Optimal Transport-guided Conditional Score-based diffusion model (OTCS) in this paper.
  arXiv  Detail & Related papers  (2023-11-02T13:28:44Z)
• Divide and Contrast: Source-free Domain Adaptation via Adaptive Contrastive Learning [122.62311703151215]
  Divide and Contrast (DaC) aims to connect the good ends of both worlds while bypassing their limitations. DaC divides the target data into source-like and target-specific samples, where either group of samples is treated with tailored goals. We further align the source-like domain with the target-specific samples using a memory bank-based Maximum Mean Discrepancy (MMD) loss to reduce the distribution mismatch.
  arXiv  Detail & Related papers  (2022-11-12T09:21:49Z)
• Two-Stage Robust and Sparse Distributed Statistical Inference for Large-Scale Data [18.34490939288318]
  We address the problem of conducting statistical inference in settings involving large-scale data that may be high-dimensional and contaminated by outliers. We propose a two-stage distributed and robust statistical inference procedures coping with high-dimensional models by promoting sparsity.
  arXiv  Detail & Related papers  (2022-08-17T11:17:47Z)
• Federated Estimation of Causal Effects from Observational Data [19.657789891394504]
  We present a novel framework for causal inference with federated data sources. We assess and integrate local causal effects from different private data sources without centralizing them.
  arXiv  Detail & Related papers  (2021-05-31T08:06:00Z)
• Decomposed Adversarial Learned Inference [118.27187231452852]
  We propose a novel approach, Decomposed Adversarial Learned Inference (DALI) DALI explicitly matches prior and conditional distributions in both data and code spaces. We validate the effectiveness of DALI on the MNIST, CIFAR-10, and CelebA datasets.
  arXiv  Detail & Related papers  (2020-04-21T20:00:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.