Debiased Inference for AI-Generated Data without Gold-Standard Labels: Identification via Multiple Imperfect Measurements

• URL: http://arxiv.org/abs/2608.18294v1
• Date: Tue, 18 Aug 2026 20:18:41 GMT
• Title: Debiased Inference for AI-Generated Data without Gold-Standard Labels: Identification via Multiple Imperfect Measurements
• Abstract summary: We propose debiased inference with multiple imperfect measurements (DMM)<n>DMM combines multiple error-prone AI measurements to enable valid downstream inference without gold-standard labels.<n>Our simulation results show that DMM yields valid inference and that adding accurate measurements can improve efficiency.
• Score: 0.6875312133832079
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: An increasing number of scholars use AI to measure variables they subsequently include in downstream analyses. Although AI-measured variables are often analyzed as if observed without error, ignoring prediction errors in automated measurement leads to substantial bias and invalid confidence intervals in downstream analyses, even if AI measurement accuracy is high, e.g., above 90%. Existing solutions, such as design-based supervised learning and prediction-powered inference, combine error-prone AI-based measurements with gold-standard labels, which may be costly and difficult to obtain in some application areas. In this paper, we propose debiased inference with multiple imperfect measurements (DMM), a framework that combines multiple error-prone AI measurements to enable valid downstream inference without gold-standard labels. Building on the established results on CP decomposition, DMM assumes that these measurements are independent conditional on the latent true label and observed unit-level features, such as text features represented by embeddings. This framework allows for unknown misclassification rates to vary across annotation methods (e.g., large language models) and across units of annotation (e.g., texts). Under this assumption, we use semiparametric inference theory to prove that the DMM estimator is consistent and asymptotically normal, enabling valid inference for a wide range of downstream statistical analyses common in the social sciences. Our simulation results show that DMM yields valid inference and that adding accurate, though imperfect, measurements can improve efficiency. Focusing on common applications of large language model annotations, we also develop diagnostics to assess the conditional independence assumption.

Related papers

• The Role of Ambiguity in Error Prediction via Uncertainty Quantification [6.973934615085226]
  This paper presents a method for improving error prediction for Large Language Models (LLMs) by disentangling input ambiguity from UQ signal.<n>We conduct experiments on the task of Question Answering (QA) with six UQ metrics and show that UQ metrics are more predictive of errors on unambiguous instances than on questions with multiple plausible answers.
  arXiv  Detail & Related papers  (2026-06-01T11:20:57Z)
• A Theoretical Framework for Statistical Evaluability of Generative Models [57.9316356505791]
  We introduce a theoretical framework for evaluating generative models and establish evaluability results for commonly used metrics.<n>We study two categories of metrics: test-based metrics, including integral probability metrics (IPMs) and Rényi divergences.<n>We show that IPMs with respect to any bounded test class can be evaluated from finite samples up to multiplicative and additive approximation errors.<n>In contrast, Rényi and KL divergences are not evaluable from finite samples, as their values can be critically determined by rare events.
  arXiv  Detail & Related papers  (2026-04-07T01:53:59Z)
• Tree-Based Predictive Models for Noisy Input Data [0.0]
  We propose measurement error BART (meBART), a novel extension to the BART model that directly incorporates measurement error in the independent variable(s)<n>We show that in the presence of measurement error, our model enables more accurate parameter estimation, more robust uncertainty quantification, and superior predictive performance.
  arXiv  Detail & Related papers  (2026-03-08T01:41:55Z)
• Multidimensional Uncertainty Quantification via Optimal Transport [87.97146725546502]
  We take a multidimensional view on uncertainty quantification (UQ) by stacking complementary UQ measures into a vector.<n>VecUQ-OT shows high efficiency even when individual measures fail.
  arXiv  Detail & Related papers  (2025-09-26T14:09:03Z)
• A Sample Efficient Conditional Independence Test in the Presence of Discretization [54.047334792855345]
  Conditional Independence (CI) tests directly to discretized data can lead to incorrect conclusions.<n>Recent advancements have sought to infer the correct CI relationship between the latent variables through binarizing observed data.<n>Motivated by this, this paper introduces a sample-efficient CI test that does not rely on the binarization process.
  arXiv  Detail & Related papers  (2025-06-10T12:41:26Z)
• Position: All Current Generative Fidelity and Diversity Metrics are Flawed [58.815519650465774]
  We show that all current generative fidelity and diversity metrics are flawed.<n>Our aim is to convince the research community to spend more effort in developing metrics, instead of models.
  arXiv  Detail & Related papers  (2025-05-28T15:10:33Z)
• Ranked from Within: Ranking Large Multimodal Models Without Labels [73.96543593298426]
  We show that uncertainty scores derived from softmax distributions provide a robust basis for ranking models across various tasks.<n>This facilitates the ranking of LMMs on unlabeled data, providing a practical approach for selecting models for diverse target domains without requiring manual annotation.
  arXiv  Detail & Related papers  (2024-12-09T13:05:43Z)
• Cycles of Thought: Measuring LLM Confidence through Stable Explanations [53.15438489398938]
  Large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, but their overconfidence in incorrect responses is still a well-documented failure mode. We propose a framework for measuring an LLM's uncertainty with respect to the distribution of generated explanations for an answer.
  arXiv  Detail & Related papers  (2024-06-05T16:35:30Z)
• Uncertainty Quantification Metrics for Deep Regression [1.1999555634662633]
  Area Under Sparsification Error, Error, Spearman's Rank Correlation, and Negative Log-Likelihood are investigated. We look into how those metrics behave under four typical types of uncertainty. Our results indicate that Error is the most stable and interpretable metric, but AUSE and NLL also have their respective use cases.
  arXiv  Detail & Related papers  (2024-05-07T12:46:45Z)
• Quantifying predictive uncertainty of aphasia severity in stroke patients with sparse heteroscedastic Bayesian high-dimensional regression [47.1405366895538]
  Sparse linear regression methods for high-dimensional data commonly assume that residuals have constant variance, which can be violated in practice. This paper proposes estimating high-dimensional heteroscedastic linear regression models using a heteroscedastic partitioned empirical Bayes Expectation Conditional Maximization algorithm.
  arXiv  Detail & Related papers  (2023-09-15T22:06:29Z)
• Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Models [0.2812395851874055]
  computational social science (CSS) analyze documents to explain social and political phenomena. One increasingly common way to annotate documents cheaply at scale is through large language models. We present a new algorithm for using imperfect annotation surrogates for downstream statistical analyses.
  arXiv  Detail & Related papers  (2023-06-07T19:49:41Z)
• Robust Bayesian Inference for Berkson and Classical Measurement Error Models [9.712913056924826]
  We propose a nonparametric framework for dealing with measurement error. It is suitable for both Classical and Berkson error models. It offers flexibility in the choice of loss function depending on the type of regression model.
  arXiv  Detail & Related papers  (2023-06-02T11:48:15Z)
• Conditional Feature Importance for Mixed Data [1.6114012813668934]
  We develop a conditional predictive impact (CPI) framework with knockoff sampling. We show that our proposed workflow controls type I error, achieves high power and is in line with results given by other conditional FI measures. Our findings highlight the necessity of developing statistically adequate, specialized methods for mixed data.
  arXiv  Detail & Related papers  (2022-10-06T16:52:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.