The Measurement Revolution? Credible Measurement and Inference in the Age of AI

• URL: http://arxiv.org/abs/2608.23524v1
• Date: Mon, 24 Aug 2026 17:28:34 GMT
• Title: The Measurement Revolution? Credible Measurement and Inference in the Age of AI
• Abstract summary: AI models convert unstructured data, such as text and images, into structured variables at low cost.<n>This shifts the bottleneck from finding any scalable measure of a phenomenon to choosing among many plausible ones.<n>We argue that credible inference with AI-generated variables requires appropriately designed validation.
• Score: 8.204799574862864
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Artificial intelligence (AI) is transforming measurement in economics. AI models convert unstructured data, such as text and images, into structured variables at low cost, making previously prohibitive measurement feasible at scale. This shifts the bottleneck from finding any scalable measure of a phenomenon to choosing among many plausible ones, which may support different empirical conclusions. This review provides guidance for navigating that shift. We describe three stages at which AI enters the measurement pipeline---discovery, construct definition, and observation---and what each demands of researchers. We argue that credible inference with AI-generated variables requires appropriately designed validation: anchoring measurement to explicit criteria, rather than informal claims that a proxy is reasonable. We then examine how validation samples support valid inference even when AI predictions are arbitrarily biased, and what can be done when a random validation sample is unavailable.

Related papers

• Debiased Inference for AI-Generated Data without Gold-Standard Labels: Identification via Multiple Imperfect Measurements [0.6875312133832079]
  We propose debiased inference with multiple imperfect measurements (DMM)<n>DMM combines multiple error-prone AI measurements to enable valid downstream inference without gold-standard labels.<n>Our simulation results show that DMM yields valid inference and that adding accurate measurements can improve efficiency.
  arXiv  Detail & Related papers  (2026-08-18T20:18:41Z)
• When Surveys Become Conversations: Adaptive Matrix Validation for AI-Assisted Interviews [1.3875545441867139]
  This paper proposes Adaptive Matrix Validation (AMV), a design in which each respondent completes an AI-assisted interview.<n>The estimator first calibrates the mapped values using validation answers from other respondents, then corrects the remaining error.<n>A design-calibration simulation, an American Time Use Survey emulation, and a CHAMPS verbal-autopsy narrative study show when sparse validation can improve precision.
  arXiv  Detail & Related papers  (2026-06-23T07:30:39Z)
• The Specification as Quality Gate: Three Hypotheses on AI-Assisted Code Review [0.0]
  The dominant industry response to AI-generated code quality problems is to deploy AI reviewers.<n>This paper argues that this response is structurally circular when executable specifications are absent.
  arXiv  Detail & Related papers  (2026-03-26T11:59:05Z)
• AI-Powered Bayesian Inference [0.0]
  Generative Artificial Intelligence (GAI) has heralded an inflection point that changed how society thinks about knowledge acquisition.<n>While GAI cannot be fully trusted for decision-making, it may still provide valuable information that can be integrated into a decision pipeline.<n> variable answers to given prompts can be leveraged to construct a prior distribution which reflects assuredness of AI predictions.
  arXiv  Detail & Related papers  (2025-02-26T15:42:06Z)
• Variability Need Not Imply Error: The Case of Adequate but Semantically Distinct Responses [7.581259361859477]
  Uncertainty quantification tools can be used to reject a response when the model is uncertain'<n>We estimate the Probability the model assigns to Adequate Responses (PROBAR)<n>We find PROBAR to outperform semantic entropy across prompts with varying degrees of ambiguity/open-endedness.
  arXiv  Detail & Related papers  (2024-12-20T09:02:26Z)
• Prototype-based Aleatoric Uncertainty Quantification for Cross-modal Retrieval [139.21955930418815]
  Cross-modal Retrieval methods build similarity relations between vision and language modalities by jointly learning a common representation space. However, the predictions are often unreliable due to the Aleatoric uncertainty, which is induced by low-quality data, e.g., corrupt images, fast-paced videos, and non-detailed texts. We propose a novel Prototype-based Aleatoric Uncertainty Quantification (PAU) framework to provide trustworthy predictions by quantifying the uncertainty arisen from the inherent data ambiguity.
  arXiv  Detail & Related papers  (2023-09-29T09:41:19Z)
• On the Possibilities of AI-Generated Text Detection [76.55825911221434]
  We argue that as machine-generated text approximates human-like quality, the sample size needed for detection bounds increases. We test various state-of-the-art text generators, including GPT-2, GPT-3.5-Turbo, Llama, Llama-2-13B-Chat-HF, and Llama-2-70B-Chat-HF, against detectors, including oBERTa-Large/Base-Detector, GPTZero.
  arXiv  Detail & Related papers  (2023-04-10T17:47:39Z)
• Calibrating AI Models for Wireless Communications via Conformal Prediction [55.47458839587949]
  Conformal prediction is applied for the first time to the design of AI for communication systems. This paper investigates the application of conformal prediction as a general framework to obtain AI models that produce decisions with formal calibration guarantees.
  arXiv  Detail & Related papers  (2022-12-15T12:52:23Z)
• Auditing AI models for Verified Deployment under Semantic Specifications [65.12401653917838]
  AuditAI bridges the gap between interpretable formal verification and scalability. We show how AuditAI allows us to obtain controlled variations for verification and certified training while addressing the limitations of verifying using only pixel-space perturbations.
  arXiv  Detail & Related papers  (2021-09-25T22:53:24Z)
• AES Systems Are Both Overstable And Oversensitive: Explaining Why And Proposing Defenses [66.49753193098356]
  We investigate the reason behind the surprising adversarial brittleness of scoring models. Our results indicate that autoscoring models, despite getting trained as "end-to-end" models, behave like bag-of-words models. We propose detection-based protection models that can detect oversensitivity and overstability causing samples with high accuracies.
  arXiv  Detail & Related papers  (2021-09-24T03:49:38Z)
• Counterfactual Invariance to Spurious Correlations: Why and How to Pass Stress Tests [87.60900567941428]
  A spurious correlation' is the dependence of a model on some aspect of the input data that an analyst thinks shouldn't matter. In machine learning, these have a know-it-when-you-see-it character. We study stress testing using the tools of causal inference.
  arXiv  Detail & Related papers  (2021-05-31T14:39:38Z)
• Information-Theoretic Probing with Minimum Description Length [74.29846942213445]
  We propose an alternative to the standard probes, information-theoretic probing with minimum description length (MDL) With MDL probing, training a probe to predict labels is recast as teaching it to effectively transmit the data. We show that these methods agree in results and are more informative and stable than the standard probes.
  arXiv  Detail & Related papers  (2020-03-27T09:35:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.