When prompt perturbations break your A/B test: A valid statistical test for generative surveying

• URL: http://arxiv.org/abs/2605.27463v1
• Date: Tue, 26 May 2026 00:35:58 GMT
• Title: When prompt perturbations break your A/B test: A valid statistical test for generative surveying
• Abstract summary: We show that standard hypothesis tests, including the sign test and Wilcoxon signed-rank test, are invalid under a statistical model for generative surveying.<n>We propose a permutation test that is valid under this model and formally characterize the conditions under which standard tests fail.
• Score: 0.3366038232188622
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generative surveying -- where collections of LLM-based personas provide feedback on messages -- has emerged as a cheap and scalable alternative to traditional market research. However, LLMs are sensitive to small variations in prompt design and conclusions drawn from generative surveys may depend on arbitrary phrasing choices. Controlling for this sensitivity requires including semantically equivalent perturbations in the analysis. In this paper, we show that standard hypothesis tests, including the sign test and Wilcoxon signed-rank test, are invalid under a statistical model for generative surveying that includes realistic perturbation structure. We propose a permutation test that is valid under this model and formally characterize the conditions under which standard tests fail. Applying our framework to a simple generative surveying problem, we estimate relevant parameters, characterize the power of the permutation test under realistic conditions, and provide practical guidance on budget allocation across personas, perturbations, and replicates. Finally, we show that both the magnitude and direction of the estimated effect are sensitive to the choice of model, even within the same model family.

Related papers

• The Debiased Score Test: Hunt-and-test for Semiparametric Hypotheses [1.7969777786551424]
  The parametric score test assesses a hypothesis through derivatives of the log-likelihood, whose expectation vanishes under the null.<n>This yields goodness-of-fit tests for common semiparametric regression models.<n>We show the test has power whenever the hunted direction is correlated with the true score.
  arXiv  Detail & Related papers  (2026-07-30T22:01:08Z)
• Statistical Hypothesis Testing for Auditing Robustness in Language Models [49.1574468325115]
  We introduce distribution-based perturbation analysis, a framework that reformulates perturbation analysis as a frequentist hypothesis testing problem.<n>We construct empirical null and alternative output distributions within a low-dimensional semantic similarity space via Monte Carlo sampling.<n>We show how we can quantify response changes, measure true/false positive rates, and evaluate alignment with reference models.
  arXiv  Detail & Related papers  (2025-06-09T17:11:07Z)
• A Fresh Look at Sanity Checks for Saliency Maps [3.371877306561591]
  We propose two modifications to the Model Randomisation Test (MPRT) The former reduces the impact of noise on evaluation outcomes via sampling, while the latter avoids the need for biased similarity measurements. Our experiments show that these modifications enhance the metric reliability, facilitating a more trustworthy deployment of explanation methods.
  arXiv  Detail & Related papers  (2024-05-03T15:47:32Z)
• Hypothesis Testing for Class-Conditional Noise Using Local Maximum Likelihood [1.8798171797988192]
  In supervised learning, automatically assessing the quality of the labels before any learning takes place remains an open research question. In this paper we show how similar procedures can be followed when the underlying model is a product of Local Maximum Likelihood Estimation. This different view allows for wider applicability of the tests by offering users access to a richer model class.
  arXiv  Detail & Related papers  (2023-12-15T22:14:58Z)
• Can You Rely on Your Model Evaluation? Improving Model Evaluation with Synthetic Test Data [75.20035991513564]
  We introduce 3S Testing, a deep generative modeling framework to facilitate model evaluation. Our experiments demonstrate that 3S Testing outperforms traditional baselines. These results raise the question of whether we need a paradigm shift away from limited real test data towards synthetic test data.
  arXiv  Detail & Related papers  (2023-10-25T10:18:44Z)
• A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts [117.72709110877939]
  Test-time adaptation (TTA) has the potential to adapt a pre-trained model to unlabeled data during testing, before making predictions.<n>We categorize TTA into several distinct groups based on the form of test data, namely, test-time domain adaptation, test-time batch adaptation, and online test-time adaptation.
  arXiv  Detail & Related papers  (2023-03-27T16:32:21Z)
• Shortcomings of Top-Down Randomization-Based Sanity Checks for Evaluations of Deep Neural Network Explanations [67.40641255908443]
  We identify limitations of model-randomization-based sanity checks for the purpose of evaluating explanations. Top-down model randomization preserves scales of forward pass activations with high probability.
  arXiv  Detail & Related papers  (2022-11-22T18:52:38Z)
• fAux: Testing Individual Fairness via Gradient Alignment [2.5329739965085785]
  We describe a new approach for testing individual fairness that does not have either requirement. We show that the proposed method effectively identifies discrimination on both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2022-10-10T21:27:20Z)
• Efficient Test-Time Model Adaptation without Forgetting [60.36499845014649]
  Test-time adaptation seeks to tackle potential distribution shifts between training and testing data. We propose an active sample selection criterion to identify reliable and non-redundant samples. We also introduce a Fisher regularizer to constrain important model parameters from drastic changes.
  arXiv  Detail & Related papers  (2022-04-06T06:39:40Z)
• Calibrating Over-Parametrized Simulation Models: A Framework via Eligibility Set [3.862247454265944]
  We develop a framework to develop calibration schemes that satisfy rigorous frequentist statistical guarantees. We demonstrate our methodology on several numerical examples, including an application to calibration of a limit order book market simulator.
  arXiv  Detail & Related papers  (2021-05-27T00:59:29Z)
• Understanding Classifier Mistakes with Generative Models [88.20470690631372]
  Deep neural networks are effective on supervised learning tasks, but have been shown to be brittle. In this paper, we leverage generative models to identify and characterize instances where classifiers fail to generalize. Our approach is agnostic to class labels from the training set which makes it applicable to models trained in a semi-supervised way.
  arXiv  Detail & Related papers  (2020-10-05T22:13:21Z)
• Achieving Equalized Odds by Resampling Sensitive Attributes [13.114114427206678]
  We present a flexible framework for learning predictive models that approximately satisfy the equalized odds notion of fairness. This differentiable functional is used as a penalty driving the model parameters towards equalized odds. We develop a formal hypothesis test to detect whether a prediction rule violates this property, the first such test in the literature.
  arXiv  Detail & Related papers  (2020-06-08T00:18:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.