Rectification Difficulty and Optimal Sample Allocation in LLM-Augmented Surveys

• URL: http://arxiv.org/abs/2604.17267v1
• Date: Sun, 19 Apr 2026 05:45:37 GMT
• Title: Rectification Difficulty and Optimal Sample Allocation in LLM-Augmented Surveys
• Abstract summary: Large Language Models can generate synthetic survey responses at low cost, but their accuracy varies unpredictably across questions.<n>We study the design problem of allocating a fixed budget of human respondents across estimation tasks when cheap LLM predictions are available for every task.<n>Our framework combines three components. First, building on Prediction-Powered Inference, we characterize a question-specific rectification difficulty that governs how quickly the estimator's variance decreases with human sample size.<n>Second, we derive a closed-form optimal allocation rule that directs more human labels to tasks where the LLM is least reliable.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models can generate synthetic survey responses at low cost, but their accuracy varies unpredictably across questions. We study the design problem of allocating a fixed budget of human respondents across estimation tasks when cheap LLM predictions are available for every task. Our framework combines three components. First, building on Prediction-Powered Inference, we characterize a question-specific rectification difficulty that governs how quickly the estimator's variance decreases with human sample size. Second, we derive a closed-form optimal allocation rule that directs more human labels to tasks where the LLM is least reliable. Third, since rectification difficulty depends on unobserved human responses for new surveys, we propose a meta-learning approach, trained on historical data, that predicts it for entirely new tasks without pilot data. The framework extends to general M-estimation, covering regression coefficients and multinomial logit partworths for conjoint analysis. We validate the framework on two datasets spanning different domains, question types, and LLMs, showing that our approach captures 61-79% of the theoretically attainable efficiency gains, achieving 11.4% and 10.5% MSE reductions without requiring any pilot human data for the target survey.

Related papers

• Adaptive Budget Allocation in LLM-Augmented Surveys [4.998402019776929]
  Large language models (LLMs) can generate survey responses at low cost, but their reliability varies substantially across questions.<n>We propose an adaptive allocation algorithm that learns which questions are hardest for the LLM while simultaneously collecting human responses.
  arXiv  Detail & Related papers  (2026-04-14T09:25:34Z)
• Estimating problem difficulty without ground truth using Large Language Model comparisons [4.599673637363014]
  We propose a new method for estimating problem difficulty, LLM compare.<n>An LLM performs pairwise difficulty comparisons, and then Bradley-Terry scores are computed based on the outcomes.<n>Our work represents a significant step towards replacing time-consuming human annotations and synthetic data generation.
  arXiv  Detail & Related papers  (2025-12-16T09:13:56Z)
• Valid Survey Simulations with Limited Human Data: The Roles of Prompting, Fine-Tuning, and Rectification [12.462181498019051]
  We study the interplay between synthesis methods that use large language models to generate survey responses and rectification methods that debias population estimates.<n>We find that synthesis alone introduces substantial bias (24-86%), whereas combining it with rectification reduces bias below 5% and increases effective sample size by up to 14%.
  arXiv  Detail & Related papers  (2025-10-13T13:48:07Z)
• Learning from Convenience Samples: A Case Study on Fine-Tuning LLMs for Survey Non-response in the German Longitudinal Election Study [0.6104510780984732]
  We fine-tune large language models to impute self-reported vote choice under both random and systematic nonresponse.<n>LLMs can recover both individual-level predictions and population-level distributions more accurately than zero-shot.<n>This suggests fine-tuned LLMs offer a promising strategy for researchers working with non-probability samples or systematic missingness.
  arXiv  Detail & Related papers  (2025-09-29T17:12:18Z)
• Staying in the Sweet Spot: Responsive Reasoning Evolution via Capability-Adaptive Hint Scaffolding [59.60915947702282]
  Reinforcement learning with verifiable rewards (RLVR) has achieved remarkable success in enhancing the reasoning capabilities of large language models (LLMs)<n>Existing RLVR methods often suffer from exploration inefficiency due to mismatches between the training data's difficulty and the model's capability.<n>We propose SEELE, a novel supervision-aided RLVR framework that dynamically adjusts problem difficulty to stay within the high-efficiency region.
  arXiv  Detail & Related papers  (2025-09-08T17:36:21Z)
• Truth in the Few: High-Value Data Selection for Efficient Multi-Modal Reasoning [71.3533541927459]
  We propose a novel data selection paradigm termed Activation Reasoning Potential (RAP)<n>RAP identifies cognitive samples by estimating each sample's potential to stimulate genuine multi-modal reasoning.<n>Our RAP method consistently achieves superior performance using only 9.3% of the training data, while reducing computational costs by over 43%.
  arXiv  Detail & Related papers  (2025-06-05T08:40:24Z)
• Uncertainty Quantification for LLM-Based Survey Simulations [9.303339416902995]
  We investigate the use of large language models (LLMs) to simulate human responses to survey questions.<n>Our approach converts imperfect LLM-simulated responses into confidence sets for population parameters.<n>A key innovation lies in determining the optimal number of simulated responses.
  arXiv  Detail & Related papers  (2025-02-25T02:07:29Z)
• Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations [49.908708778200115]
  We are the first to specialize large language models (LLMs) for simulating survey response distributions.<n>As a testbed, we use country-level results from two global cultural surveys.<n>We devise a fine-tuning method based on first-token probabilities to minimize divergence between predicted and actual response distributions.
  arXiv  Detail & Related papers  (2025-02-10T21:59:27Z)
• Uncertainty Aware Learning for Language Model Alignment [97.36361196793929]
  We propose uncertainty-aware learning (UAL) to improve the model alignment of different task scenarios. We implement UAL in a simple fashion -- adaptively setting the label smoothing value of training according to the uncertainty of individual samples. Experiments on widely used benchmarks demonstrate that our UAL significantly and consistently outperforms standard supervised fine-tuning.
  arXiv  Detail & Related papers  (2024-06-07T11:37:45Z)
• How to Train Data-Efficient LLMs [56.41105687693619]
  We study data-efficient approaches for pre-training language models (LLMs) We find that Ask-LLM and Density sampling are the best methods in their respective categories. In our comparison of 19 samplers, involving hundreds of evaluation tasks and pre-training runs, we find that Ask-LLM and Density are the best methods in their respective categories.
  arXiv  Detail & Related papers  (2024-02-15T02:27:57Z)
• DRFLM: Distributionally Robust Federated Learning with Inter-client Noise via Local Mixup [58.894901088797376]
  federated learning has emerged as a promising approach for training a global model using data from multiple organizations without leaking their raw data. We propose a general framework to solve the above two challenges simultaneously. We provide comprehensive theoretical analysis including robustness analysis, convergence analysis, and generalization ability.
  arXiv  Detail & Related papers  (2022-04-16T08:08:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.