Exploring LLMs for Automated Pre-Testing of Cross-Cultural Surveys

• URL: http://arxiv.org/abs/2501.05985v1
• Date: Fri, 10 Jan 2025 14:17:48 GMT
• Title: Exploring LLMs for Automated Pre-Testing of Cross-Cultural Surveys
• Authors: Divya Mani Adhikari, Vikram Kamath Cannanure, Alexander Hartland, Ingmar Weber,
• Abstract summary: We propose using large language models (LLMs) to automate the questionnaire pretesting process in cross-cultural settings.<n>Our study used LLMs to adapt a U.S.-focused climate opinion survey for a South African audience.
• Score: 41.34785468969536
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Designing culturally relevant questionnaires for ICTD research is challenging, particularly when adapting surveys for populations to non-western contexts. Prior work adapted questionnaires through expert reviews and pilot studies, which are resource-intensive and time-consuming. To address these challenges, we propose using large language models (LLMs) to automate the questionnaire pretesting process in cross-cultural settings. Our study used LLMs to adapt a U.S.-focused climate opinion survey for a South African audience. We then tested the adapted questionnaire with 116 South African participants via Prolific, asking them to provide feedback on both versions. Participants perceived the LLM-adapted questions as slightly more favorable than the traditional version. Our note opens discussions on the potential role of LLMs in adapting surveys and facilitating cross-cultural questionnaire design.

Related papers

• Synthesizing Public Opinions with LLMs: Role Creation, Impacts, and the Future to eDemorcacy [5.92971970173011]
  This paper investigates the use of Large Language Models to synthesize public opinion data. It addresses challenges in traditional survey methods like declining response rates and non-response bias. We introduce a novel technique: role creation based on knowledge injection.
  arXiv  Detail & Related papers  (2025-03-31T21:21:52Z)
• Disparities in LLM Reasoning Accuracy and Explanations: A Case Study on African American English [66.97110551643722]
  We investigate dialectal disparities in Large Language Models (LLMs) reasoning tasks. We find that LLMs produce less accurate responses and simpler reasoning chains and explanations for AAE inputs. These findings highlight systematic differences in how LLMs process and reason about different language varieties.
  arXiv  Detail & Related papers  (2025-03-06T05:15:34Z)
• Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations [49.908708778200115]
  We are the first to specialize large language models (LLMs) for simulating survey response distributions. As a testbed, we use country-level results from two global cultural surveys. We devise a fine-tuning method based on first-token probabilities to minimize divergence between predicted and actual response distributions.
  arXiv  Detail & Related papers  (2025-02-10T21:59:27Z)
• Vox Populi, Vox AI? Using Language Models to Estimate German Public Opinion [45.84205238554709]
  We generate a synthetic sample of personas matching the individual characteristics of the 2017 German Longitudinal Election Study respondents. We ask the LLM GPT-3.5 to predict each respondent's vote choice and compare these predictions to the survey-based estimates. We find that GPT-3.5 does not predict citizens' vote choice accurately, exhibiting a bias towards the Green and Left parties.
  arXiv  Detail & Related papers  (2024-07-11T14:52:18Z)
• CaLMQA: Exploring culturally specific long-form question answering across 23 languages [58.18984409715615]
  CaLMQA is a collection of 1.5K culturally specific questions spanning 23 languages and 51 culturally translated questions from English into 22 other languages. We collect naturally-occurring questions from community web forums and hire native speakers to write questions to cover under-studied languages such as Fijian and Kirundi. Our dataset contains diverse, complex questions that reflect cultural topics (e.g. traditions, laws, news) and the language usage of native speakers.
  arXiv  Detail & Related papers  (2024-06-25T17:45:26Z)
• Let LLMs Take on the Latest Challenges! A Chinese Dynamic Question Answering Benchmark [69.3415799675046]
  We introduce CDQA, a Chinese Dynamic QA benchmark containing question-answer pairs related to the latest news on the Chinese Internet. We obtain high-quality data through a pipeline that combines humans and models. We have also evaluated and analyzed mainstream and advanced Chinese LLMs on CDQA.
  arXiv  Detail & Related papers  (2024-02-29T15:22:13Z)
• Crowdsourced Adaptive Surveys [0.0]
  This paper introduces a crowdsourced adaptive survey methodology (CSAS)<n>The method converts open-ended text provided by participants into survey items and applies a multi-armed bandit algorithm to determine which questions should be prioritized in the survey.<n>I conclude by highlighting CSAS's potential to bridge conceptual gaps between researchers and participants in survey research.
  arXiv  Detail & Related papers  (2024-01-16T04:05:25Z)
• You don't need a personality test to know these models are unreliable: Assessing the Reliability of Large Language Models on Psychometric Instruments [37.03210795084276]
  We examine whether the current format of prompting Large Language Models elicits responses in a consistent and robust manner. Our experiments on 17 different LLMs reveal that even simple perturbations significantly downgrade a model's question-answering ability. Our results suggest that the currently widespread practice of prompting is insufficient to accurately and reliably capture model perceptions.
  arXiv  Detail & Related papers  (2023-11-16T09:50:53Z)
• Can LLMs Grade Short-Answer Reading Comprehension Questions : An Empirical Study with a Novel Dataset [0.0]
  This paper investigates the potential for the newest version of Large Language Models (LLMs) to be used in short answer questions for formative assessments. It introduces a novel dataset of short answer reading comprehension questions, drawn from a set of reading assessments conducted with over 150 students in Ghana. The paper empirically evaluates how well various configurations of generative LLMs grade student short answer responses compared to expert human raters.
  arXiv  Detail & Related papers  (2023-10-26T17:05:40Z)
• AI-Augmented Surveys: Leveraging Large Language Models and Surveys for Opinion Prediction [0.0]
  Large language models (LLMs) that produce human-like responses have begun to revolutionize research practices in the social sciences. We develop a novel methodological framework that fine-tunes LLMs with repeated cross-sectional surveys. We introduce two new applications of the AI-augmented survey: retrodiction (i.e., predict year-level missing responses) and unasked opinion prediction.
  arXiv  Detail & Related papers  (2023-05-16T17:13:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.