United in Diversity? Contextual Biases in LLM-Based Predictions of the 2024 European Parliament Elections

• URL: http://arxiv.org/abs/2409.09045v2
• Date: Thu, 17 Apr 2025 21:21:10 GMT
• Title: United in Diversity? Contextual Biases in LLM-Based Predictions of the 2024 European Parliament Elections
• Authors: Leah von der Heyde, Anna-Carolina Haensch, Alexander Wenz, Bolei Ma,
• Abstract summary: "Synthetic samples" based on large language models (LLMs) have been argued to serve as efficient alternatives to surveys of humans.<n>"Synthetic samples" might exhibit bias due to training data and fine-tuning processes being unrepresentative of diverse contexts.<n>This study investigates if and under which conditions LLM-generated synthetic samples can be used for public opinion prediction.
• Score: 42.72938925647165
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: "Synthetic samples" based on large language models (LLMs) have been argued to serve as efficient alternatives to surveys of humans, assuming that their training data includes information on human attitudes and behavior. However, LLM-synthetic samples might exhibit bias, for example due to training data and fine-tuning processes being unrepresentative of diverse contexts. Such biases risk reinforcing existing biases in research, policymaking, and society. Therefore, researchers need to investigate if and under which conditions LLM-generated synthetic samples can be used for public opinion prediction. In this study, we examine to what extent LLM-based predictions of individual public opinion exhibit context-dependent biases by predicting the results of the 2024 European Parliament elections. Prompting three LLMs with individual-level background information of 26,000 eligible European voters, we ask the LLMs to predict each person's voting behavior. By comparing them to the actual results, we show that LLM-based predictions of future voting behavior largely fail, their accuracy is unequally distributed across national and linguistic contexts, and they require detailed attitudinal information in the prompt. The findings emphasize the limited applicability of LLM-synthetic samples to public opinion prediction. In investigating their contextual biases, this study contributes to the understanding and mitigation of inequalities in the development of LLMs and their applications in computational social science.

Related papers

• Evaluating the Bias in LLMs for Surveying Opinion and Decision Making in Healthcare [7.075750841525739]
  Generative agents have been increasingly used to simulate human behaviour in silico, driven by large language models (LLMs) This work compares survey data from the Understanding America Study (UAS) on healthcare decision-making with simulated responses from generative agents. Using demographic-based prompt engineering, we create digital twins of survey respondents and analyse how well different LLMs reproduce real-world behaviours.
  arXiv  Detail & Related papers  (2025-04-11T05:11:40Z)
• Fairness in LLM-Generated Surveys [0.5720786928479238]
  Large Language Models (LLMs) excel in text generation and understanding, especially simulating socio-political and economic patterns. This study examines how LLMs perform across diverse populations by analyzing public surveys from Chile and the United States. Political identity and race significantly influence prediction accuracy, while in Chile, gender, education, and religious affiliation play more pronounced roles.
  arXiv  Detail & Related papers  (2025-01-25T23:42:20Z)
• Algorithmic Fidelity of Large Language Models in Generating Synthetic German Public Opinions: A Case Study [23.458234676060716]
  This study investigates the algorithmic fidelity of large language models (LLMs) We prompt different LLMs to generate synthetic public opinions reflective of German subpopulations by incorporating demographic features into the persona prompts. Our results show that Llama performs better than other LLMs at representing subpopulations, particularly when there is lower opinion diversity within those groups.
  arXiv  Detail & Related papers  (2024-12-17T18:46:32Z)
• Large Language Models Reflect the Ideology of their Creators [73.25935570218375]
  Large language models (LLMs) are trained on vast amounts of data to generate natural language. We uncover notable diversity in the ideological stance exhibited across different LLMs and languages.
  arXiv  Detail & Related papers  (2024-10-24T04:02:30Z)
• A Comprehensive Survey of Bias in LLMs: Current Landscape and Future Directions [0.0]
  Large Language Models (LLMs) have revolutionized various applications in natural language processing (NLP) by providing unprecedented text generation, translation, and comprehension capabilities. Their widespread deployment has brought to light significant concerns regarding biases embedded within these models. This paper presents a comprehensive survey of biases in LLMs, aiming to provide an extensive review of the types, sources, impacts, and mitigation strategies related to these biases.
  arXiv  Detail & Related papers  (2024-09-24T19:50:38Z)
• GermanPartiesQA: Benchmarking Commercial Large Language Models for Political Bias and Sycophancy [20.06753067241866]
  We evaluate and compare the alignment of six LLMs by OpenAI, Anthropic, and Cohere with German party positions. We conduct our prompt experiment for which we use the benchmark and sociodemographic data of leading German parliamentarians.
  arXiv  Detail & Related papers  (2024-07-25T13:04:25Z)
• Vox Populi, Vox AI? Using Language Models to Estimate German Public Opinion [45.84205238554709]
  We generate a synthetic sample of personas matching the individual characteristics of the 2017 German Longitudinal Election Study respondents. We ask the LLM GPT-3.5 to predict each respondent's vote choice and compare these predictions to the survey-based estimates. We find that GPT-3.5 does not predict citizens' vote choice accurately, exhibiting a bias towards the Green and Left parties.
  arXiv  Detail & Related papers  (2024-07-11T14:52:18Z)
• Whose Side Are You On? Investigating the Political Stance of Large Language Models [56.883423489203786]
  We investigate the political orientation of Large Language Models (LLMs) across a spectrum of eight polarizing topics. Our investigation delves into the political alignment of LLMs across a spectrum of eight polarizing topics, spanning from abortion to LGBTQ issues. The findings suggest that users should be mindful when crafting queries, and exercise caution in selecting neutral prompt language.
  arXiv  Detail & Related papers  (2024-03-15T04:02:24Z)
• Exploring Value Biases: How LLMs Deviate Towards the Ideal [57.99044181599786]
  Large-Language-Models (LLMs) are deployed in a wide range of applications, and their response has an increasing social impact. We show that value bias is strong in LLMs across different categories, similar to the results found in human studies.
  arXiv  Detail & Related papers  (2024-02-16T18:28:43Z)
• Large Language Models: A Survey [69.72787936480394]
  Large Language Models (LLMs) have drawn a lot of attention due to their strong performance on a wide range of natural language tasks. LLMs' ability of general-purpose language understanding and generation is acquired by training billions of model's parameters on massive amounts of text data.
  arXiv  Detail & Related papers  (2024-02-09T05:37:09Z)
• LLM Voting: Human Choices and AI Collective Decision Making [0.0]
  This paper investigates the voting behaviors of Large Language Models (LLMs), specifically GPT-4 and LLaMA-2. We observed that the choice of voting methods and the presentation order influenced LLM voting outcomes. We found that varying the persona can reduce some of these biases and enhance alignment with human choices.
  arXiv  Detail & Related papers  (2024-01-31T14:52:02Z)
• Exploring the Jungle of Bias: Political Bias Attribution in Language Models via Dependency Analysis [86.49858739347412]
  Large Language Models (LLMs) have sparked intense debate regarding the prevalence of bias in these models and its mitigation. We propose a prompt-based method for the extraction of confounding and mediating attributes which contribute to the decision process. We find that the observed disparate treatment can at least in part be attributed to confounding and mitigating attributes and model misalignment.
  arXiv  Detail & Related papers  (2023-11-15T00:02:25Z)
• Do LLMs exhibit human-like response biases? A case study in survey design [66.1850490474361]
  We investigate the extent to which large language models (LLMs) reflect human response biases, if at all. We design a dataset and framework to evaluate whether LLMs exhibit human-like response biases in survey questionnaires. Our comprehensive evaluation of nine models shows that popular open and commercial LLMs generally fail to reflect human-like behavior.
  arXiv  Detail & Related papers  (2023-11-07T15:40:43Z)
• Can Large Language Models Capture Public Opinion about Global Warming? An Empirical Assessment of Algorithmic Fidelity and Bias [0.0]
  Large language models (LLMs) have demonstrated their potential in social science research by emulating human perceptions and behaviors. This study assesses the algorithmic fidelity and bias of LLMs by utilizing two nationally representative climate change surveys.
  arXiv  Detail & Related papers  (2023-11-01T01:32:59Z)
• Bias and Fairness in Large Language Models: A Survey [73.87651986156006]
  We present a comprehensive survey of bias evaluation and mitigation techniques for large language models (LLMs) We first consolidate, formalize, and expand notions of social bias and fairness in natural language processing. We then unify the literature by proposing three intuitive, two for bias evaluation, and one for mitigation.
  arXiv  Detail & Related papers  (2023-09-02T00:32:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.