Representation Bias in Political Sample Simulations with Large Language Models

• URL: http://arxiv.org/abs/2407.11409v1
• Date: Tue, 16 Jul 2024 05:52:26 GMT
• Title: Representation Bias in Political Sample Simulations with Large Language Models
• Authors: Weihong Qi, Hanjia Lyu, Jiebo Luo,
• Abstract summary: This study seeks to identify and quantify biases in simulating political samples with Large Language Models. Using the GPT-3.5-Turbo model, we leverage data from the American National Election Studies, German Longitudinal Election Study, Zuobiao dataset, and China Family Panel Studies.
• Score: 54.48283690603358
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This study seeks to identify and quantify biases in simulating political samples with Large Language Models, specifically focusing on vote choice and public opinion. Using the GPT-3.5-Turbo model, we leverage data from the American National Election Studies, German Longitudinal Election Study, Zuobiao Dataset, and China Family Panel Studies to simulate voting behaviors and public opinions. This methodology enables us to examine three types of representation bias: disparities based on the the country's language, demographic groups, and political regime types. The findings reveal that simulation performance is generally better for vote choice than for public opinions, more accurate in English-speaking countries, more effective in bipartisan systems than in multi-partisan systems, and stronger in democratic settings than in authoritarian regimes. These results contribute to enhancing our understanding and developing strategies to mitigate biases in AI applications within the field of computational social science.

Related papers

• Language-Dependent Political Bias in AI: A Study of ChatGPT and Gemini [0.0]
  This study investigates the political tendency of large language models and the existence of differentiation according to the query language. ChatGPT and Gemini were subjected to a political axis test using 14 different languages. A comparative analysis revealed that Gemini exhibited a more pronounced liberal and left-wing tendency compared to ChatGPT.
  arXiv  Detail & Related papers  (2025-04-08T21:13:01Z)
• Only a Little to the Left: A Theory-grounded Measure of Political Bias in Large Language Models [4.8869340671593475]
  Political bias in prompt-based language models can affect their performance. We build on survey design principles to test a wide variety of input prompts, while taking into account prompt sensitivity. We compute political bias profiles across different prompt variations and find that measures of political bias are often unstable.
  arXiv  Detail & Related papers  (2025-03-20T13:51:06Z)
• ElectionSim: Massive Population Election Simulation Powered by Large Language Model Driven Agents [70.17229548653852]
  We introduce ElectionSim, an innovative election simulation framework based on large language models. We present a million-level voter pool sampled from social media platforms to support accurate individual simulation. We also introduce PPE, a poll-based presidential election benchmark to assess the performance of our framework under the U.S. presidential election scenario.
  arXiv  Detail & Related papers  (2024-10-28T05:25:50Z)
• Large Language Models Reflect the Ideology of their Creators [71.65505524599888]
  Large language models (LLMs) are trained on vast amounts of data to generate natural language. This paper shows that the ideological stance of an LLM appears to reflect the worldview of its creators.
  arXiv  Detail & Related papers  (2024-10-24T04:02:30Z)
• Will Trump Win in 2024? Predicting the US Presidential Election via Multi-step Reasoning with Large Language Models [12.939107088730513]
  Election prediction poses unique challenges, such as limited voter-level data, rapidly changing political landscapes, and the need to model complex human behavior. We introduce a multi-step reasoning framework designed for political analysis. Our approach is validated on real-world data from the American National Election Studies (ANES) 2016 and 2020. We apply our framework to predict the outcome of the 2024 U.S. presidential election in advance.
  arXiv  Detail & Related papers  (2024-10-21T06:18:53Z)
• From Experts to the Public: Governing Multimodal Language Models in Politically Sensitive Video Analysis [48.14390493099495]
  This paper examines the governance of large language models (MM-LLMs) through individual and collective deliberation. We conducted a two-step study: first, interviews with 10 journalists established a baseline understanding of expert video interpretation; second, 114 individuals from the general public engaged in deliberation using Inclusive.AI.
  arXiv  Detail & Related papers  (2024-09-15T03:17:38Z)
• Spoken Stereoset: On Evaluating Social Bias Toward Speaker in Speech Large Language Models [50.40276881893513]
  This study introduces Spoken Stereoset, a dataset specifically designed to evaluate social biases in Speech Large Language Models (SLLMs) By examining how different models respond to speech from diverse demographic groups, we aim to identify these biases. The findings indicate that while most models show minimal bias, some still exhibit slightly stereotypical or anti-stereotypical tendencies.
  arXiv  Detail & Related papers  (2024-08-14T16:55:06Z)
• High Risk of Political Bias in Black Box Emotion Inference Models [0.0]
  This paper investigates the presence of political bias in machine learning models used for sentiment analysis (SA) in social science research. We conducted a bias audit on a Polish sentiment analysis model developed in our lab. Our findings indicate that annotations by human raters propagate political biases into the model's predictions.
  arXiv  Detail & Related papers  (2024-07-18T20:31:07Z)
• Generative AI Voting: Fair Collective Choice is Resilient to LLM Biases and Inconsistencies [21.444936180683147]
  We show for the first time in real-world a proportional representation of voters in direct democracy. We also show that fair ballot aggregation methods, such as equal shares, prove to be a win-win: fairer voting outcomes for humans with fairer AI representation.
  arXiv  Detail & Related papers  (2024-05-31T01:41:48Z)
• Fairly Allocating Utility in Constrained Multiwinner Elections [0.0]
  A common denominator to ensure fairness across all such contexts is the use of constraints. Across these contexts, the candidates selected to satisfy the given constraints may systematically lead to unfair outcomes for historically disadvantaged voter populations. We develop a model to select candidates that satisfy the constraints fairly across voter populations.
  arXiv  Detail & Related papers  (2022-11-23T10:04:26Z)
• Towards Understanding and Mitigating Social Biases in Language Models [107.82654101403264]
  Large-scale pretrained language models (LMs) can be potentially dangerous in manifesting undesirable representational biases. We propose steps towards mitigating social biases during text generation. Our empirical results and human evaluation demonstrate effectiveness in mitigating bias while retaining crucial contextual information.
  arXiv  Detail & Related papers  (2021-06-24T17:52:43Z)
• Electoral Forecasting Using a Novel Temporal Attenuation Model: Predicting the US Presidential Elections [91.3755431537592]
  We develop a novel macro-scale temporal attenuation (TA) model, which uses pre-election poll data to improve forecasting accuracy. Our hypothesis is that the timing of publicizing opinion polls plays a significant role in how opinion oscillates, especially right before elections. We present two different implementations of the TA model, which accumulate an average forecasting error of 2.8-3.28 points over the 48-year period.
  arXiv  Detail & Related papers  (2020-04-30T09:21:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.