Representation Bias in Political Sample Simulations with Large Language Models

• URL: http://arxiv.org/abs/2407.11409v1
• Date: Tue, 16 Jul 2024 05:52:26 GMT
• Title: Representation Bias in Political Sample Simulations with Large Language Models
• Authors: Weihong Qi, Hanjia Lyu, Jiebo Luo,
• Abstract summary: This study seeks to identify and quantify biases in simulating political samples with Large Language Models. Using the GPT-3.5-Turbo model, we leverage data from the American National Election Studies, German Longitudinal Election Study, Zuobiao dataset, and China Family Panel Studies.
• Score: 54.48283690603358
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This study seeks to identify and quantify biases in simulating political samples with Large Language Models, specifically focusing on vote choice and public opinion. Using the GPT-3.5-Turbo model, we leverage data from the American National Election Studies, German Longitudinal Election Study, Zuobiao Dataset, and China Family Panel Studies to simulate voting behaviors and public opinions. This methodology enables us to examine three types of representation bias: disparities based on the the country's language, demographic groups, and political regime types. The findings reveal that simulation performance is generally better for vote choice than for public opinions, more accurate in English-speaking countries, more effective in bipartisan systems than in multi-partisan systems, and stronger in democratic settings than in authoritarian regimes. These results contribute to enhancing our understanding and developing strategies to mitigate biases in AI applications within the field of computational social science.

Related papers

• Democratic or Authoritarian? Probing a New Dimension of Political Biases in Large Language Models [72.89977583150748]
  We propose a novel methodology to assess how Large Language Models align with broader geopolitical value systems.<n>We find that LLMs generally favor democratic values and leaders, but exhibit increases favorability toward authoritarian figures when prompted in Mandarin.
  arXiv  Detail & Related papers  (2025-06-15T07:52:07Z)
• Geopolitical biases in LLMs: what are the "good" and the "bad" countries according to contemporary language models [52.00270888041742]
  We introduce a novel dataset with neutral event descriptions and contrasting viewpoints from different countries.<n>Our findings show significant geopolitical biases, with models favoring specific national narratives.<n>Simple debiasing prompts had a limited effect on reducing these biases.
  arXiv  Detail & Related papers  (2025-06-07T10:45:17Z)
• KOKKAI DOC: An LLM-driven framework for scaling parliamentary representatives [0.0]
  This paper introduces an LLM-driven framework designed to accurately scale the political issue stances of parliamentary representatives.<n>By leveraging advanced natural language processing techniques and large language models, the proposed methodology refines and enhances previous approaches.<n>The framework incorporates three major innovations: (1) de-noising parliamentary speeches via summarization to produce cleaner, more consistent opinion embeddings; (2) automatic extraction of axes of political controversy from legislators' speech summaries; and (3) a diachronic analysis that tracks the evolution of party positions over time.
  arXiv  Detail & Related papers  (2025-05-11T21:03:53Z)
• Language-Dependent Political Bias in AI: A Study of ChatGPT and Gemini [0.0]
  This study investigates the political tendency of large language models and the existence of differentiation according to the query language. ChatGPT and Gemini were subjected to a political axis test using 14 different languages. A comparative analysis revealed that Gemini exhibited a more pronounced liberal and left-wing tendency compared to ChatGPT.
  arXiv  Detail & Related papers  (2025-04-08T21:13:01Z)
• Only a Little to the Left: A Theory-grounded Measure of Political Bias in Large Language Models [4.8869340671593475]
  Political bias in prompt-based language models can affect their performance. We build on survey design principles to test a wide variety of input prompts, while taking into account prompt sensitivity. We compute political bias profiles across different prompt variations and find that measures of political bias are often unstable.
  arXiv  Detail & Related papers  (2025-03-20T13:51:06Z)
• Beyond Partisan Leaning: A Comparative Analysis of Political Bias in Large Language Models [6.549047699071195]
  This study adopts a persona-free, topic-specific approach to evaluate political behavior in large language models.<n>We analyze responses from 43 large language models developed in the U.S., Europe, China, and the Middle East.<n>Findings show most models lean center-left or left ideologically and vary in their nonpartisan engagement patterns.
  arXiv  Detail & Related papers  (2024-12-21T19:42:40Z)
• ElectionSim: Massive Population Election Simulation Powered by Large Language Model Driven Agents [70.17229548653852]
  We introduce ElectionSim, an innovative election simulation framework based on large language models. We present a million-level voter pool sampled from social media platforms to support accurate individual simulation. We also introduce PPE, a poll-based presidential election benchmark to assess the performance of our framework under the U.S. presidential election scenario.
  arXiv  Detail & Related papers  (2024-10-28T05:25:50Z)
• Large Language Models Reflect the Ideology of their Creators [71.65505524599888]
  Large language models (LLMs) are trained on vast amounts of data to generate natural language. This paper shows that the ideological stance of an LLM appears to reflect the worldview of its creators.
  arXiv  Detail & Related papers  (2024-10-24T04:02:30Z)
• Will Trump Win in 2024? Predicting the US Presidential Election via Multi-step Reasoning with Large Language Models [12.939107088730513]
  Election prediction poses unique challenges, such as limited voter-level data, rapidly changing political landscapes, and the need to model complex human behavior. We introduce a multi-step reasoning framework designed for political analysis. Our approach is validated on real-world data from the American National Election Studies (ANES) 2016 and 2020. We apply our framework to predict the outcome of the 2024 U.S. presidential election in advance.
  arXiv  Detail & Related papers  (2024-10-21T06:18:53Z)
• From Experts to the Public: Governing Multimodal Language Models in Politically Sensitive Video Analysis [48.14390493099495]
  This paper examines the governance of large language models (MM-LLMs) through individual and collective deliberation. We conducted a two-step study: first, interviews with 10 journalists established a baseline understanding of expert video interpretation; second, 114 individuals from the general public engaged in deliberation using Inclusive.AI.
  arXiv  Detail & Related papers  (2024-09-15T03:17:38Z)
• Spoken Stereoset: On Evaluating Social Bias Toward Speaker in Speech Large Language Models [50.40276881893513]
  This study introduces Spoken Stereoset, a dataset specifically designed to evaluate social biases in Speech Large Language Models (SLLMs) By examining how different models respond to speech from diverse demographic groups, we aim to identify these biases. The findings indicate that while most models show minimal bias, some still exhibit slightly stereotypical or anti-stereotypical tendencies.
  arXiv  Detail & Related papers  (2024-08-14T16:55:06Z)
• High Risk of Political Bias in Black Box Emotion Inference Models [0.0]
  This paper investigates the presence of political bias in machine learning models used for sentiment analysis (SA) in social science research. We conducted a bias audit on a Polish sentiment analysis model developed in our lab. Our findings indicate that annotations by human raters propagate political biases into the model's predictions.
  arXiv  Detail & Related papers  (2024-07-18T20:31:07Z)
• Generative AI Voting: Fair Collective Choice is Resilient to LLM Biases and Inconsistencies [21.444936180683147]
  We show for the first time in real-world a proportional representation of voters in direct democracy. We also show that fair ballot aggregation methods, such as equal shares, prove to be a win-win: fairer voting outcomes for humans with fairer AI representation.
  arXiv  Detail & Related papers  (2024-05-31T01:41:48Z)
• Fairly Allocating Utility in Constrained Multiwinner Elections [0.0]
  A common denominator to ensure fairness across all such contexts is the use of constraints. Across these contexts, the candidates selected to satisfy the given constraints may systematically lead to unfair outcomes for historically disadvantaged voter populations. We develop a model to select candidates that satisfy the constraints fairly across voter populations.
  arXiv  Detail & Related papers  (2022-11-23T10:04:26Z)
• Towards Understanding and Mitigating Social Biases in Language Models [107.82654101403264]
  Large-scale pretrained language models (LMs) can be potentially dangerous in manifesting undesirable representational biases. We propose steps towards mitigating social biases during text generation. Our empirical results and human evaluation demonstrate effectiveness in mitigating bias while retaining crucial contextual information.
  arXiv  Detail & Related papers  (2021-06-24T17:52:43Z)
• Electoral Forecasting Using a Novel Temporal Attenuation Model: Predicting the US Presidential Elections [91.3755431537592]
  We develop a novel macro-scale temporal attenuation (TA) model, which uses pre-election poll data to improve forecasting accuracy. Our hypothesis is that the timing of publicizing opinion polls plays a significant role in how opinion oscillates, especially right before elections. We present two different implementations of the TA model, which accumulate an average forecasting error of 2.8-3.28 points over the 48-year period.
  arXiv  Detail & Related papers  (2020-04-30T09:21:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.