Representation Bias in Political Sample Simulations with Large Language Models
- URL: http://arxiv.org/abs/2407.11409v1
- Date: Tue, 16 Jul 2024 05:52:26 GMT
- Title: Representation Bias in Political Sample Simulations with Large Language Models
- Authors: Weihong Qi, Hanjia Lyu, Jiebo Luo,
- Abstract summary: This study seeks to identify and quantify biases in simulating political samples with Large Language Models.
Using the GPT-3.5-Turbo model, we leverage data from the American National Election Studies, German Longitudinal Election Study, Zuobiao dataset, and China Family Panel Studies.
- Score: 54.48283690603358
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: This study seeks to identify and quantify biases in simulating political samples with Large Language Models, specifically focusing on vote choice and public opinion. Using the GPT-3.5-Turbo model, we leverage data from the American National Election Studies, German Longitudinal Election Study, Zuobiao Dataset, and China Family Panel Studies to simulate voting behaviors and public opinions. This methodology enables us to examine three types of representation bias: disparities based on the the country's language, demographic groups, and political regime types. The findings reveal that simulation performance is generally better for vote choice than for public opinions, more accurate in English-speaking countries, more effective in bipartisan systems than in multi-partisan systems, and stronger in democratic settings than in authoritarian regimes. These results contribute to enhancing our understanding and developing strategies to mitigate biases in AI applications within the field of computational social science.
Related papers
- Uncovering Political Bias in Emotion Inference Models: Implications for sentiment analysis in social science research [0.0]
This paper investigates the presence of political bias in machine learning models used for sentiment analysis (SA) in social science research.
We conducted a bias audit on a Polish sentiment analysis model developed in our lab.
Our findings indicate that annotations by human raters propagate political biases into the model's predictions.
arXiv Detail & Related papers (2024-07-18T20:31:07Z) - Lessons from the Trenches on Reproducible Evaluation of Language Models [60.522749986793094]
We draw on three years of experience in evaluating large language models to provide guidance and lessons for researchers.
We present the Language Model Evaluation Harness (lm-eval), an open source library for independent, reproducible, and evaluation of language models.
arXiv Detail & Related papers (2024-05-23T16:50:49Z) - Can LLMs Help Predict Elections? (Counter)Evidence from the World's Largest Democracy [3.0915192911449796]
The study of how social media affects the formation of public opinion and its influence on political results has been a popular field of inquiry.
We introduce a new method: harnessing the capabilities of Large Language Models (LLMs) to examine social media data and forecast election outcomes.
arXiv Detail & Related papers (2024-05-13T15:13:23Z) - Whose Side Are You On? Investigating the Political Stance of Large Language Models [56.883423489203786]
We investigate the political orientation of Large Language Models (LLMs) across a spectrum of eight polarizing topics.
Our investigation delves into the political alignment of LLMs across a spectrum of eight polarizing topics, spanning from abortion to LGBTQ issues.
The findings suggest that users should be mindful when crafting queries, and exercise caution in selecting neutral prompt language.
arXiv Detail & Related papers (2024-03-15T04:02:24Z) - The Face of Populism: Examining Differences in Facial Emotional
Expressions of Political Leaders Using Machine Learning [57.70351255180495]
We apply a deep-learning-based computer-vision algorithm to a sample of 220 YouTube videos depicting political leaders from 15 different countries.
We observe statistically significant differences in the average score of expressed negative emotions between groups of leaders with varying degrees of populist rhetoric.
arXiv Detail & Related papers (2023-04-19T18:32:49Z) - Fairly Allocating Utility in Constrained Multiwinner Elections [0.0]
A common denominator to ensure fairness across all such contexts is the use of constraints.
Across these contexts, the candidates selected to satisfy the given constraints may systematically lead to unfair outcomes for historically disadvantaged voter populations.
We develop a model to select candidates that satisfy the constraints fairly across voter populations.
arXiv Detail & Related papers (2022-11-23T10:04:26Z) - Out of One, Many: Using Language Models to Simulate Human Samples [3.278541277919869]
We show that the "algorithmic bias" within one such tool -- the GPT-3 language model -- is both fine-grained and demographically correlated.
We create "silicon samples" by conditioning the model on thousands of socio-demographic backstories from real human participants.
arXiv Detail & Related papers (2022-09-14T19:53:32Z) - Exploring Fairness in District-based Multi-party Elections under
different Voting Rules using Stochastic Simulations [0.5076419064097732]
Many democratic societies use district-based elections, where the region under consideration is geographically divided into districts and a representative is chosen for each district based on the preferences of the electors who reside there.
We show that this can lead to situations where many electors are dissatisfied with the election results, which is not desirable in a democracy.
Inspired by current literature on fairness of Machine Learning algorithms, we define measures of fairness to quantify the satisfaction of electors, irrespective of their political choices.
arXiv Detail & Related papers (2022-02-25T18:03:03Z) - Towards Understanding and Mitigating Social Biases in Language Models [107.82654101403264]
Large-scale pretrained language models (LMs) can be potentially dangerous in manifesting undesirable representational biases.
We propose steps towards mitigating social biases during text generation.
Our empirical results and human evaluation demonstrate effectiveness in mitigating bias while retaining crucial contextual information.
arXiv Detail & Related papers (2021-06-24T17:52:43Z) - Electoral Forecasting Using a Novel Temporal Attenuation Model:
Predicting the US Presidential Elections [91.3755431537592]
We develop a novel macro-scale temporal attenuation (TA) model, which uses pre-election poll data to improve forecasting accuracy.
Our hypothesis is that the timing of publicizing opinion polls plays a significant role in how opinion oscillates, especially right before elections.
We present two different implementations of the TA model, which accumulate an average forecasting error of 2.8-3.28 points over the 48-year period.
arXiv Detail & Related papers (2020-04-30T09:21:52Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.