Analyzing Political Bias in LLMs via Target-Oriented Sentiment Classification

• URL: http://arxiv.org/abs/2505.19776v1
• Date: Mon, 26 May 2025 10:01:24 GMT
• Title: Analyzing Political Bias in LLMs via Target-Oriented Sentiment Classification
• Authors: Akram Elbouanani, Evan Dufraisse, Adrian Popescu,
• Abstract summary: Political biases encoded by LLMs might have detrimental effects on downstream applications.<n>We propose a new approach leveraging the observation that LLM sentiment predictions vary with the target entity in the same sentence.<n>We insert 1319 demographically and politically diverse politician names in 450 political sentences and predict target-oriented sentiment using seven models in six widely spoken languages.
• Score: 4.352835414206441
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Political biases encoded by LLMs might have detrimental effects on downstream applications. Existing bias analysis methods rely on small-size intermediate tasks (questionnaire answering or political content generation) and rely on the LLMs themselves for analysis, thus propagating bias. We propose a new approach leveraging the observation that LLM sentiment predictions vary with the target entity in the same sentence. We define an entropy-based inconsistency metric to encode this prediction variability. We insert 1319 demographically and politically diverse politician names in 450 political sentences and predict target-oriented sentiment using seven models in six widely spoken languages. We observe inconsistencies in all tested combinations and aggregate them in a statistically robust analysis at different granularity levels. We observe positive and negative bias toward left and far-right politicians and positive correlations between politicians with similar alignment. Bias intensity is higher for Western languages than for others. Larger models exhibit stronger and more consistent biases and reduce discrepancies between similar languages. We partially mitigate LLM unreliability in target-oriented sentiment classification (TSC) by replacing politician names with fictional but plausible counterparts.

Related papers

• Geopolitical biases in LLMs: what are the "good" and the "bad" countries according to contemporary language models [52.00270888041742]
  We introduce a novel dataset with neutral event descriptions and contrasting viewpoints from different countries.<n>Our findings show significant geopolitical biases, with models favoring specific national narratives.<n>Simple debiasing prompts had a limited effect on reducing these biases.
  arXiv  Detail & Related papers  (2025-06-07T10:45:17Z)
• Large Means Left: Political Bias in Large Language Models Increases with Their Number of Parameters [0.571853823214391]
  Large language models (LLMs) are predominantly used by many as a primary source of information for various topics.<n>LLMs frequently make factual errors, fabricate data (hallucinations), or present biases, exposing users to misinformation and influencing opinions.<n>We quantify the political bias of popular LLMs in the context of the recent vote of the German Bundestag using the score produced by the Wahl-O-Mat.
  arXiv  Detail & Related papers  (2025-05-07T13:18:41Z)
• Through the LLM Looking Glass: A Socratic Probing of Donkeys, Elephants, and Markets [42.55423041662188]
  The study aims to directly measure the models' biases rather than relying on external interpretations.<n>Our results reveal a consistent preference of Democratic over Republican positions across all models.<n>In economic topics, biases vary among Western LLMs, while those developed in China lean more strongly toward socialism.
  arXiv  Detail & Related papers  (2025-03-20T19:40:40Z)
• Beyond Partisan Leaning: A Comparative Analysis of Political Bias in Large Language Models [6.549047699071195]
  This study adopts a persona-free, topic-specific approach to evaluate political behavior in large language models.<n>We analyze responses from 43 large language models developed in the U.S., Europe, China, and the Middle East.<n>Findings show most models lean center-left or left ideologically and vary in their nonpartisan engagement patterns.
  arXiv  Detail & Related papers  (2024-12-21T19:42:40Z)
• Large Language Models Reflect the Ideology of their Creators [71.65505524599888]
  Large language models (LLMs) are trained on vast amounts of data to generate natural language.<n>This paper shows that the ideological stance of an LLM appears to reflect the worldview of its creators.
  arXiv  Detail & Related papers  (2024-10-24T04:02:30Z)
• Examining the Influence of Political Bias on Large Language Model Performance in Stance Classification [5.8229466650067065]
  We investigate whether large language models (LLMs) exhibit a tendency to more accurately classify politically-charged stances. Our findings reveal a statistically significant difference in the performance of LLMs across various politically oriented stance classification tasks. LLMs have poorer stance classification accuracy when there is greater ambiguity in the target the statement is directed towards.
  arXiv  Detail & Related papers  (2024-07-25T01:11:38Z)
• Evaluating Implicit Bias in Large Language Models by Attacking From a Psychometric Perspective [66.34066553400108]
  We conduct a rigorous evaluation of large language models' implicit bias towards certain demographics.<n>Inspired by psychometric principles, we propose three attack approaches, i.e., Disguise, Deception, and Teaching.<n>Our methods can elicit LLMs' inner bias more effectively than competitive baselines.
  arXiv  Detail & Related papers  (2024-06-20T06:42:08Z)
• Assessing Political Bias in Large Language Models [0.624709220163167]
  We evaluate the political bias of open-source Large Language Models (LLMs) concerning political issues within the European Union (EU) from a German voter's perspective. We show that larger models, such as Llama3-70B, tend to align more closely with left-leaning political parties, while smaller models often remain neutral.
  arXiv  Detail & Related papers  (2024-05-17T15:30:18Z)
• Measuring Political Bias in Large Language Models: What Is Said and How It Is Said [46.1845409187583]
  We propose to measure political bias in LLMs by analyzing both the content and style of their generated content regarding political issues. Our proposed measure looks at different political issues such as reproductive rights and climate change, at both the content (the substance of the generation) and the style (the lexical polarity) of such bias.
  arXiv  Detail & Related papers  (2024-03-27T18:22:48Z)
• Whose Side Are You On? Investigating the Political Stance of Large Language Models [56.883423489203786]
  We investigate the political orientation of Large Language Models (LLMs) across a spectrum of eight polarizing topics. Our investigation delves into the political alignment of LLMs across a spectrum of eight polarizing topics, spanning from abortion to LGBTQ issues. The findings suggest that users should be mindful when crafting queries, and exercise caution in selecting neutral prompt language.
  arXiv  Detail & Related papers  (2024-03-15T04:02:24Z)
• Exploring the Jungle of Bias: Political Bias Attribution in Language Models via Dependency Analysis [86.49858739347412]
  Large Language Models (LLMs) have sparked intense debate regarding the prevalence of bias in these models and its mitigation. We propose a prompt-based method for the extraction of confounding and mediating attributes which contribute to the decision process. We find that the observed disparate treatment can at least in part be attributed to confounding and mitigating attributes and model misalignment.
  arXiv  Detail & Related papers  (2023-11-15T00:02:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.