PolBiX: Detecting LLMs' Political Bias in Fact-Checking through X-phemisms

• URL: http://arxiv.org/abs/2509.15335v2
• Date: Tue, 23 Sep 2025 11:42:25 GMT
• Title: PolBiX: Detecting LLMs' Political Bias in Fact-Checking through X-phemisms
• Authors: Charlott Jakob, David Harbecke, Patrick Parschan, Pia Wenzel Neves, Vera Schmitt,
• Abstract summary: We investigate political bias through exchanging words with euphemisms or dysphemisms in German claims.<n>We find that, more than political leaning, the presence of judgmental words significantly influences truthfulness assessment.
• Score: 1.0439136407307046
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Large Language Models are increasingly used in applications requiring objective assessment, which could be compromised by political bias. Many studies found preferences for left-leaning positions in LLMs, but downstream effects on tasks like fact-checking remain underexplored. In this study, we systematically investigate political bias through exchanging words with euphemisms or dysphemisms in German claims. We construct minimal pairs of factually equivalent claims that differ in political connotation, to assess the consistency of LLMs in classifying them as true or false. We evaluate six LLMs and find that, more than political leaning, the presence of judgmental words significantly influences truthfulness assessment. While a few models show tendencies of political bias, this is not mitigated by explicitly calling for objectivism in prompts. Warning: This paper contains content that may be offensive or upsetting.

Related papers

• Geopolitical biases in LLMs: what are the "good" and the "bad" countries according to contemporary language models [52.00270888041742]
  We introduce a novel dataset with neutral event descriptions and contrasting viewpoints from different countries.<n>Our findings show significant geopolitical biases, with models favoring specific national narratives.<n>Simple debiasing prompts had a limited effect on reducing these biases.
  arXiv  Detail & Related papers  (2025-06-07T10:45:17Z)
• LLMs Struggle to Reject False Presuppositions when Misinformation Stakes are High [7.9042053398943075]
  Presuppositions subtly introduce information as given, making them highly effective at embedding disputable or false information.<n>This raises concerns about whether LLMs, like humans, may fail to detect and correct misleading assumptions introduced as false presuppositions.
  arXiv  Detail & Related papers  (2025-05-28T13:35:07Z)
• Analyzing Political Bias in LLMs via Target-Oriented Sentiment Classification [4.352835414206441]
  Political biases encoded by LLMs might have detrimental effects on downstream applications.<n>We propose a new approach leveraging the observation that LLM sentiment predictions vary with the target entity in the same sentence.<n>We insert 1319 demographically and politically diverse politician names in 450 political sentences and predict target-oriented sentiment using seven models in six widely spoken languages.
  arXiv  Detail & Related papers  (2025-05-26T10:01:24Z)
• Large Means Left: Political Bias in Large Language Models Increases with Their Number of Parameters [0.571853823214391]
  Large language models (LLMs) are predominantly used by many as a primary source of information for various topics.<n>LLMs frequently make factual errors, fabricate data (hallucinations), or present biases, exposing users to misinformation and influencing opinions.<n>We quantify the political bias of popular LLMs in the context of the recent vote of the German Bundestag using the score produced by the Wahl-O-Mat.
  arXiv  Detail & Related papers  (2025-05-07T13:18:41Z)
• Through the LLM Looking Glass: A Socratic Probing of Donkeys, Elephants, and Markets [42.55423041662188]
  The study aims to directly measure the models' biases rather than relying on external interpretations.<n>Our results reveal a consistent preference of Democratic over Republican positions across all models.<n>In economic topics, biases vary among Western LLMs, while those developed in China lean more strongly toward socialism.
  arXiv  Detail & Related papers  (2025-03-20T19:40:40Z)
• Large Language Models Reflect the Ideology of their Creators [71.65505524599888]
  Large language models (LLMs) are trained on vast amounts of data to generate natural language.<n>This paper shows that the ideological stance of an LLM appears to reflect the worldview of its creators.
  arXiv  Detail & Related papers  (2024-10-24T04:02:30Z)
• Whose Side Are You On? Investigating the Political Stance of Large Language Models [56.883423489203786]
  We investigate the political orientation of Large Language Models (LLMs) across a spectrum of eight polarizing topics. Our investigation delves into the political alignment of LLMs across a spectrum of eight polarizing topics, spanning from abortion to LGBTQ issues. The findings suggest that users should be mindful when crafting queries, and exercise caution in selecting neutral prompt language.
  arXiv  Detail & Related papers  (2024-03-15T04:02:24Z)
• Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models [61.45529177682614]
  We challenge the prevailing constrained evaluation paradigm for values and opinions in large language models. We show that models give substantively different answers when not forced. We distill these findings into recommendations and open challenges in evaluating values and opinions in LLMs.
  arXiv  Detail & Related papers  (2024-02-26T18:00:49Z)
• The Political Preferences of LLMs [0.0]
  I administer 11 political orientation tests, designed to identify the political preferences of the test taker, to 24 state-of-the-art conversational LLMs. Most conversational LLMs generate responses that are diagnosed by most political test instruments as manifesting preferences for left-of-center viewpoints. I demonstrate that LLMs can be steered towards specific locations in the political spectrum through Supervised Fine-Tuning.
  arXiv  Detail & Related papers  (2024-02-02T02:43:10Z)
• What Do Llamas Really Think? Revealing Preference Biases in Language Model Representations [62.91799637259657]
  Do large language models (LLMs) exhibit sociodemographic biases, even when they decline to respond? We study this research question by probing contextualized embeddings and exploring whether this bias is encoded in its latent representations. We propose a logistic Bradley-Terry probe which predicts word pair preferences of LLMs from the words' hidden vectors.
  arXiv  Detail & Related papers  (2023-11-30T18:53:13Z)
• Exploring the Jungle of Bias: Political Bias Attribution in Language Models via Dependency Analysis [86.49858739347412]
  Large Language Models (LLMs) have sparked intense debate regarding the prevalence of bias in these models and its mitigation. We propose a prompt-based method for the extraction of confounding and mediating attributes which contribute to the decision process. We find that the observed disparate treatment can at least in part be attributed to confounding and mitigating attributes and model misalignment.
  arXiv  Detail & Related papers  (2023-11-15T00:02:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.