Poli-Bias: Understanding and Measuring Large Language Model Biases in International Political Conflicts

• URL: http://arxiv.org/abs/2608.06123v2
• Date: Mon, 10 Aug 2026 14:43:02 GMT
• Title: Poli-Bias: Understanding and Measuring Large Language Model Biases in International Political Conflicts
• Abstract summary: Poli-Bias is a framework for measuring whether large language models treat legally equivalent conflict scenarios differently depending on the countries involved.<n>We find that country identities and user affiliations can systematically affect how equivalent actions are described, evaluated, and defended under international law.
• Score: 33.852036995882465
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Measuring political bias in large language models (LLMs) remains challenging as it can manifest through subtle differences in framing, argumentation, and legal reasoning that are difficult to capture with a single metric. In this work, we introduce Poli-Bias, a counterfactual framework for measuring whether LLMs treat legally equivalent conflict scenarios differently depending on the countries involved. Poli-Bias compares responses to paired prompts in which country identities are systematically swapped across diverse geopolitical relationships, legal violations, and reasoning tasks. Rather than reducing bias to a single judgment, our framework decomposes response disparities into five interpretable dimensions, revealing how and where unequal treatment manifests. Across 13 contemporary LLMs spanning diverse model families and sizes, we find that country identities and user affiliations can systematically affect how equivalent actions are described, evaluated, and defended under international law. Our results thus establish Poli-Bias as a fine-grained framework for auditing political even-handedness and sycophancy in LLMs.

Related papers

• Who Would You Vote For? Auditing Political Alignment in LLMs: An Italian Case-Study [0.0]
  We investigate whether and how Large Language Models express preferences toward political parties and political leaders.<n>Rather than attempting to infer the models' "true" political beliefs, we focus on their observable behavior.
  arXiv  Detail & Related papers  (2026-08-12T04:47:06Z)
• Stability and Political Orientation of International LLMs: An Exploratory Multi-Run Study Conducted in French [0.0]
  This study explores the stability and ideological orientation of political responses produced by various large language models (LLMs) in French.<n>Eleven models from various organisations and countries were tested, each subjected to twenty independent runs to assess intra-model variability.<n>The results show that, despite a general degree of stability, significant variations appear from one run to the next and between models.
  arXiv  Detail & Related papers  (2026-06-11T17:18:38Z)
• A Multi-Dimensional Audit of Politically Aligned Large Language Models [3.147897243459717]
  This work aims to ensure politically-aligned language models generate legitimate, harmless arguments.<n>We propose a multi-dimensional framework inspired by Habermas' Theory of Communicative Action to audit politically aligned language models.
  arXiv  Detail & Related papers  (2026-04-27T12:57:52Z)
• PolicyLLM: Towards Excellent Comprehension of Public Policy for Large Language Models [56.37206340175191]
  Large Language Models (LLMs) are increasingly integrated into real-world decision-making.<n>Yet, their ability to comprehend and reason about policy-related content remains underexplored.<n>We present textbftextitPolicyBench, the first large-scale cross-system benchmark (US-China) evaluating policy comprehension.
  arXiv  Detail & Related papers  (2026-04-14T17:27:50Z)
• Uncovering Political Bias in Large Language Models using Parliamentary Voting Records [2.272052150526026]
  This paper introduces a general methodology for constructing political bias benchmarks.<n>We instantiate this methodology in three national case studies.<n>We assess ideological tendencies and political entity bias in LLM behavior.
  arXiv  Detail & Related papers  (2026-01-13T18:18:25Z)
• Geopolitical biases in LLMs: what are the "good" and the "bad" countries according to contemporary language models [52.00270888041742]
  We introduce a novel dataset with neutral event descriptions and contrasting viewpoints from different countries.<n>Our findings show significant geopolitical biases, with models favoring specific national narratives.<n>Simple debiasing prompts had a limited effect on reducing these biases.
  arXiv  Detail & Related papers  (2025-06-07T10:45:17Z)
• Beyond Partisan Leaning: A Comparative Analysis of Political Bias in Large Language Models [6.549047699071195]
  This study adopts a persona-free, topic-specific approach to evaluate political behavior in large language models.<n>We analyze responses from 43 large language models developed in the U.S., Europe, China, and the Middle East.<n>Findings show most models lean center-left or left ideologically and vary in their nonpartisan engagement patterns.
  arXiv  Detail & Related papers  (2024-12-21T19:42:40Z)
• Large Language Models Reflect the Ideology of their Creators [71.65505524599888]
  Large language models (LLMs) are trained on vast amounts of data to generate natural language.<n>This paper shows that the ideological stance of an LLM appears to reflect the worldview of its creators.
  arXiv  Detail & Related papers  (2024-10-24T04:02:30Z)
• Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models [61.45529177682614]
  We challenge the prevailing constrained evaluation paradigm for values and opinions in large language models. We show that models give substantively different answers when not forced. We distill these findings into recommendations and open challenges in evaluating values and opinions in LLMs.
  arXiv  Detail & Related papers  (2024-02-26T18:00:49Z)
• Exploring the Jungle of Bias: Political Bias Attribution in Language Models via Dependency Analysis [86.49858739347412]
  Large Language Models (LLMs) have sparked intense debate regarding the prevalence of bias in these models and its mitigation. We propose a prompt-based method for the extraction of confounding and mediating attributes which contribute to the decision process. We find that the observed disparate treatment can at least in part be attributed to confounding and mitigating attributes and model misalignment.
  arXiv  Detail & Related papers  (2023-11-15T00:02:25Z)
• This Land is {Your, My} Land: Evaluating Geopolitical Biases in Language Models [40.61046400448044]
  We show that large language models (LLM) recall certain geographical knowledge inconsistently when queried in different languages. As a targeted case study, we consider territorial disputes, an inherently controversial and multilingual task. We propose a suite of evaluation metrics to precisely quantify bias and consistency in responses across different languages.
  arXiv  Detail & Related papers  (2023-05-24T01:16:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.