Navigating the digital spectrum: Assessing political bias, stability, and downstream fairness in Large Language Models

• URL: http://arxiv.org/abs/2609.08637v1
• Date: Tue, 08 Sep 2026 12:07:30 GMT
• Title: Navigating the digital spectrum: Assessing political bias, stability, and downstream fairness in Large Language Models
• Abstract summary: We evaluate eight Gemma 3 and Qwen 3 models across 14 languages and three quantization levels.<n>Most models lean Libertarian-Left on average, but instruction phrasing, language, and answer format significantly affect recovered coordinates.<n>Reverse-engineering the test exposes axis-weighting imbalances and the collapse of degenerate responses toward the center.
• Score: 3.937573365677865
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models are increasingly deployed as information intermediaries, yet measuring their political behavior remains fragile because questionnaire results mix model dispositions with measurement artifacts and response-elicitation biases. We introduce a robust Political Compass Test evaluation framework that samples 300 configurations across an eight-dimensional perturbation space varying language, framing, instructions, answer format, option order, and persona wording. We evaluate eight Gemma 3 and Qwen 3 models across 14 languages and three quantization levels, obtaining design-averaged political coordinates with quantified uncertainty. Most models lean Libertarian-Left on average, but instruction phrasing, language, and answer format significantly affect recovered coordinates. Cross-lingual differences primarily reflect coordinate drift rather than distinct cultural reasoning. Reverse-engineering the test also exposes axis-weighting imbalances and the collapse of degenerate responses toward the center, so near-origin estimates for the smallest models can reflect weak signal rather than centrism. Free-text reasoning and chat-then-classify elicitation alter recovered coordinates, and larger models show clearer persona separation, with a specific failure of the Authoritarian-Left persona to move most models in the intended social direction. In downstream tasks, persona effects are modest relative to model size and target group for hate-speech detection, while base and centrist prompts give the highest agreement for topic-level sentiment. Political role prompting therefore has measurable but task- and dataset-specific downstream effects.

Related papers

• IndicQE-APE: A Benchmark for Quality Estimation and Automatic Post-Editing for Indic Languages [77.88427845181154]
  We consolidate the WMT 2020--2024 shared-task lineage with an extended English--Malayalam resource into indicqe: $126,754$ instances over nine directional pairs.<n>We benchmark six prompted LLMs and three COMET metrics on segment-level QE, and three systems on APE.
  arXiv  Detail & Related papers  (2026-08-17T09:50:52Z)
• Auditing Alignment Controllability in LLMs via Political Axes [0.0]
  Political audits usually reduce each to one point on a political compass.<n>What counts is how far, and in which directions, a model's answers can be steered.<n>We run a dispersion-first stress test of prompt-based controllability across 12 ideological personas.
  arXiv  Detail & Related papers  (2026-07-26T07:38:11Z)
• Attribute-Based Diagnosis of LLM Alignment with Hate Speech Annotations [48.69228180369574]
  Hate speech annotation is costly, subjective, and prone to annotator disagreement.<n>We analyze how well large language models (LLMs) align with human judgments.<n>We reconstruct continuous hate speech scores from the Measuring Hate Speech corpus.
  arXiv  Detail & Related papers  (2026-05-26T13:44:48Z)
• Political Alignment in Large Language Models: A Multidimensional Audit of Psychometric Identity and Behavioral Bias [2.464003792743989]
  This study presents a sociotechnical audit of 26 prominent large language models (LLMs)<n> Alignment signals appear to be consistent architectural traits rather than noise.<n>Models exhibit a systematic "center-shift," frequently categorizing neutral articles as left-leaning.
  arXiv  Detail & Related papers  (2026-01-08T03:26:15Z)
• Bridging Human and Model Perspectives: A Comparative Analysis of Political Bias Detection in News Media Using Large Language Models [0.3227658251731014]
  This study aims to present a comparative framework for evaluating the detection of political bias across human annotations and multiple language models.<n>We construct a manually annotated dataset of news articles and assess annotation consistency, bias polarity, and inter-model agreement.<n> Experimental results show that among traditional transformer-based models, RoBERTa achieves the highest alignment with human labels.
  arXiv  Detail & Related papers  (2025-11-18T15:58:04Z)
• Are Stereotypes Leading LLMs' Zero-Shot Stance Detection ? [4.861653297482551]
  Large Language Models inherit stereotypes from their pretraining data, leading to biased behavior toward certain social groups.<n>In this paper, we focus on the bias of Large Language Models when performing stance detection in a zero-shot setting.<n>We show that LLMs exhibit significant stereotypes in stance detection tasks, such as incorrectly associating pro-marijuana views with low text complexity and African American dialect with opposition to Donald Trump.
  arXiv  Detail & Related papers  (2025-10-23T03:05:25Z)
• Multilingual Political Views of Large Language Models: Identification and Steering [11.071018930042909]
  Large language models (LLMs) are increasingly used in everyday tools and applications, raising concerns about their potential influence on political views.<n>We evaluate seven models across 14 languages using the Political Compass Test with 11 semantically equivalent paraphrases per statement to ensure robust measurement.<n>Our results reveal that larger models consistently shift toward libertarian-left positions, with significant variations across languages and model families.
  arXiv  Detail & Related papers  (2025-07-30T12:42:35Z)
• Spoken Stereoset: On Evaluating Social Bias Toward Speaker in Speech Large Language Models [50.40276881893513]
  This study introduces Spoken Stereoset, a dataset specifically designed to evaluate social biases in Speech Large Language Models (SLLMs) By examining how different models respond to speech from diverse demographic groups, we aim to identify these biases. The findings indicate that while most models show minimal bias, some still exhibit slightly stereotypical or anti-stereotypical tendencies.
  arXiv  Detail & Related papers  (2024-08-14T16:55:06Z)
• Analyzing the Limits of Self-Supervision in Handling Bias in Language [52.26068057260399]
  We evaluate how well language models capture the semantics of four tasks for bias: diagnosis, identification, extraction and rephrasing. Our analyses indicate that language models are capable of performing these tasks to widely varying degrees across different bias dimensions, such as gender and political affiliation.
  arXiv  Detail & Related papers  (2021-12-16T05:36:08Z)
• Few-shot Instruction Prompts for Pretrained Language Models to Detect Social Biases [55.45617404586874]
  We propose a few-shot instruction-based method for prompting pre-trained language models (LMs) We show that large LMs can detect different types of fine-grained biases with similar and sometimes superior accuracy to fine-tuned models.
  arXiv  Detail & Related papers  (2021-12-15T04:19:52Z)
• AES Systems Are Both Overstable And Oversensitive: Explaining Why And Proposing Defenses [66.49753193098356]
  We investigate the reason behind the surprising adversarial brittleness of scoring models. Our results indicate that autoscoring models, despite getting trained as "end-to-end" models, behave like bag-of-words models. We propose detection-based protection models that can detect oversensitivity and overstability causing samples with high accuracies.
  arXiv  Detail & Related papers  (2021-09-24T03:49:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.