Mapping and Influencing the Political Ideology of Large Language Models using Synthetic Personas

• URL: http://arxiv.org/abs/2412.14843v2
• Date: Fri, 20 Dec 2024 02:59:23 GMT
• Title: Mapping and Influencing the Political Ideology of Large Language Models using Synthetic Personas
• Authors: Pietro Bernardelle, Leon Fröhling, Stefano Civelli, Riccardo Lunardi, Kevin Roitero, Gianluca Demartini,
• Abstract summary: We map the political distribution of persona-based prompted large language models using the Political Compass Test (PCT) Our experiments reveal that synthetic personas predominantly cluster in the left-libertarian quadrant, with models demonstrating varying degrees of responsiveness when prompted with explicit ideological descriptors. While all models demonstrate significant shifts towards right-authoritarian positions, they exhibit more limited shifts towards left-libertarian positions, suggesting an asymmetric response to ideological manipulation that may reflect inherent biases in model training.
• Score: 5.237116285113809
• License:
• Abstract: The analysis of political biases in large language models (LLMs) has primarily examined these systems as single entities with fixed viewpoints. While various methods exist for measuring such biases, the impact of persona-based prompting on LLMs' political orientation remains unexplored. In this work we leverage PersonaHub, a collection of synthetic persona descriptions, to map the political distribution of persona-based prompted LLMs using the Political Compass Test (PCT). We then examine whether these initial compass distributions can be manipulated through explicit ideological prompting towards diametrically opposed political orientations: right-authoritarian and left-libertarian. Our experiments reveal that synthetic personas predominantly cluster in the left-libertarian quadrant, with models demonstrating varying degrees of responsiveness when prompted with explicit ideological descriptors. While all models demonstrate significant shifts towards right-authoritarian positions, they exhibit more limited shifts towards left-libertarian positions, suggesting an asymmetric response to ideological manipulation that may reflect inherent biases in model training.

Related papers

• Political-LLM: Large Language Models in Political Science [159.95299889946637]
  Large language models (LLMs) have been widely adopted in political science tasks. Political-LLM aims to advance the comprehensive understanding of integrating LLMs into computational political science.
  arXiv  Detail & Related papers  (2024-12-09T08:47:50Z)
• PRISM: A Methodology for Auditing Biases in Large Language Models [9.751718230639376]
  PRISM is a flexible, inquiry-based methodology for auditing Large Language Models. It seeks to illicit such positions indirectly through task-based inquiry prompting rather than direct inquiry of said preferences.
  arXiv  Detail & Related papers  (2024-10-24T16:57:20Z)
• Large Language Models Reflect the Ideology of their Creators [71.65505524599888]
  Large language models (LLMs) are trained on vast amounts of data to generate natural language. This paper shows that the ideological stance of an LLM appears to reflect the worldview of its creators.
  arXiv  Detail & Related papers  (2024-10-24T04:02:30Z)
• Representation Bias in Political Sample Simulations with Large Language Models [54.48283690603358]
  This study seeks to identify and quantify biases in simulating political samples with Large Language Models. Using the GPT-3.5-Turbo model, we leverage data from the American National Election Studies, German Longitudinal Election Study, Zuobiao dataset, and China Family Panel Studies.
  arXiv  Detail & Related papers  (2024-07-16T05:52:26Z)
• Whose Side Are You On? Investigating the Political Stance of Large Language Models [56.883423489203786]
  We investigate the political orientation of Large Language Models (LLMs) across a spectrum of eight polarizing topics. Our investigation delves into the political alignment of LLMs across a spectrum of eight polarizing topics, spanning from abortion to LGBTQ issues. The findings suggest that users should be mindful when crafting queries, and exercise caution in selecting neutral prompt language.
  arXiv  Detail & Related papers  (2024-03-15T04:02:24Z)
• Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models [61.45529177682614]
  We challenge the prevailing constrained evaluation paradigm for values and opinions in large language models. We show that models give substantively different answers when not forced. We distill these findings into recommendations and open challenges in evaluating values and opinions in LLMs.
  arXiv  Detail & Related papers  (2024-02-26T18:00:49Z)
• The Political Preferences of LLMs [0.0]
  I administer 11 political orientation tests, designed to identify the political preferences of the test taker, to 24 state-of-the-art conversational LLMs. Most conversational LLMs generate responses that are diagnosed by most political test instruments as manifesting preferences for left-of-center viewpoints. I demonstrate that LLMs can be steered towards specific locations in the political spectrum through Supervised Fine-Tuning.
  arXiv  Detail & Related papers  (2024-02-02T02:43:10Z)
• Inducing Political Bias Allows Language Models Anticipate Partisan Reactions to Controversies [5.958974943807783]
  This study addresses the challenge of understanding political bias in digitized discourse using Large Language Models (LLMs) We present a comprehensive analytical framework, consisting of Partisan Bias Divergence Assessment and Partisan Class Tendency Prediction. Our findings reveal the model's effectiveness in capturing emotional and moral nuances, albeit with some challenges in stance detection.
  arXiv  Detail & Related papers  (2023-11-16T08:57:53Z)
• Exploring the Jungle of Bias: Political Bias Attribution in Language Models via Dependency Analysis [86.49858739347412]
  Large Language Models (LLMs) have sparked intense debate regarding the prevalence of bias in these models and its mitigation. We propose a prompt-based method for the extraction of confounding and mediating attributes which contribute to the decision process. We find that the observed disparate treatment can at least in part be attributed to confounding and mitigating attributes and model misalignment.
  arXiv  Detail & Related papers  (2023-11-15T00:02:25Z)
• Examining Political Rhetoric with Epistemic Stance Detection [13.829628375546568]
  We develop a simple RoBERTa-based model for multi-source stance predictions that outperforms more complex state-of-the-art modeling. We demonstrate its novel application to political science by conducting a large-scale analysis of the Mass Market Manifestos corpus of U.S. political opinion books.
  arXiv  Detail & Related papers  (2022-12-29T23:47:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.