Through the LLM Looking Glass: A Socratic Self-Assessment of Donkeys, Elephants, and Markets

• URL: http://arxiv.org/abs/2503.16674v1
• Date: Thu, 20 Mar 2025 19:40:40 GMT
• Title: Through the LLM Looking Glass: A Socratic Self-Assessment of Donkeys, Elephants, and Markets
• Authors: Molly Kennedy, Ayyoob Imani, Timo Spinde, Hinrich Schütze,
• Abstract summary: The study aims to directly measure the models' biases rather than relying on external interpretations.<n>Our results reveal a consistent preference of Democratic over Republican positions across all models.<n> biases vary among Western LLMs, while those developed in China lean more strongly toward socialism.
• Score: 42.55423041662188
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While detecting and avoiding bias in LLM-generated text is becoming increasingly important, media bias often remains subtle and subjective, making it particularly difficult to identify and mitigate. In this study, we assess media bias in LLM-generated content and LLMs' ability to detect subtle ideological bias. We conduct this evaluation using two datasets, PoliGen and EconoLex, covering political and economic discourse, respectively. We evaluate eight widely used LLMs by prompting them to generate articles and analyze their ideological preferences via self-assessment. By using self-assessment, the study aims to directly measure the models' biases rather than relying on external interpretations, thereby minimizing subjective judgments about media bias. Our results reveal a consistent preference of Democratic over Republican positions across all models. Conversely, in economic topics, biases vary among Western LLMs, while those developed in China lean more strongly toward socialism.

Related papers

• Unpacking Political Bias in Large Language Models: A Cross-Model Comparison on U.S. Politics [6.253258189994455]
  Political bias, as a universal phenomenon in human society, may be transferred to Large Language Models.<n>Political biases evolve with model scale and release date, and are also influenced by regional factors of LLMs.
  arXiv  Detail & Related papers  (2024-12-21T19:42:40Z)
• Large Language Models Reflect the Ideology of their Creators [71.65505524599888]
  Large language models (LLMs) are trained on vast amounts of data to generate natural language.<n>This paper shows that the ideological stance of an LLM appears to reflect the worldview of its creators.
  arXiv  Detail & Related papers  (2024-10-24T04:02:30Z)
• Bias in the Mirror: Are LLMs opinions robust to their own adversarial attacks ? [22.0383367888756]
  Large language models (LLMs) inherit biases from their training data and alignment processes, influencing their responses in subtle ways. We introduce a novel approach where two instances of an LLM engage in self-debate, arguing opposing viewpoints to persuade a neutral version of the model. We evaluate how firmly biases hold and whether models are susceptible to reinforcing misinformation or shifting to harmful viewpoints.
  arXiv  Detail & Related papers  (2024-10-17T13:06:02Z)
• Bias in LLMs as Annotators: The Effect of Party Cues on Labelling Decision by Large Language Models [0.0]
  We test similar biases in Large Language Models (LLMs) as annotators. Unlike humans, who are only biased when faced with statements from extreme parties, LLMs exhibit significant bias even when prompted with statements from center-left and center-right parties.
  arXiv  Detail & Related papers  (2024-08-28T16:05:20Z)
• Evaluating Implicit Bias in Large Language Models by Attacking From a Psychometric Perspective [66.34066553400108]
  We conduct a rigorous evaluation of large language models' implicit bias towards certain demographics. Inspired by psychometric principles, we propose three attack approaches, i.e., Disguise, Deception, and Teaching. Our methods can elicit LLMs' inner bias more effectively than competitive baselines.
  arXiv  Detail & Related papers  (2024-06-20T06:42:08Z)
• Assessing Political Bias in Large Language Models [0.624709220163167]
  We evaluate the political bias of open-source Large Language Models (LLMs) concerning political issues within the European Union (EU) from a German voter's perspective. We show that larger models, such as Llama3-70B, tend to align more closely with left-leaning political parties, while smaller models often remain neutral.
  arXiv  Detail & Related papers  (2024-05-17T15:30:18Z)
• Investigating Bias in LLM-Based Bias Detection: Disparities between LLMs and Human Perception [13.592532358127293]
  We investigate the presence and nature of bias within Large Language Models (LLMs)<n>We probe whether LLMs exhibit biases, particularly in political bias prediction and text continuation tasks.<n>We propose debiasing strategies, including prompt engineering and model fine-tuning.
  arXiv  Detail & Related papers  (2024-03-22T00:59:48Z)
• Whose Side Are You On? Investigating the Political Stance of Large Language Models [56.883423489203786]
  We investigate the political orientation of Large Language Models (LLMs) across a spectrum of eight polarizing topics. Our investigation delves into the political alignment of LLMs across a spectrum of eight polarizing topics, spanning from abortion to LGBTQ issues. The findings suggest that users should be mindful when crafting queries, and exercise caution in selecting neutral prompt language.
  arXiv  Detail & Related papers  (2024-03-15T04:02:24Z)
• Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models [61.45529177682614]
  We challenge the prevailing constrained evaluation paradigm for values and opinions in large language models. We show that models give substantively different answers when not forced. We distill these findings into recommendations and open challenges in evaluating values and opinions in LLMs.
  arXiv  Detail & Related papers  (2024-02-26T18:00:49Z)
• Large Language Models are Geographically Biased [47.88767211956144]
  We study what Large Language Models (LLMs) know about the world we live in through the lens of geography. We show various problematic geographic biases, which we define as systemic errors in geospatial predictions.
  arXiv  Detail & Related papers  (2024-02-05T02:32:09Z)
• Exploring the Jungle of Bias: Political Bias Attribution in Language Models via Dependency Analysis [86.49858739347412]
  Large Language Models (LLMs) have sparked intense debate regarding the prevalence of bias in these models and its mitigation. We propose a prompt-based method for the extraction of confounding and mediating attributes which contribute to the decision process. We find that the observed disparate treatment can at least in part be attributed to confounding and mitigating attributes and model misalignment.
  arXiv  Detail & Related papers  (2023-11-15T00:02:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.