Exploring Changes in Nation Perception with Nationality-Assigned Personas in LLMs

• URL: http://arxiv.org/abs/2406.13993v2
• Date: Wed, 16 Oct 2024 07:14:20 GMT
• Title: Exploring Changes in Nation Perception with Nationality-Assigned Personas in LLMs
• Abstract summary: We find that all LLM-persona combinations tend to favor Western European nations. Nation-personas push LLM behaviors to focus more on and treat the nation-persona's own region more favorably. Eastern European, Latin American, and African nations are treated more negatively by different nationality personas.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Persona assignment has become a common strategy for customizing LLM use to particular tasks and contexts. In this study, we explore how evaluation of different nations change when LLMs are assigned specific nationality personas. We assign 193 different nationality personas (e.g., an American person) to four LLMs and examine how the LLM evaluations (or ''perceptions'')of countries change. We find that all LLM-persona combinations tend to favor Western European nations, though nation-personas push LLM behaviors to focus more on and treat the nation-persona's own region more favorably. Eastern European, Latin American, and African nations are treated more negatively by different nationality personas. We additionally find that evaluations by nation-persona LLMs of other nations correlate with human survey responses but fail to match the values closely. Our study provides insight into how biases and stereotypes are realized within LLMs when adopting different national personas. In line with the ''Blueprint for an AI Bill of Rights'', our findings underscore the critical need for developing mechanisms to ensure that LLM outputs promote fairness and avoid over-generalization.

Related papers

• Representational Harms in LLM-Generated Narratives Against Global Majority Nationalities [1.5371856226502076]
  We study how national origin identities are portrayed by widely-adopted large language models (LLMs)<n>Our findings demonstrate the presence of persistent representational harms by national origin, including harmful stereotypes, erasure, and one-dimensional portrayals of Global Majority identities.<n>We find that the harms we identify cannot be explained away via sycophancy, as US-centric biases persist even when replacing US nationality cues with non-US national identities in the prompts.
  arXiv  Detail & Related papers  (2026-04-24T17:49:09Z)
• On the Alignment of Large Language Models with Global Human Opinion [36.1655217879788]
  This study is the first comprehensive investigation of the topic of opinion alignment in large language models (LLMs) across global, language, and temporal dimensions.<n>We create an evaluation framework based on the World Values Survey (WVS) to systematically assess the alignment of LLMs with human opinions.<n>We find that LLMs appropriately or over-align the opinions with only a few countries while under-aligning the opinions with most countries.
  arXiv  Detail & Related papers  (2025-09-01T12:19:17Z)
• An Evaluation of Cultural Value Alignment in LLM [27.437888319382893]
  We conduct the first large-scale evaluation of LLM culture assessing 20 countries' cultures and languages across ten LLMs. Our findings show that the output over all models represents a moderate cultural middle ground. Deeper investigation sheds light on the influence of model origin, prompt language, and value dimensions on cultural output.
  arXiv  Detail & Related papers  (2025-04-11T09:13:19Z)
• Sometimes the Model doth Preach: Quantifying Religious Bias in Open LLMs through Demographic Analysis in Asian Nations [8.769839351949997]
  Large Language Models (LLMs) are capable of generating opinions and propagating bias unknowingly. Our work proposes a novel method that quantitatively analyzes the opinions generated by LLMs. We evaluate modern, open LLMs such as Llama and Mistral on surveys conducted in various global south countries.
  arXiv  Detail & Related papers  (2025-03-10T16:32:03Z)
• Large Language Models Reflect the Ideology of their Creators [73.25935570218375]
  Large language models (LLMs) are trained on vast amounts of data to generate natural language. We uncover notable diversity in the ideological stance exhibited across different LLMs and languages.
  arXiv  Detail & Related papers  (2024-10-24T04:02:30Z)
• Hate Personified: Investigating the role of LLMs in content moderation [64.26243779985393]
  For subjective tasks such as hate detection, where people perceive hate differently, the Large Language Model's (LLM) ability to represent diverse groups is unclear. By including additional context in prompts, we analyze LLM's sensitivity to geographical priming, persona attributes, and numerical information to assess how well the needs of various groups are reflected.
  arXiv  Detail & Related papers  (2024-10-03T16:43:17Z)
• GermanPartiesQA: Benchmarking Commercial Large Language Models for Political Bias and Sycophancy [20.06753067241866]
  We evaluate and compare the alignment of six LLMs by OpenAI, Anthropic, and Cohere with German party positions. We conduct our prompt experiment for which we use the benchmark and sociodemographic data of leading German parliamentarians.
  arXiv  Detail & Related papers  (2024-07-25T13:04:25Z)
• How Well Do LLMs Represent Values Across Cultures? Empirical Analysis of LLM Responses Based on Hofstede Cultural Dimensions [9.275967682881944]
  It is critical to understand whether Large Language Models showcase different values to the user based on the stereotypical values of a user's known country. We prompt different LLMs with a series of advice requests based on 5 Hofstede Cultural Dimensions. We found that LLMs can differentiate between one side of a value and another, as well as understand that countries have differing values.
  arXiv  Detail & Related papers  (2024-06-21T00:58:01Z)
• CulturalTeaming: AI-Assisted Interactive Red-Teaming for Challenging LLMs' (Lack of) Multicultural Knowledge [69.82940934994333]
  We introduce CulturalTeaming, an interactive red-teaming system that leverages human-AI collaboration to build challenging evaluation dataset. Our study reveals that CulturalTeaming's various modes of AI assistance support annotators in creating cultural questions. CULTURALBENCH-V0.1 is a compact yet high-quality evaluation dataset with users' red-teaming attempts.
  arXiv  Detail & Related papers  (2024-04-10T00:25:09Z)
• OMGEval: An Open Multilingual Generative Evaluation Benchmark for Large Language Models [59.54423478596468]
  We introduce OMGEval, the first Open-source Multilingual Generative test set that can assess the capability of LLMs in different languages. For each language, OMGEval provides 804 open-ended questions, covering a wide range of important capabilities of LLMs. Specifically, the current version of OMGEval includes 5 languages (i.e., Zh, Ru, Fr, Es, Ar)
  arXiv  Detail & Related papers  (2024-02-21T04:42:41Z)
• Assessing LLMs for Moral Value Pluralism [2.860608352191896]
  We utilize a Recognizing Value Resonance (RVR) NLP model to identify World Values Survey (WVS) values that resonate and conflict with a given passage of text. We find that LLMs exhibit several Western-centric value biases. Our results highlight value misalignment and age groups, and a need for social science informed technological solutions.
  arXiv  Detail & Related papers  (2023-12-08T16:18:15Z)
• A Survey on Evaluation of Large Language Models [87.60417393701331]
  Large language models (LLMs) are gaining increasing popularity in both academia and industry. This paper focuses on three key dimensions: what to evaluate, where to evaluate, and how to evaluate.
  arXiv  Detail & Related papers  (2023-07-06T16:28:35Z)
• Heterogeneous Value Alignment Evaluation for Large Language Models [91.96728871418]
  Large Language Models (LLMs) have made it crucial to align their values with those of humans. We propose a Heterogeneous Value Alignment Evaluation (HVAE) system to assess the success of aligning LLMs with heterogeneous values.
  arXiv  Detail & Related papers  (2023-05-26T02:34:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.