How Well Do LLMs Represent Values Across Cultures? Empirical Analysis of LLM Responses Based on Hofstede Cultural Dimensions

• URL: http://arxiv.org/abs/2406.14805v1
• Date: Fri, 21 Jun 2024 00:58:01 GMT
• Title: How Well Do LLMs Represent Values Across Cultures? Empirical Analysis of LLM Responses Based on Hofstede Cultural Dimensions
• Authors: Julia Kharchenko, Tanya Roosta, Aman Chadha, Chirag Shah,
• Abstract summary: It is critical to understand whether Large Language Models showcase different values to the user based on the stereotypical values of a user's known country. We prompt different LLMs with a series of advice requests based on 5 Hofstede Cultural Dimensions. We found that LLMs can differentiate between one side of a value and another, as well as understand that countries have differing values.
• Score: 9.275967682881944
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Large Language Models (LLMs) attempt to imitate human behavior by responding to humans in a way that pleases them, including by adhering to their values. However, humans come from diverse cultures with different values. It is critical to understand whether LLMs showcase different values to the user based on the stereotypical values of a user's known country. We prompt different LLMs with a series of advice requests based on 5 Hofstede Cultural Dimensions -- a quantifiable way of representing the values of a country. Throughout each prompt, we incorporate personas representing 36 different countries and, separately, languages predominantly tied to each country to analyze the consistency in the LLMs' cultural understanding. Through our analysis of the responses, we found that LLMs can differentiate between one side of a value and another, as well as understand that countries have differing values, but will not always uphold the values when giving advice, and fail to understand the need to answer differently based on different cultural values. Rooted in these findings, we present recommendations for training value-aligned and culturally sensitive LLMs. More importantly, the methodology and the framework developed here can help further understand and mitigate culture and language alignment issues with LLMs.

Related papers

• Do LLMs have Consistent Values? [27.09321034836509]
  We study the values exhibited in text generated by Large Language Models (LLM) We ask whether LLMs exhibit the same value structure that has been demonstrated in humans, including the ranking of values, and correlation between values.
  arXiv  Detail & Related papers  (2024-07-16T08:58:00Z)
• Multilingual Trolley Problems for Language Models [138.0995992619116]
  This study is inspired by a large-scale cross-cultural study of human moral preferences, "The Moral Machine Experiment" We show that large language models (LLMs) are more aligned with human preferences in languages such as English, Korean, Hungarian, and Chinese, but less aligned in languages such as Hindi and Somali (in Africa) We also characterize the explanations LLMs give for their moral choices and find that fairness is the most dominant supporting reason behind GPT-4's decisions and utilitarianism by GPT-3.
  arXiv  Detail & Related papers  (2024-07-02T14:02:53Z)
• Translating Across Cultures: LLMs for Intralingual Cultural Adaptation [12.5954253354303]
  We define the task of cultural adaptation and create an evaluation framework to benchmark different models for this task. We analyze possible issues with automatic adaptation including cultural biases and stereotypes.
  arXiv  Detail & Related papers  (2024-06-20T17:06:58Z)
• CIVICS: Building a Dataset for Examining Culturally-Informed Values in Large Language Models [59.22460740026037]
  "CIVICS: Culturally-Informed & Values-Inclusive Corpus for Societal impacts" dataset is designed to evaluate the social and cultural variation of Large Language Models (LLMs) We create a hand-crafted, multilingual dataset of value-laden prompts which address specific socially sensitive topics, including LGBTQI rights, social welfare, immigration, disability rights, and surrogacy.
  arXiv  Detail & Related papers  (2024-05-22T20:19:10Z)
• Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsense [98.09670425244462]
  Large language models (LLMs) have demonstrated substantial commonsense understanding. This paper examines the capabilities and limitations of several state-of-the-art LLMs in the context of cultural commonsense tasks.
  arXiv  Detail & Related papers  (2024-05-07T20:28:34Z)
• CULTURE-GEN: Revealing Global Cultural Perception in Language Models through Natural Language Prompting [68.37589899302161]
  We uncover culture perceptions of three SOTA models on 110 countries and regions on 8 culture-related topics through culture-conditioned generations. We discover that culture-conditioned generation consist of linguistic "markers" that distinguish marginalized cultures apart from default cultures.
  arXiv  Detail & Related papers  (2024-04-16T00:50:43Z)
• CulturalTeaming: AI-Assisted Interactive Red-Teaming for Challenging LLMs' (Lack of) Multicultural Knowledge [69.82940934994333]
  We introduce CulturalTeaming, an interactive red-teaming system that leverages human-AI collaboration to build challenging evaluation dataset. Our study reveals that CulturalTeaming's various modes of AI assistance support annotators in creating cultural questions. CULTURALBENCH-V0.1 is a compact yet high-quality evaluation dataset with users' red-teaming attempts.
  arXiv  Detail & Related papers  (2024-04-10T00:25:09Z)
• Assessing LLMs for Moral Value Pluralism [2.860608352191896]
  We utilize a Recognizing Value Resonance (RVR) NLP model to identify World Values Survey (WVS) values that resonate and conflict with a given passage of text. We find that LLMs exhibit several Western-centric value biases. Our results highlight value misalignment and age groups, and a need for social science informed technological solutions.
  arXiv  Detail & Related papers  (2023-12-08T16:18:15Z)
• Cultural Alignment in Large Language Models: An Explanatory Analysis Based on Hofstede's Cultural Dimensions [10.415002561977655]
  This research proposes a Cultural Alignment Test (Hoftede's CAT) to quantify cultural alignment using Hofstede's cultural dimension framework. We quantitatively evaluate large language models (LLMs) against the cultural dimensions of regions like the United States, China, and Arab countries. Our results quantify the cultural alignment of LLMs and reveal the difference between LLMs in explanatory cultural dimensions.
  arXiv  Detail & Related papers  (2023-08-25T14:50:13Z)
• Heterogeneous Value Alignment Evaluation for Large Language Models [91.96728871418]
  Large Language Models (LLMs) have made it crucial to align their values with those of humans. We propose a Heterogeneous Value Alignment Evaluation (HVAE) system to assess the success of aligning LLMs with heterogeneous values.
  arXiv  Detail & Related papers  (2023-05-26T02:34:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.