An Evaluation of Cultural Value Alignment in LLM

• URL: http://arxiv.org/abs/2504.08863v1
• Date: Fri, 11 Apr 2025 09:13:19 GMT
• Title: An Evaluation of Cultural Value Alignment in LLM
• Authors: Nicholas Sukiennik, Chen Gao, Fengli Xu, Yong Li,
• Abstract summary: We conduct the first large-scale evaluation of LLM culture assessing 20 countries' cultures and languages across ten LLMs.<n>Our findings show that the output over all models represents a moderate cultural middle ground.<n> Deeper investigation sheds light on the influence of model origin, prompt language, and value dimensions on cultural output.
• Score: 27.437888319382893
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLMs as intelligent agents are being increasingly applied in scenarios where human interactions are involved, leading to a critical concern about whether LLMs are faithful to the variations in culture across regions. Several works have investigated this question in various ways, finding that there are biases present in the cultural representations of LLM outputs. To gain a more comprehensive view, in this work, we conduct the first large-scale evaluation of LLM culture assessing 20 countries' cultures and languages across ten LLMs. With a renowned cultural values questionnaire and by carefully analyzing LLM output with human ground truth scores, we thoroughly study LLMs' cultural alignment across countries and among individual models. Our findings show that the output over all models represents a moderate cultural middle ground. Given the overall skew, we propose an alignment metric, revealing that the United States is the best-aligned country and GLM-4 has the best ability to align to cultural values. Deeper investigation sheds light on the influence of model origin, prompt language, and value dimensions on cultural output. Specifically, models, regardless of where they originate, align better with the US than they do with China. The conclusions provide insight to how LLMs can be better aligned to various cultures as well as provoke further discussion of the potential for LLMs to propagate cultural bias and the need for more culturally adaptable models.

Related papers

• CAReDiO: Cultural Alignment of LLM via Representativeness and Distinctiveness Guided Data Optimization [50.90288681622152]
  Large Language Models (LLMs) more deeply integrate into human life across various regions. Existing approaches develop culturally aligned LLMs through fine-tuning with culture-specific corpora. We introduce CAReDiO, a novel cultural data construction framework.
  arXiv  Detail & Related papers  (2025-04-09T13:40:13Z)
• Cultural Learning-Based Culture Adaptation of Language Models [70.1063219524999]
  Adapting large language models (LLMs) to diverse cultural values is a challenging task. We present CLCA, a novel framework for enhancing LLM alignment with cultural values based on cultural learning.
  arXiv  Detail & Related papers  (2025-04-03T18:16:26Z)
• Meta-Cultural Competence: Climbing the Right Hill of Cultural Awareness [11.98067475490853]
  We argue that it is not cultural awareness or knowledge, rather meta-cultural competence, which is required of an AI system.<n>We lay out the principles of meta-cultural competence AI systems, and discuss ways to measure and model those.
  arXiv  Detail & Related papers  (2025-02-09T04:51:59Z)
• Through the Prism of Culture: Evaluating LLMs' Understanding of Indian Subcultures and Traditions [9.357186653223332]
  We evaluate the capacity of Large Language Models to recognize and accurately respond to the Little Traditions within Indian society. Through a series of case studies, we assess whether LLMs can balance the interplay between dominant Great Traditions and localized Little Traditions. Our findings reveal that while LLMs demonstrate an ability to articulate cultural nuances, they often struggle to apply this understanding in practical, context-specific scenarios.
  arXiv  Detail & Related papers  (2025-01-28T06:58:25Z)
• Self-Pluralising Culture Alignment for Large Language Models [36.689491885394034]
  We propose CultureSPA, a framework that allows large language models to align to pluralistic cultures. By comparing culture-aware/unaware outputs, we are able to detect and collect culture-related instances. Extensive experiments demonstrate that CultureSPA significantly improves the alignment of LLMs to diverse cultures without compromising general abilities.
  arXiv  Detail & Related papers  (2024-10-16T19:06:08Z)
• How Well Do LLMs Represent Values Across Cultures? Empirical Analysis of LLM Responses Based on Hofstede Cultural Dimensions [9.275967682881944]
  It is critical to understand whether Large Language Models showcase different values to the user based on the stereotypical values of a user's known country. We prompt different LLMs with a series of advice requests based on 5 Hofstede Cultural Dimensions. We found that LLMs can differentiate between one side of a value and another, as well as understand that countries have differing values.
  arXiv  Detail & Related papers  (2024-06-21T00:58:01Z)
• Translating Across Cultures: LLMs for Intralingual Cultural Adaptation [12.5954253354303]
  We define the task of cultural adaptation and create an evaluation framework to evaluate the performance of modern LLMs. We analyze possible issues with automatic adaptation. We hope that this paper will offer more insight into the cultural understanding of LLMs and their creativity in cross-cultural scenarios.
  arXiv  Detail & Related papers  (2024-06-20T17:06:58Z)
• Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsense [98.09670425244462]
  Large language models (LLMs) have demonstrated substantial commonsense understanding. This paper examines the capabilities and limitations of several state-of-the-art LLMs in the context of cultural commonsense tasks.
  arXiv  Detail & Related papers  (2024-05-07T20:28:34Z)
• CULTURE-GEN: Revealing Global Cultural Perception in Language Models through Natural Language Prompting [73.94059188347582]
  We uncover culture perceptions of three SOTA models on 110 countries and regions on 8 culture-related topics through culture-conditioned generations. We discover that culture-conditioned generation consist of linguistic "markers" that distinguish marginalized cultures apart from default cultures.
  arXiv  Detail & Related papers  (2024-04-16T00:50:43Z)
• Does Mapo Tofu Contain Coffee? Probing LLMs for Food-related Cultural Knowledge [47.57055368312541]
  We introduce FmLAMA, a multilingual dataset centered on food-related cultural facts and variations in food practices. We analyze LLMs across various architectures and configurations, evaluating their performance in both monolingual and multilingual settings.
  arXiv  Detail & Related papers  (2024-04-10T08:49:27Z)
• CulturalTeaming: AI-Assisted Interactive Red-Teaming for Challenging LLMs' (Lack of) Multicultural Knowledge [69.82940934994333]
  We introduce CulturalTeaming, an interactive red-teaming system that leverages human-AI collaboration to build challenging evaluation dataset. Our study reveals that CulturalTeaming's various modes of AI assistance support annotators in creating cultural questions. CULTURALBENCH-V0.1 is a compact yet high-quality evaluation dataset with users' red-teaming attempts.
  arXiv  Detail & Related papers  (2024-04-10T00:25:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.