Not All Countries Celebrate Thanksgiving: On the Cultural Dominance in
Large Language Models
- URL: http://arxiv.org/abs/2310.12481v2
- Date: Fri, 16 Feb 2024 14:06:41 GMT
- Title: Not All Countries Celebrate Thanksgiving: On the Cultural Dominance in
Large Language Models
- Authors: Wenxuan Wang, Wenxiang Jiao, Jingyuan Huang, Ruyi Dai, Jen-tse Huang,
Zhaopeng Tu, Michael R. Lyu
- Abstract summary: This paper identifies a cultural dominance issue within large language models (LLMs)
LLMs often provide inappropriate English-culture-related answers that are not relevant to the expected culture when users ask in non-English languages.
- Score: 89.94270049334479
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: This paper identifies a cultural dominance issue within large language models
(LLMs) due to the predominant use of English data in model training (e.g.,
ChatGPT). LLMs often provide inappropriate English-culture-related answers that
are not relevant to the expected culture when users ask in non-English
languages. To systematically evaluate the cultural dominance issue, we build a
benchmark of concrete (e.g., holidays and songs) and abstract (e.g., values and
opinions) cultural objects. Empirical results show that the representative GPT
models suffer from the culture dominance problem, where GPT-4 is the most
affected while text-davinci-003 suffers the least from this problem. Our study
emphasizes the need to critically examine cultural dominance and ethical
consideration in their development and deployment. We show that two
straightforward methods in model development (i.e., pretraining on more diverse
data) and deployment (e.g., culture-aware prompting) can significantly mitigate
the cultural dominance issue in LLMs.
Related papers
- Extrinsic Evaluation of Cultural Competence in Large Language Models [53.626808086522985]
We focus on extrinsic evaluation of cultural competence in two text generation tasks.
We evaluate model outputs when an explicit cue of culture, specifically nationality, is perturbed in the prompts.
We find weak correlations between text similarity of outputs for different countries and the cultural values of these countries.
arXiv Detail & Related papers (2024-06-17T14:03:27Z) - CulturePark: Boosting Cross-cultural Understanding in Large Language Models [63.452948673344395]
This paper introduces CulturePark, an LLM-powered multi-agent communication framework for cultural data collection.
It generates high-quality cross-cultural dialogues encapsulating human beliefs, norms, and customs.
We evaluate these models across three downstream tasks: content moderation, cultural alignment, and cultural education.
arXiv Detail & Related papers (2024-05-24T01:49:02Z) - Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsense [98.09670425244462]
Large language models (LLMs) have demonstrated substantial commonsense understanding.
This paper examines the capabilities and limitations of several state-of-the-art LLMs in the context of cultural commonsense tasks.
arXiv Detail & Related papers (2024-05-07T20:28:34Z) - CULTURE-GEN: Revealing Global Cultural Perception in Language Models through Natural Language Prompting [68.37589899302161]
We uncover culture perceptions of three SOTA models on 110 countries and regions on 8 culture-related topics through culture-conditioned generations.
We discover that culture-conditioned generation consist of linguistic "markers" that distinguish marginalized cultures apart from default cultures.
arXiv Detail & Related papers (2024-04-16T00:50:43Z) - Investigating Cultural Alignment of Large Language Models [10.738300803676655]
We show that Large Language Models (LLMs) genuinely encapsulate the diverse knowledge adopted by different cultures.
We quantify cultural alignment by simulating sociological surveys, comparing model responses to those of actual survey participants as references.
We introduce Anthropological Prompting, a novel method leveraging anthropological reasoning to enhance cultural alignment.
arXiv Detail & Related papers (2024-02-20T18:47:28Z) - Massively Multi-Cultural Knowledge Acquisition & LM Benchmarking [48.21982147529661]
This paper introduces a novel approach for massively multicultural knowledge acquisition.
Our method strategically navigates from densely informative Wikipedia documents on cultural topics to an extensive network of linked pages.
Our work marks an important step towards deeper understanding and bridging the gaps of cultural disparities in AI.
arXiv Detail & Related papers (2024-02-14T18:16:54Z) - Cultural Bias and Cultural Alignment of Large Language Models [0.9374652839580183]
We conduct a disaggregated evaluation of cultural bias for five widely used large language models.
All models exhibit cultural values resembling English-speaking and Protestant European countries.
We suggest using cultural prompting and ongoing evaluation to reduce cultural bias in the output of generative AI.
arXiv Detail & Related papers (2023-11-23T16:45:56Z) - Cultural Alignment in Large Language Models: An Explanatory Analysis Based on Hofstede's Cultural Dimensions [10.415002561977655]
This research proposes a Cultural Alignment Test (Hoftede's CAT) to quantify cultural alignment using Hofstede's cultural dimension framework.
We quantitatively evaluate large language models (LLMs) against the cultural dimensions of regions like the United States, China, and Arab countries.
Our results quantify the cultural alignment of LLMs and reveal the difference between LLMs in explanatory cultural dimensions.
arXiv Detail & Related papers (2023-08-25T14:50:13Z) - Assessing Cross-Cultural Alignment between ChatGPT and Human Societies:
An Empirical Study [9.919972416590124]
ChatGPT has garnered widespread recognition for its exceptional ability to generate human-like responses in dialogue.
We investigate the underlying cultural background of ChatGPT by analyzing its responses to questions designed to quantify human cultural differences.
arXiv Detail & Related papers (2023-03-30T15:43:39Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.