Massively Multi-Cultural Knowledge Acquisition & LM Benchmarking
- URL: http://arxiv.org/abs/2402.09369v1
- Date: Wed, 14 Feb 2024 18:16:54 GMT
- Title: Massively Multi-Cultural Knowledge Acquisition & LM Benchmarking
- Authors: Yi Fung, Ruining Zhao, Jae Doo, Chenkai Sun, Heng Ji
- Abstract summary: This paper introduces a novel approach for massively multicultural knowledge acquisition.
Our method strategically navigates from densely informative Wikipedia documents on cultural topics to an extensive network of linked pages.
Our work marks an important step towards deeper understanding and bridging the gaps of cultural disparities in AI.
- Score: 48.21982147529661
- License: http://creativecommons.org/licenses/by-sa/4.0/
- Abstract: Pretrained large language models have revolutionized many applications but
still face challenges related to cultural bias and a lack of cultural
commonsense knowledge crucial for guiding cross-culture communication and
interactions. Recognizing the shortcomings of existing methods in capturing the
diverse and rich cultures across the world, this paper introduces a novel
approach for massively multicultural knowledge acquisition. Specifically, our
method strategically navigates from densely informative Wikipedia documents on
cultural topics to an extensive network of linked pages. Leveraging this
valuable source of data collection, we construct the CultureAtlas dataset,
which covers a wide range of sub-country level geographical regions and
ethnolinguistic groups, with data cleaning and preprocessing to ensure textual
assertion sentence self-containment, as well as fine-grained cultural profile
information extraction. Our dataset not only facilitates the evaluation of
language model performance in culturally diverse contexts but also serves as a
foundational tool for the development of culturally sensitive and aware
language models. Our work marks an important step towards deeper understanding
and bridging the gaps of cultural disparities in AI, to promote a more
inclusive and balanced representation of global cultures in the digital domain.
Related papers
- Extrinsic Evaluation of Cultural Competence in Large Language Models [53.626808086522985]
We focus on extrinsic evaluation of cultural competence in two text generation tasks.
We evaluate model outputs when an explicit cue of culture, specifically nationality, is perturbed in the prompts.
We find weak correlations between text similarity of outputs for different countries and the cultural values of these countries.
arXiv Detail & Related papers (2024-06-17T14:03:27Z) - CulturePark: Boosting Cross-cultural Understanding in Large Language Models [63.452948673344395]
This paper introduces CulturePark, an LLM-powered multi-agent communication framework for cultural data collection.
It generates high-quality cross-cultural dialogues encapsulating human beliefs, norms, and customs.
We evaluate these models across three downstream tasks: content moderation, cultural alignment, and cultural education.
arXiv Detail & Related papers (2024-05-24T01:49:02Z) - CIVICS: Building a Dataset for Examining Culturally-Informed Values in Large Language Models [59.22460740026037]
"CIVICS: Culturally-Informed & Values-Inclusive Corpus for Societal impacts" dataset is designed to evaluate the social and cultural variation of Large Language Models (LLMs)
We create a hand-crafted, multilingual dataset of value-laden prompts which address specific socially sensitive topics, including LGBTQI rights, social welfare, immigration, disability rights, and surrogacy.
arXiv Detail & Related papers (2024-05-22T20:19:10Z) - Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsense [98.09670425244462]
Large language models (LLMs) have demonstrated substantial commonsense understanding.
This paper examines the capabilities and limitations of several state-of-the-art LLMs in the context of cultural commonsense tasks.
arXiv Detail & Related papers (2024-05-07T20:28:34Z) - CRAFT: Extracting and Tuning Cultural Instructions from the Wild [38.255242754975654]
This paper introduces a novel pipeline for extracting high-quality, culturally-related instruction tuning datasets from vast unstructured corpora.
We utilize a self-instruction generation pipeline to identify cultural concepts and trigger instruction.
We conduct experiments across three regions: Singapore, the Philippines, and the United States, achieving performance improvement of up to 6%.
arXiv Detail & Related papers (2024-05-06T03:21:55Z) - CULTURE-GEN: Revealing Global Cultural Perception in Language Models through Natural Language Prompting [68.37589899302161]
We uncover culture perceptions of three SOTA models on 110 countries and regions on 8 culture-related topics through culture-conditioned generations.
We discover that culture-conditioned generation consist of linguistic "markers" that distinguish marginalized cultures apart from default cultures.
arXiv Detail & Related papers (2024-04-16T00:50:43Z) - Investigating Cultural Alignment of Large Language Models [10.738300803676655]
We show that Large Language Models (LLMs) genuinely encapsulate the diverse knowledge adopted by different cultures.
We quantify cultural alignment by simulating sociological surveys, comparing model responses to those of actual survey participants as references.
We introduce Anthropological Prompting, a novel method leveraging anthropological reasoning to enhance cultural alignment.
arXiv Detail & Related papers (2024-02-20T18:47:28Z) - Cultural Compass: Predicting Transfer Learning Success in Offensive
Language Detection with Cultural Features [21.54368550883955]
Our study delves into the intersection of cultural features and transfer learning effectiveness.
Based on these results, we advocate for the integration of cultural information into datasets.
Our research signifies a step forward in the quest for more inclusive, culturally sensitive language technologies.
arXiv Detail & Related papers (2023-10-10T09:29:38Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.