The Echoes of Multilinguality: Tracing Cultural Value Shifts during LM Fine-tuning
- URL: http://arxiv.org/abs/2405.12744v1
- Date: Tue, 21 May 2024 12:55:15 GMT
- Title: The Echoes of Multilinguality: Tracing Cultural Value Shifts during LM Fine-tuning
- Authors: Rochelle Choenni, Anne Lauscher, Ekaterina Shutova,
- Abstract summary: We study how languages can exert influence on the cultural values encoded for different test languages, by studying how such values are revised during fine-tuning.
Lastly, we use a training data attribution method to find patterns in the fine-tuning examples, and the languages that they come from, that tend to instigate value shifts.
- Score: 23.418656688405605
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Texts written in different languages reflect different culturally-dependent beliefs of their writers. Thus, we expect multilingual LMs (MLMs), that are jointly trained on a concatenation of text in multiple languages, to encode different cultural values for each language. Yet, as the 'multilinguality' of these LMs is driven by cross-lingual sharing, we also have reason to belief that cultural values bleed over from one language into another. This limits the use of MLMs in practice, as apart from being proficient in generating text in multiple languages, creating language technology that can serve a community also requires the output of LMs to be sensitive to their biases (Naous et al., 2023). Yet, little is known about how cultural values emerge and evolve in MLMs (Hershcovich et al., 2022a). We are the first to study how languages can exert influence on the cultural values encoded for different test languages, by studying how such values are revised during fine-tuning. Focusing on the fine-tuning stage allows us to study the interplay between value shifts when exposed to new linguistic experience from different data sources and languages. Lastly, we use a training data attribution method to find patterns in the fine-tuning examples, and the languages that they come from, that tend to instigate value shifts.
Related papers
- Translating Across Cultures: LLMs for Intralingual Cultural Adaptation [12.5954253354303]
We define the task of cultural adaptation and create an evaluation framework to benchmark different models for this task.
We analyze possible issues with automatic adaptation including cultural biases and stereotypes.
arXiv Detail & Related papers (2024-06-20T17:06:58Z) - CIVICS: Building a Dataset for Examining Culturally-Informed Values in Large Language Models [59.22460740026037]
"CIVICS: Culturally-Informed & Values-Inclusive Corpus for Societal impacts" dataset is designed to evaluate the social and cultural variation of Large Language Models (LLMs)
We create a hand-crafted, multilingual dataset of value-laden prompts which address specific socially sensitive topics, including LGBTQI rights, social welfare, immigration, disability rights, and surrogacy.
arXiv Detail & Related papers (2024-05-22T20:19:10Z) - Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsense [98.09670425244462]
Large language models (LLMs) have demonstrated substantial commonsense understanding.
This paper examines the capabilities and limitations of several state-of-the-art LLMs in the context of cultural commonsense tasks.
arXiv Detail & Related papers (2024-05-07T20:28:34Z) - Exploring Multilingual Concepts of Human Value in Large Language Models: Is Value Alignment Consistent, Transferable and Controllable across Languages? [34.38469832305664]
We explore 7 types of human values, 16 languages and 3 LLM series with distinct multilinguality.
Cross-lingual analysis on these concepts discloses 3 traits arising from language resource disparities.
We provide suggestions on the composition of multilingual data for LLMs pre-training.
arXiv Detail & Related papers (2024-02-28T07:18:39Z) - Multi-lingual and Multi-cultural Figurative Language Understanding [69.47641938200817]
Figurative language permeates human communication, but is relatively understudied in NLP.
We create a dataset for seven diverse languages associated with a variety of cultures: Hindi, Indonesian, Javanese, Kannada, Sundanese, Swahili and Yoruba.
Our dataset reveals that each language relies on cultural and regional concepts for figurative expressions, with the highest overlap between languages originating from the same region.
All languages exhibit a significant deficiency compared to English, with variations in performance reflecting the availability of pre-training and fine-tuning data.
arXiv Detail & Related papers (2023-05-25T15:30:31Z) - Benchmarking LLM-based Machine Translation on Cultural Awareness [53.83912076814508]
Translating cultural-specific content is crucial for effective cross-cultural communication.
Recent advancements in in-context learning utilize lightweight prompts to guide large language models (LLMs) in machine translation tasks.
We introduce a new data curation pipeline to construct a culturally relevant parallel corpus.
arXiv Detail & Related papers (2023-05-23T17:56:33Z) - Cross-Cultural Transfer Learning for Chinese Offensive Language
Detection [9.341003339029221]
We aim to investigate the impact of transfer learning using offensive language detection data from different cultural backgrounds.
We find that culture-specific biases in what is considered offensive negatively impact the transferability of language models.
In a few-shot learning scenario, however, our study shows promising prospects for non-English offensive language detection with limited resources.
arXiv Detail & Related papers (2023-03-31T09:50:07Z) - Cross-Lingual Ability of Multilingual Masked Language Models: A Study of
Language Structure [54.01613740115601]
We study three language properties: constituent order, composition and word co-occurrence.
Our main conclusion is that the contribution of constituent order and word co-occurrence is limited, while the composition is more crucial to the success of cross-linguistic transfer.
arXiv Detail & Related papers (2022-03-16T07:09:35Z) - Gender Bias in Multilingual Embeddings and Cross-Lingual Transfer [101.58431011820755]
We study gender bias in multilingual embeddings and how it affects transfer learning for NLP applications.
We create a multilingual dataset for bias analysis and propose several ways for quantifying bias in multilingual representations.
arXiv Detail & Related papers (2020-05-02T04:34:37Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.