Do Multilingual Language Models Capture Differing Moral Norms?

• URL: http://arxiv.org/abs/2203.09904v1
• Date: Fri, 18 Mar 2022 12:26:37 GMT
• Title: Do Multilingual Language Models Capture Differing Moral Norms?
• Authors: Katharina H\"ammerl, Bj\"orn Deiseroth, Patrick Schramowski, Jind\v{r}ich Libovick\'y, Alexander Fraser, Kristian Kersting
• Abstract summary: Massively multilingual sentence representations are trained on large corpora of uncurated data. This may cause the models to grasp cultural values including moral judgments from the high-resource languages. The lack of data in certain languages can also lead to developing random and thus potentially harmful beliefs.
• Score: 71.52261949766101
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Massively multilingual sentence representations are trained on large corpora of uncurated data, with a very imbalanced proportion of languages included in the training. This may cause the models to grasp cultural values including moral judgments from the high-resource languages and impose them on the low-resource languages. The lack of data in certain languages can also lead to developing random and thus potentially harmful beliefs. Both these issues can negatively influence zero-shot cross-lingual model transfer and potentially lead to harmful outcomes. Therefore, we aim to (1) detect and quantify these issues by comparing different models in different languages, (2) develop methods for improving undesirable properties of the models. Our initial experiments using the multilingual model XLM-R show that indeed multilingual LMs capture moral norms, even with potentially higher human-agreement than monolingual ones. However, it is not yet clear to what extent these moral norms differ between languages.

Related papers

• Multilingual Text-to-Image Generation Magnifies Gender Stereotypes and Prompt Engineering May Not Help You [64.74707085021858]
  We show that multilingual models suffer from significant gender biases just as monolingual models do. We propose a novel benchmark, MAGBIG, intended to foster research on gender bias in multilingual models. Our results show that not only do models exhibit strong gender biases but they also behave differently across languages.
  arXiv  Detail & Related papers  (2024-01-29T12:02:28Z)
• SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoning [44.53966523376327]
  SeaEval is a benchmark for multilingual foundation models. We characterize how these models understand and reason with natural language. We also investigate how well they comprehend cultural practices, nuances, and values.
  arXiv  Detail & Related papers  (2023-09-09T11:42:22Z)
• Language Versatilists vs. Specialists: An Empirical Revisiting on Multilingual Transfer Ability [11.000499414131324]
  We conduct experiments across four types of reasoning tasks. We find that the multilingual pretrained model does not always outperform an English-centric model. English appears to be a less suitable source language, and the choice of source language becomes less important when the English-centric model scales up.
  arXiv  Detail & Related papers  (2023-06-11T14:03:09Z)
• Speaking Multiple Languages Affects the Moral Bias of Language Models [70.94372902010232]
  Pre-trained multilingual language models (PMLMs) are commonly used when dealing with data from multiple languages and cross-lingual transfer. Do the models capture moral norms from English and impose them on other languages? Our experiments demonstrate that, indeed, PMLMs encode differing moral biases, but these do not necessarily correspond to cultural differences or commonalities in human opinions.
  arXiv  Detail & Related papers  (2022-11-14T20:08:54Z)
• Cross-lingual Transfer Learning for Check-worthy Claim Identification over Twitter [7.601937548486356]
  Misinformation spread over social media has become an undeniable infodemic. We present a systematic study of six approaches for cross-lingual check-worthiness estimation across pairs of five diverse languages with the help of Multilingual BERT (mBERT) model. Our results show that for some language pairs, zero-shot cross-lingual transfer is possible and can perform as good as monolingual models that are trained on the target language.
  arXiv  Detail & Related papers  (2022-11-09T18:18:53Z)
• MonoByte: A Pool of Monolingual Byte-level Language Models [4.491765479948667]
  We release 10 monolingual byte-level models rigorously pretrained under the same configuration. Because they are tokenizer-free, the problem of unseen token embeddings is eliminated. Experiments on QA and NLI tasks show that our monolingual models achieve competitive performance to the multilingual one.
  arXiv  Detail & Related papers  (2022-09-22T14:32:48Z)
• High-resource Language-specific Training for Multilingual Neural Machine Translation [109.31892935605192]
  We propose the multilingual translation model with the high-resource language-specific training (HLT-MT) to alleviate the negative interference. Specifically, we first train the multilingual model only with the high-resource pairs and select the language-specific modules at the top of the decoder. HLT-MT is further trained on all available corpora to transfer knowledge from high-resource languages to low-resource languages.
  arXiv  Detail & Related papers  (2022-07-11T14:33:13Z)
• Same Neurons, Different Languages: Probing Morphosyntax in Multilingual Pre-trained Models [84.86942006830772]
  We conjecture that multilingual pre-trained models can derive language-universal abstractions about grammar. We conduct the first large-scale empirical study over 43 languages and 14 morphosyntactic categories with a state-of-the-art neuron-level probe.
  arXiv  Detail & Related papers  (2022-05-04T12:22:31Z)
• Limits of Detecting Text Generated by Large-Scale Language Models [65.46403462928319]
  Some consider large-scale language models that can generate long and coherent pieces of text as dangerous, since they may be used in misinformation campaigns. Here we formulate large-scale language model output detection as a hypothesis testing problem to classify text as genuine or generated.
  arXiv  Detail & Related papers  (2020-02-09T19:53:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.