Multilingual large language models leak human stereotypes across language boundaries

• URL: http://arxiv.org/abs/2312.07141v2
• Date: Wed, 8 May 2024 20:19:09 GMT
• Title: Multilingual large language models leak human stereotypes across language boundaries
• Authors: Yang Trista Cao, Anna Sotnikova, Jieyu Zhao, Linda X. Zou, Rachel Rudinger, Hal Daume III,
• Abstract summary: We investigate how stereotypical associations leak across four languages: English, Russian, Chinese, and Hindi. Hindi appears to be the most susceptible to influence from other languages, while Chinese is the least.
• Score: 25.903732543380528
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Multilingual large language models have been increasingly popular for their proficiency in processing and generating text across various languages. Previous research has shown that the presence of stereotypes and biases in monolingual large language models can be attributed to the nature of their training data, which is collected from humans and reflects societal biases. Multilingual language models undergo the same training procedure as monolingual ones, albeit with training data sourced from various languages. This raises the question: do stereotypes present in one social context leak across languages within the model? In our work, we first define the term ``stereotype leakage'' and propose a framework for its measurement. With this framework, we investigate how stereotypical associations leak across four languages: English, Russian, Chinese, and Hindi. To quantify the stereotype leakage, we employ an approach from social psychology, measuring stereotypes via group-trait associations. We evaluate human stereotypes and stereotypical associations manifested in multilingual large language models such as mBERT, mT5, and GPT-3.5. Our findings show a noticeable leakage of positive, negative, and non-polar associations across all languages. Notably, Hindi within multilingual models appears to be the most susceptible to influence from other languages, while Chinese is the least. Additionally, GPT-3.5 exhibits a better alignment with human scores than other models. WARNING: This paper contains model outputs which could be offensive in nature.

Related papers

• Spoken Stereoset: On Evaluating Social Bias Toward Speaker in Speech Large Language Models [50.40276881893513]
  This study introduces Spoken Stereoset, a dataset specifically designed to evaluate social biases in Speech Large Language Models (SLLMs) By examining how different models respond to speech from diverse demographic groups, we aim to identify these biases. The findings indicate that while most models show minimal bias, some still exhibit slightly stereotypical or anti-stereotypical tendencies.
  arXiv  Detail & Related papers  (2024-08-14T16:55:06Z)
• Does Liking Yellow Imply Driving a School Bus? Semantic Leakage in Language Models [113.58052868898173]
  We identify and characterize a phenomenon never discussed before, where models leak irrelevant information from the prompt into the generation in unexpected ways. We propose an evaluation setting to detect semantic leakage both by humans and automatically, curate a diverse test suite for diagnosing this behavior, and measure significant semantic leakage in 13 flagship models.
  arXiv  Detail & Related papers  (2024-08-12T22:30:55Z)
• Multilingual Text-to-Image Generation Magnifies Gender Stereotypes and Prompt Engineering May Not Help You [64.74707085021858]
  We show that multilingual models suffer from significant gender biases just as monolingual models do. We propose a novel benchmark, MAGBIG, intended to foster research on gender bias in multilingual models. Our results show that not only do models exhibit strong gender biases but they also behave differently across languages.
  arXiv  Detail & Related papers  (2024-01-29T12:02:28Z)
• Counteracts: Testing Stereotypical Representation in Pre-trained Language Models [4.211128681972148]
  We use counterexamples to examine the internal stereotypical knowledge in pre-trained language models (PLMs) We evaluate 7 PLMs on 9 types of cloze-style prompt with different information and base knowledge.
  arXiv  Detail & Related papers  (2023-01-11T07:52:59Z)
• Are Pretrained Multilingual Models Equally Fair Across Languages? [0.0]
  This work investigates the group fairness of multilingual models, asking whether these models are equally fair across languages. We evaluate three multilingual models on MozArt -- mBERT, XLM-R, and mT5 -- and show that across the four target languages, the three models exhibit different levels of group disparity.
  arXiv  Detail & Related papers  (2022-10-11T13:59:19Z)
• Language Contamination Explains the Cross-lingual Capabilities of English Pretrained Models [79.38278330678965]
  We find that common English pretraining corpora contain significant amounts of non-English text. This leads to hundreds of millions of foreign language tokens in large-scale datasets. We then demonstrate that even these small percentages of non-English data facilitate cross-lingual transfer for models trained on them.
  arXiv  Detail & Related papers  (2022-04-17T23:56:54Z)
• Do Multilingual Language Models Capture Differing Moral Norms? [71.52261949766101]
  Massively multilingual sentence representations are trained on large corpora of uncurated data. This may cause the models to grasp cultural values including moral judgments from the high-resource languages. The lack of data in certain languages can also lead to developing random and thus potentially harmful beliefs.
  arXiv  Detail & Related papers  (2022-03-18T12:26:37Z)
• Discovering Representation Sprachbund For Multilingual Pre-Training [139.05668687865688]
  We generate language representation from multilingual pre-trained models and conduct linguistic analysis. We cluster all the target languages into multiple groups and name each group as a representation sprachbund. Experiments are conducted on cross-lingual benchmarks and significant improvements are achieved compared to strong baselines.
  arXiv  Detail & Related papers  (2021-09-01T09:32:06Z)
• Uncovering Constraint-Based Behavior in Neural Models via Targeted Fine-Tuning [9.391375268580806]
  We show that competing linguistic processes within a language obscure underlying linguistic knowledge. While human behavior has been found to be similar across languages, we find cross-linguistic variation in model behavior. Our results suggest that models need to learn both the linguistic constraints in a language and their relative ranking, with mismatches in either producing non-human-like behavior.
  arXiv  Detail & Related papers  (2021-06-02T14:52:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.