The Ghost in the Machine has an American accent: value conflict in GPT-3

• URL: http://arxiv.org/abs/2203.07785v1
• Date: Tue, 15 Mar 2022 11:06:54 GMT
• Title: The Ghost in the Machine has an American accent: value conflict in GPT-3
• Authors: Rebecca L Johnson, Giada Pistilli, Natalia Men\'edez-Gonz\'alez, Leslye Denisse Dias Duran, Enrico Panai, Julija Kalpokiene, Donald Jay Bertulfo
• Abstract summary: We discuss how the co-creation of language and cultural value impacts large language models. We stress tested GPT-3 with a range of value-rich texts representing several languages and nations. We observed when values embedded in the input text were mutated in the generated outputs.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The alignment problem in the context of large language models must consider the plurality of human values in our world. Whilst there are many resonant and overlapping values amongst the world's cultures, there are also many conflicting, yet equally valid, values. It is important to observe which cultural values a model exhibits, particularly when there is a value conflict between input prompts and generated outputs. We discuss how the co-creation of language and cultural value impacts large language models (LLMs). We explore the constitution of the training data for GPT-3 and compare that to the world's language and internet access demographics, as well as to reported statistical profiles of dominant values in some Nation-states. We stress tested GPT-3 with a range of value-rich texts representing several languages and nations; including some with values orthogonal to dominant US public opinion as reported by the World Values Survey. We observed when values embedded in the input text were mutated in the generated outputs and noted when these conflicting values were more aligned with reported dominant US values. Our discussion of these results uses a moral value pluralism (MVP) lens to better understand these value mutations. Finally, we provide recommendations for how our work may contribute to other current work in the field.

Related papers

• Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation [71.59208664920452]
  Cultural biases in multilingual datasets pose significant challenges for their effectiveness as global benchmarks. We show that progress on MMLU depends heavily on learning Western-centric concepts, with 28% of all questions requiring culturally sensitive knowledge. We release Global MMLU, an improved MMLU with evaluation coverage across 42 languages.
  arXiv  Detail & Related papers  (2024-12-04T13:27:09Z)
• Large Language Models as Mirrors of Societal Moral Standards [0.5852077003870417]
  Language models can, to a limited extent, represent moral norms in a variety of cultural contexts. This study evaluates the effectiveness of these models using information from two surveys, the WVS and the PEW, that encompass moral perspectives from over 40 countries. The results show that biases exist in both monolingual and multilingual models, and they typically fall short of accurately capturing the moral intricacies of diverse cultures.
  arXiv  Detail & Related papers  (2024-12-01T20:20:35Z)
• Language Model Alignment in Multilingual Trolley Problems [138.5684081822807]
  Building on the Moral Machine experiment, we develop a cross-lingual corpus of moral dilemma vignettes in over 100 languages called MultiTP. Our analysis explores the alignment of 19 different LLMs with human judgments, capturing preferences across six moral dimensions. We discover significant variance in alignment across languages, challenging the assumption of uniform moral reasoning in AI systems.
  arXiv  Detail & Related papers  (2024-07-02T14:02:53Z)
• Extrinsic Evaluation of Cultural Competence in Large Language Models [53.626808086522985]
  We focus on extrinsic evaluation of cultural competence in two text generation tasks. We evaluate model outputs when an explicit cue of culture, specifically nationality, is perturbed in the prompts. We find weak correlations between text similarity of outputs for different countries and the cultural values of these countries.
  arXiv  Detail & Related papers  (2024-06-17T14:03:27Z)
• CIVICS: Building a Dataset for Examining Culturally-Informed Values in Large Language Models [59.22460740026037]
  "CIVICS: Culturally-Informed & Values-Inclusive Corpus for Societal impacts" dataset is designed to evaluate the social and cultural variation of Large Language Models (LLMs) We create a hand-crafted, multilingual dataset of value-laden prompts which address specific socially sensitive topics, including LGBTQI rights, social welfare, immigration, disability rights, and surrogacy.
  arXiv  Detail & Related papers  (2024-05-22T20:19:10Z)
• D3CODE: Disentangling Disagreements in Data across Cultures on Offensiveness Detection and Evaluation [5.9053106775634685]
  We introduce the dataset: a large-scale cross-cultural dataset of parallel annotations for offensive language in over 4.5K sentences annotated by a pool of over 4k annotators. The dataset contains annotators' moral values captured along six moral foundations: care, equality, proportionality, authority, loyalty, and purity. Our analyses reveal substantial regional variations in annotators' perceptions that are shaped by individual moral values.
  arXiv  Detail & Related papers  (2024-04-16T19:12:03Z)
• Not All Countries Celebrate Thanksgiving: On the Cultural Dominance in Large Language Models [89.94270049334479]
  This paper identifies a cultural dominance issue within large language models (LLMs) LLMs often provide inappropriate English-culture-related answers that are not relevant to the expected culture when users ask in non-English languages.
  arXiv  Detail & Related papers  (2023-10-19T05:38:23Z)
• Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties [68.66719970507273]
  Value pluralism is the view that multiple correct values may be held in tension with one another. As statistical learners, AI systems fit to averages by default, washing out potentially irreducible value conflicts. We introduce ValuePrism, a large-scale dataset of 218k values, rights, and duties connected to 31k human-written situations.
  arXiv  Detail & Related papers  (2023-09-02T01:24:59Z)
• Probing Pre-Trained Language Models for Cross-Cultural Differences in Values [42.45033681054207]
  We introduce probes to study which values across cultures are embedded in Pre-Trained Language models. We find that PTLMs capture differences in values across cultures, but those only weakly align with established value surveys.
  arXiv  Detail & Related papers  (2022-03-25T15:45:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.