Do language models practice what they preach? Examining language ideologies about gendered language reform encoded in LLMs

• URL: http://arxiv.org/abs/2409.13852v1
• Date: Fri, 20 Sep 2024 18:55:48 GMT
• Title: Do language models practice what they preach? Examining language ideologies about gendered language reform encoded in LLMs
• Authors: Julia Watson, Sophia Lee, Barend Beekhuizen, Suzanne Stevenson,
• Abstract summary: We study language ideologies in text produced by LLMs through a case study on English gendered language reform. We find political bias: when asked to use language that is "correct" or "natural", LLMs use language most similarly to when asked to align with conservative (vs. progressive) values. This shows how the language ideologies expressed in text produced by LLMs can vary, which may be unexpected to users.
• Score: 6.06227550292852
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study language ideologies in text produced by LLMs through a case study on English gendered language reform (related to role nouns like congressperson/-woman/-man, and singular they). First, we find political bias: when asked to use language that is "correct" or "natural", LLMs use language most similarly to when asked to align with conservative (vs. progressive) values. This shows how LLMs' metalinguistic preferences can implicitly communicate the language ideologies of a particular political group, even in seemingly non-political contexts. Second, we find LLMs exhibit internal inconsistency: LLMs use gender-neutral variants more often when more explicit metalinguistic context is provided. This shows how the language ideologies expressed in text produced by LLMs can vary, which may be unexpected to users. We discuss the broader implications of these findings for value alignment.

Related papers

• Large Language Models Reflect the Ideology of their Creators [73.25935570218375]
  Large language models (LLMs) are trained on vast amounts of data to generate natural language. We uncover notable diversity in the ideological stance exhibited across different LLMs and languages.
  arXiv  Detail & Related papers  (2024-10-24T04:02:30Z)
• LLMs' Understanding of Natural Language Revealed [0.0]
  Large language models (LLMs) are the result of a massive experiment in bottom-up, data-driven reverse engineering of language at scale. We will focus on testing LLMs for their language understanding capabilities, their supposed forte.
  arXiv  Detail & Related papers  (2024-07-29T01:21:11Z)
• What an Elegant Bridge: Multilingual LLMs are Biased Similarly in Different Languages [51.0349882045866]
  This paper investigates biases of Large Language Models (LLMs) through the lens of grammatical gender. We prompt a model to describe nouns with adjectives in various languages, focusing specifically on languages with grammatical gender. We find that a simple classifier can not only predict noun gender above chance but also exhibit cross-language transferability.
  arXiv  Detail & Related papers  (2024-07-12T22:10:16Z)
• Whose Side Are You On? Investigating the Political Stance of Large Language Models [56.883423489203786]
  We investigate the political orientation of Large Language Models (LLMs) across a spectrum of eight polarizing topics. Our investigation delves into the political alignment of LLMs across a spectrum of eight polarizing topics, spanning from abortion to LGBTQ issues. The findings suggest that users should be mindful when crafting queries, and exercise caution in selecting neutral prompt language.
  arXiv  Detail & Related papers  (2024-03-15T04:02:24Z)
• The Political Preferences of LLMs [0.0]
  I administer 11 political orientation tests, designed to identify the political preferences of the test taker, to 24 state-of-the-art conversational LLMs. Most conversational LLMs generate responses that are diagnosed by most political test instruments as manifesting preferences for left-of-center viewpoints. I demonstrate that LLMs can be steered towards specific locations in the political spectrum through Supervised Fine-Tuning.
  arXiv  Detail & Related papers  (2024-02-02T02:43:10Z)
• How Proficient Are Large Language Models in Formal Languages? An In-Depth Insight for Knowledge Base Question Answering [52.86931192259096]
  Knowledge Base Question Answering (KBQA) aims to answer natural language questions based on facts in knowledge bases. Recent works leverage the capabilities of large language models (LLMs) for logical form generation to improve performance.
  arXiv  Detail & Related papers  (2024-01-11T09:27:50Z)
• Measurement in the Age of LLMs: An Application to Ideological Scaling [1.9413548770753526]
  This paper explores the use of large language models (LLMs) to navigate the conceptual clutter inherent to social scientific measurement tasks. We rely on LLMs' remarkable linguistic fluency to elicit ideological scales of both legislators and text.
  arXiv  Detail & Related papers  (2023-12-14T18:34:06Z)
• Probing Explicit and Implicit Gender Bias through LLM Conditional Text Generation [64.79319733514266]
  Large Language Models (LLMs) can generate biased and toxic responses. We propose a conditional text generation mechanism without the need for predefined gender phrases and stereotypes.
  arXiv  Detail & Related papers  (2023-11-01T05:31:46Z)
• Large Language Models: The Need for Nuance in Current Debates and a Pragmatic Perspective on Understanding [1.3654846342364308]
  Large Language Models (LLMs) are unparalleled in their ability to generate grammatically correct, fluent text. This position paper critically assesses three points recurring in critiques of LLM capacities. We outline a pragmatic perspective on the issue of real' understanding and intentionality in LLMs.
  arXiv  Detail & Related papers  (2023-10-30T15:51:04Z)
• In-Context Impersonation Reveals Large Language Models' Strengths and Biases [56.61129643802483]
  We ask LLMs to assume different personas before solving vision and language tasks. We find that LLMs pretending to be children of different ages recover human-like developmental stages. In a language-based reasoning task, we find that LLMs impersonating domain experts perform better than LLMs impersonating non-domain experts.
  arXiv  Detail & Related papers  (2023-05-24T09:13:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.