Fine-Tuned LLMs are "Time Capsules" for Tracking Societal Bias Through Books

• URL: http://arxiv.org/abs/2502.05331v2
• Date: Thu, 13 Feb 2025 17:27:15 GMT
• Title: Fine-Tuned LLMs are "Time Capsules" for Tracking Societal Bias Through Books
• Authors: Sangmitra Madhusudan, Robert Morabito, Skye Reid, Nikta Gohari Sadr, Ali Emami,
• Abstract summary: We develop Book 0% to 0%, a corpus comprising 593 fictional books across seven decades (1950-2019) We examine shifts in biases related to gender, sexual orientation, race, and religion. Our findings indicate that LLMs trained on decade-specific books manifest biases reflective of their times.
• Score: 5.770485638414148
• License:
• Abstract: Books, while often rich in cultural insights, can also mirror societal biases of their eras - biases that Large Language Models (LLMs) may learn and perpetuate during training. We introduce a novel method to trace and quantify these biases using fine-tuned LLMs. We develop BookPAGE, a corpus comprising 593 fictional books across seven decades (1950-2019), to track bias evolution. By fine-tuning LLMs on books from each decade and using targeted prompts, we examine shifts in biases related to gender, sexual orientation, race, and religion. Our findings indicate that LLMs trained on decade-specific books manifest biases reflective of their times, with both gradual trends and notable shifts. For example, model responses showed a progressive increase in the portrayal of women in leadership roles (from 8% to 22%) from the 1950s to 2010s, with a significant uptick in the 1990s (from 4% to 12%), possibly aligning with third-wave feminism. Same-sex relationship references increased markedly from the 1980s to 2000s (from 0% to 10%), mirroring growing LGBTQ+ visibility. Concerningly, negative portrayals of Islam rose sharply in the 2000s (26% to 38%), likely reflecting post-9/11 sentiments. Importantly, we demonstrate that these biases stem mainly from the books' content and not the models' architecture or initial training. Our study offers a new perspective on societal bias trends by bridging AI, literary studies, and social science research.

Related papers

• How far can bias go? -- Tracing bias from pretraining data to alignment [54.51310112013655]
  This study examines the correlation between gender-occupation bias in pre-training data and their manifestation in LLMs. Our findings reveal that biases present in pre-training data are amplified in model outputs.
  arXiv  Detail & Related papers  (2024-11-28T16:20:25Z)
• LLMs are Biased Teachers: Evaluating LLM Bias in Personalized Education [6.354025374447606]
  We evaluate large language models (LLMs) for bias in the personalized educational setting. We reveal significant biases in how models generate and select educational content tailored to different demographic groups.
  arXiv  Detail & Related papers  (2024-10-17T20:27:44Z)
• Gender Biases in LLMs: Higher intelligence in LLM does not necessarily solve gender bias and stereotyping [0.0]
  Large Language Models (LLMs) are finding applications in all aspects of life, but their susceptibility to biases, particularly gender stereotyping, raises ethical concerns. This study introduces a novel methodology, a persona-based framework, and a unisex name methodology to investigate whether higher-intelligence LLMs reduce such biases.
  arXiv  Detail & Related papers  (2024-09-30T05:22:54Z)
• Evaluating Gender, Racial, and Age Biases in Large Language Models: A Comparative Analysis of Occupational and Crime Scenarios [0.0]
  This paper examines bias in Large Language Models (LLMs) Findings reveal that LLMs often depict female characters more frequently than male ones in various occupations. Efforts to reduce gender and racial bias often lead to outcomes that may over-index one sub-class.
  arXiv  Detail & Related papers  (2024-09-22T20:21:20Z)
• Delving into ChatGPT usage in academic writing through excess vocabulary [4.58733012283457]
  Large language models (LLMs) like ChatGPT can generate and revise text with human-level performance. We study vocabulary changes in 14 million PubMed abstracts from 2010--2024, and show how the appearance of LLMs led to an abrupt increase in the frequency of certain style words. We show that LLMs have had an unprecedented impact on the scientific literature, surpassing the effect of major world events such as the Covid pandemic.
  arXiv  Detail & Related papers  (2024-06-11T07:16:34Z)
• Whose Side Are You On? Investigating the Political Stance of Large Language Models [56.883423489203786]
  We investigate the political orientation of Large Language Models (LLMs) across a spectrum of eight polarizing topics. Our investigation delves into the political alignment of LLMs across a spectrum of eight polarizing topics, spanning from abortion to LGBTQ issues. The findings suggest that users should be mindful when crafting queries, and exercise caution in selecting neutral prompt language.
  arXiv  Detail & Related papers  (2024-03-15T04:02:24Z)
• Large Language Models are Geographically Biased [47.88767211956144]
  We study what Large Language Models (LLMs) know about the world we live in through the lens of geography. We show various problematic geographic biases, which we define as systemic errors in geospatial predictions.
  arXiv  Detail & Related papers  (2024-02-05T02:32:09Z)
• Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs [67.51906565969227]
  We study the unintended side-effects of persona assignment on the ability of LLMs to perform basic reasoning tasks. Our study covers 24 reasoning datasets, 4 LLMs, and 19 diverse personas (e.g. an Asian person) spanning 5 socio-demographic groups.
  arXiv  Detail & Related papers  (2023-11-08T18:52:17Z)
• "Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letters [97.11173801187816]
  Large Language Models (LLMs) have recently emerged as an effective tool to assist individuals in writing various types of content. This paper critically examines gender biases in LLM-generated reference letters.
  arXiv  Detail & Related papers  (2023-10-13T16:12:57Z)
• Gender bias and stereotypes in Large Language Models [0.6882042556551611]
  This paper investigates Large Language Models' behavior with respect to gender stereotypes. We use a simple paradigm to test the presence of gender bias, building on but differing from WinoBias. Our contributions in this paper are as follows: (a) LLMs are 3-6 times more likely to choose an occupation that stereotypically aligns with a person's gender; (b) these choices align with people's perceptions better than with the ground truth as reflected in official job statistics; (d) LLMs ignore crucial ambiguities in sentence structure 95% of the time in our study items, but when explicitly prompted, they recognize
  arXiv  Detail & Related papers  (2023-08-28T22:32:05Z)
• Forgotten Knowledge: Examining the Citational Amnesia in NLP [63.13508571014673]
  We show how far back in time do we tend to go to cite papers? How has that changed over time, and what factors correlate with this citational attention/amnesia? We show that around 62% of cited papers are from the immediate five years prior to publication, whereas only about 17% are more than ten years old. We show that the median age and age diversity of cited papers were steadily increasing from 1990 to 2014, but since then, the trend has reversed, and current NLP papers have an all-time low temporal citation diversity.
  arXiv  Detail & Related papers  (2023-05-29T18:30:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.