Investigating Counterfactual Unfairness in LLMs towards Identities through Humor

• URL: http://arxiv.org/abs/2604.18729v1
• Date: Mon, 20 Apr 2026 18:26:52 GMT
• Title: Investigating Counterfactual Unfairness in LLMs towards Identities through Humor
• Abstract summary: Humor holds up a mirror to social perception: what we find funny often reflects who we are and how we judge others.<n>In this paper, we investigate counterfactual unfairness through humor by observing how the model's responses change when we swap who speaks.<n> Experiments across state-of-the-art models reveal consistent relational disparities.
• Score: 38.71109030688384
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Humor holds up a mirror to social perception: what we find funny often reflects who we are and how we judge others. When language models engage with humor, their reactions expose the social assumptions they have internalized from training data. In this paper, we investigate counterfactual unfairness through humor by observing how the model's responses change when we swap who speaks and who is addressed while holding other factors constant. Our framework spans three tasks: humor generation refusal, speaker intention inference, and relational/societal impact prediction, covering both identity-agnostic humor and identity-specific disparagement humor. We introduce interpretable bias metrics that capture asymmetric patterns under identity swaps. Experiments across state-of-the-art models reveal consistent relational disparities: jokes told by privileged speakers are refused up to 67.5% more often, judged as malicious 64.7% more frequently, and rated up to 1.5 points higher in social harm on a 5-point scale. These patterns highlight how sensitivity and stereotyping coexist in generative models, complicating efforts toward fairness and cultural alignment.

Related papers

• Engagement Undermines Safety: How Stereotypes and Toxicity Shape Humor in Language Models [55.98686105081078]
  Large language models are increasingly used for creative writing and engagement content, raising safety concerns about the outputs.<n>This work evaluates how funniness optimization in modern LLM pipelines couples with harmful content by measuring humor, stereotypicality, and toxicity.
  arXiv  Detail & Related papers  (2025-10-21T09:28:09Z)
• The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMs [60.15472325639723]
  Personality traits have long been studied as predictors of human behavior.<n>Recent advances in Large Language Models (LLMs) suggest similar patterns may emerge in artificial systems.
  arXiv  Detail & Related papers  (2025-09-03T21:27:10Z)
• Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Models [75.85319609088354]
  Sentient Agent as a Judge (SAGE) is an evaluation framework for large language models.<n>SAGE instantiates a Sentient Agent that simulates human-like emotional changes and inner thoughts during interaction.<n>SAGE provides a principled, scalable and interpretable tool for tracking progress toward genuinely empathetic and socially adept language agents.
  arXiv  Detail & Related papers  (2025-05-01T19:06:10Z)
• From Punchlines to Predictions: A Metric to Assess LLM Performance in Identifying Humor in Stand-Up Comedy [6.124881326867511]
  In light of the widespread adoption of Large Language Models, the intersection of humor and AI has become no laughing matter.<n>In this study, we assess the ability of models in accurately identifying humorous quotes from a stand-up comedy transcript.<n>We propose a novel humor detection metric designed to evaluate LLMs amongst various prompts on their capability to extract humorous punchlines.
  arXiv  Detail & Related papers  (2025-04-12T02:19:53Z)
• Where Fact Ends and Fairness Begins: Redefining AI Bias Evaluation through Cognitive Biases [77.3489598315447]
  We argue that identifying the boundary between fact and fair is essential for meaningful fairness evaluation.<n>We introduce Fact-or-Fair, a benchmark with (i) objective queries aligned with descriptive, fact-based judgments, and (ii) subjective queries aligned with normative, fairness-based judgments.
  arXiv  Detail & Related papers  (2025-02-09T10:54:11Z)
• THInC: A Theory-Driven Framework for Computational Humor Detection [2.0960189135529212]
  There is still no agreement on a single, comprehensive humor theory. Most computational approaches to detecting humor are not based on existing humor theories. This paper contributes to bridging this long-standing gap by creating an interpretable framework for humor classification.
  arXiv  Detail & Related papers  (2024-09-02T13:09:26Z)
• Moral Sparks in Social Media Narratives [14.025768295979184]
  We examine interactions on social media to understand human moral judgments in real-life ethical scenarios. Specifically, we examine posts from a popular Reddit subreddit (i.e., a subcommunity) called r/AmITheAsshole.
  arXiv  Detail & Related papers  (2023-10-30T05:03:26Z)
• The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarks [75.58692290694452]
  We compare social biases with non-social biases stemming from choices made during dataset construction that might not even be discernible to the human eye. We observe that these shallow modifications have a surprising effect on the resulting degree of bias across various models.
  arXiv  Detail & Related papers  (2022-10-18T17:58:39Z)
• Uncertainty and Surprisal Jointly Deliver the Punchline: Exploiting Incongruity-Based Features for Humor Recognition [0.6445605125467573]
  We break down any joke into two distinct components: the set-up and the punchline. Inspired by the incongruity theory of humor, we model the set-up as the part developing semantic uncertainty. With increasingly powerful language models, we were able to feed the set-up along with the punchline into the GPT-2 language model.
  arXiv  Detail & Related papers  (2020-12-22T13:48:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.