Lost in Moderation: How Commercial Content Moderation APIs Over- and Under-Moderate Group-Targeted Hate Speech and Linguistic Variations

• URL: http://arxiv.org/abs/2503.01623v1
• Date: Mon, 03 Mar 2025 14:56:47 GMT
• Title: Lost in Moderation: How Commercial Content Moderation APIs Over- and Under-Moderate Group-Targeted Hate Speech and Linguistic Variations
• Abstract summary: Commercial content moderation APIs are marketed as scalable solutions to combat online hate speech.<n>The reliance on these APIs risks both silencing legitimate speech and failing to protect online platforms from harmful speech.<n>This paper introduces a framework for auditing black-box NLP systems.
• Score: 5.277756703318046
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Commercial content moderation APIs are marketed as scalable solutions to combat online hate speech. However, the reliance on these APIs risks both silencing legitimate speech, called over-moderation, and failing to protect online platforms from harmful speech, known as under-moderation. To assess such risks, this paper introduces a framework for auditing black-box NLP systems. Using the framework, we systematically evaluate five widely used commercial content moderation APIs. Analyzing five million queries based on four datasets, we find that APIs frequently rely on group identity terms, such as ``black'', to predict hate speech. While OpenAI's and Amazon's services perform slightly better, all providers under-moderate implicit hate speech, which uses codified messages, especially against LGBTQIA+ individuals. Simultaneously, they over-moderate counter-speech, reclaimed slurs and content related to Black, LGBTQIA+, Jewish, and Muslim people. We recommend that API providers offer better guidance on API implementation and threshold setting and more transparency on their APIs' limitations. Warning: This paper contains offensive and hateful terms and concepts. We have chosen to reproduce these terms for reasons of transparency.

Related papers

• IYKYK (But AI Doesn't): Automated Content Moderation Does Not Capture Communities' Heterogeneous Attitudes Towards Reclaimed Language [45.4201325387611]
  We use quantitative and qualitative methods to examine attitudes of social media users in LGBTQIA+, Black, and women communities around reclaimed slurs.<n>We observe low inter-annotator agreement, indicating substantial disagreement among in-group annotators.<n>We find poor alignment between annotator judgments and automated hate speech assessments produced by Perspective API.
  arXiv  Detail & Related papers  (2026-04-17T19:19:57Z)
• Explain the Flag: Contextualizing Hate Speech Beyond Censorship [2.796818629124347]
  We present a hybrid approach that combines Large Language Models (LLMs) with three newly created and curated vocabularies to detect and explain hate speech in English, French, and Greek.<n>Our system captures both inherently derogatory expressions tied to identity characteristics and direct group-targeted content through two complementary pipelines.<n>The outputs are fused into grounded explanations that clarify why content is flagged.
  arXiv  Detail & Related papers  (2026-04-16T13:06:28Z)
• More Than Sum of Its Parts: Deciphering Intent Shifts in Multimodal Hate Speech Detection [49.07221734365168]
  Hate speech on social media is critical for securing cyberspace, yet relies heavily on automated detection systems.<n>We characterize semantic intent shifts where modalities interact to construct implicit hate from benign cues or neutralize toxicity through semantic inversion.<n>We propose the Asymmetric Reasoning via Courtroom Agent DEbate framework to effectively decipher these complex cues.
  arXiv  Detail & Related papers  (2026-03-22T15:46:23Z)
• Synthetic Voices, Real Threats: Evaluating Large Text-to-Speech Models in Generating Harmful Audio [63.18443674004945]
  This work explores a content-centric threat: exploiting TTS systems to produce speech containing harmful content.<n>We present HARMGEN, a suite of five attacks organized into two families that address these challenges.
  arXiv  Detail & Related papers  (2025-11-14T03:00:04Z)
• Digital Guardians: Can GPT-4, Perspective API, and Moderation API reliably detect hate speech in reader comments of German online newspapers? [36.636686462749275]
  GPT-4o is compared with Google's Perspective API and OpenAI's Moderation API.<n>For GPT-4o, three different promptings are used, employing a Zero-Shot, One-Shot, and Few-Shot approach.
  arXiv  Detail & Related papers  (2025-01-02T13:48:56Z)
• A Hate Speech Moderated Chat Application: Use Case for GDPR and DSA Compliance [0.0]
  This research presents a novel application capable of implementing legal and ethical reasoning into the content moderation process. Two use cases fundamental to online communication are presented and implemented using technologies such as GPT-3.5, Solid Pods, and the rule language Prova. The work proposes a novel approach to reason within different legal and ethical definitions of hate speech and plan the fitting counter hate speech.
  arXiv  Detail & Related papers  (2024-10-10T08:28:38Z)
• Demarked: A Strategy for Enhanced Abusive Speech Moderation through Counterspeech, Detoxification, and Message Management [71.99446449877038]
  We propose a more comprehensive approach called Demarcation scoring abusive speech based on four aspect -- (i) severity scale; (ii) presence of a target; (iii) context scale; (iv) legal scale. Our work aims to inform future strategies for effectively addressing abusive speech online.
  arXiv  Detail & Related papers  (2024-06-27T21:45:33Z)
• ViTHSD: Exploiting Hatred by Targets for Hate Speech Detection on Vietnamese Social Media Texts [0.0]
  We first introduce the ViTHSD - a targeted hate speech detection dataset for Vietnamese Social Media Texts.<n>The dataset contains 10K comments, each comment is labeled to specific targets with three levels: clean, offensive, and hate.<n>The inter-annotator agreement obtained from the dataset is 0.45 by Cohen's Kappa index, which is indicated as a moderate level.
  arXiv  Detail & Related papers  (2024-04-30T04:16:55Z)
• Towards Interpretable Hate Speech Detection using Large Language Model-extracted Rationales [15.458557611029518]
  Social media platforms are a prominent arena for users to engage in interpersonal discussions and express opinions. There arises a need to automatically identify and flag instances of hate speech. We propose to use state-of-the-art Large Language Models (LLMs) to extract features in the form of rationales from the input text.
  arXiv  Detail & Related papers  (2024-03-19T03:22:35Z)
• HateModerate: Testing Hate Speech Detectors against Content Moderation Policies [6.893854392439938]
  We create HateModerate, a dataset for testing the behaviors of automated content moderators against content policies. We test the performance of state-of-the-art hate speech detectors against HateModerate. We observe significant improvement in the models' conformity to content policies while having comparable scores on the original test data.
  arXiv  Detail & Related papers  (2023-07-23T20:08:38Z)
• Analyzing Norm Violations in Live-Stream Chat [49.120561596550395]
  We study the first NLP study dedicated to detecting norm violations in conversations on live-streaming platforms. We define norm violation categories in live-stream chats and annotate 4,583 moderated comments from Twitch. Our results show that appropriate contextual information can boost moderation performance by 35%.
  arXiv  Detail & Related papers  (2023-05-18T05:58:27Z)
• A User-Driven Framework for Regulating and Auditing Social Media [94.70018274127231]
  We propose that algorithmic filtering should be regulated with respect to a flexible, user-driven baseline. We require that the feeds a platform filters contain "similar" informational content as their respective baseline feeds. We present an auditing procedure that checks whether a platform honors this requirement.
  arXiv  Detail & Related papers  (2023-04-20T17:53:34Z)
• CoSyn: Detecting Implicit Hate Speech in Online Conversations Using a Context Synergized Hyperbolic Network [52.85130555886915]
  CoSyn is a context-synergized neural network that explicitly incorporates user- and conversational context for detecting implicit hate speech in online conversations. We show that CoSyn outperforms all our baselines in detecting implicit hate speech with absolute improvements in the range of 1.24% - 57.8%.
  arXiv  Detail & Related papers  (2023-03-02T17:30:43Z)
• Beyond Plain Toxic: Detection of Inappropriate Statements on Flammable Topics for the Russian Language [76.58220021791955]
  We present two text collections labelled according to binary notion of inapropriateness and a multinomial notion of sensitive topic. To objectivise the notion of inappropriateness, we define it in a data-driven way though crowdsourcing.
  arXiv  Detail & Related papers  (2022-03-04T15:59:06Z)
• Addressing the Challenges of Cross-Lingual Hate Speech Detection [115.1352779982269]
  In this paper we focus on cross-lingual transfer learning to support hate speech detection in low-resource languages. We leverage cross-lingual word embeddings to train our neural network systems on the source language and apply it to the target language. We investigate the issue of label imbalance of hate speech datasets, since the high ratio of non-hate examples compared to hate examples often leads to low model performance.
  arXiv  Detail & Related papers  (2022-01-15T20:48:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.