WATCHED: A Web AI Agent Tool for Combating Hate Speech by Expanding Data

• URL: http://arxiv.org/abs/2509.01379v1
• Date: Mon, 01 Sep 2025 11:26:46 GMT
• Title: WATCHED: A Web AI Agent Tool for Combating Hate Speech by Expanding Data
• Authors: Paloma Piot, Diego Sánchez, Javier Parapar,
• Abstract summary: Online harms are a growing problem in digital spaces, putting user safety at risk and reducing trust in social media platforms.<n>To address this, we need tools that combine the speed and scale of automated systems with the judgment and insight of human moderators.<n>These tools should not only find harmful content but also explain their decisions clearly, helping to build trust and understanding.
• Score: 5.127121704630949
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Online harms are a growing problem in digital spaces, putting user safety at risk and reducing trust in social media platforms. One of the most persistent forms of harm is hate speech. To address this, we need tools that combine the speed and scale of automated systems with the judgment and insight of human moderators. These tools should not only find harmful content but also explain their decisions clearly, helping to build trust and understanding. In this paper, we present WATCHED, a chatbot designed to support content moderators in tackling hate speech. The chatbot is built as an Artificial Intelligence Agent system that uses Large Language Models along with several specialised tools. It compares new posts with real examples of hate speech and neutral content, uses a BERT-based classifier to help flag harmful messages, looks up slang and informal language using sources like Urban Dictionary, generates chain-of-thought reasoning, and checks platform guidelines to explain and support its decisions. This combination allows the chatbot not only to detect hate speech but to explain why content is considered harmful, grounded in both precedent and policy. Experimental results show that our proposed method surpasses existing state-of-the-art methods, reaching a macro F1 score of 0.91. Designed for moderators, safety teams, and researchers, the tool helps reduce online harms by supporting collaboration between AI and human oversight.

Related papers

• Synthetic Voices, Real Threats: Evaluating Large Text-to-Speech Models in Generating Harmful Audio [63.18443674004945]
  This work explores a content-centric threat: exploiting TTS systems to produce speech containing harmful content.<n>We present HARMGEN, a suite of five attacks organized into two families that address these challenges.
  arXiv  Detail & Related papers  (2025-11-14T03:00:04Z)
• Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboards [93.16294577018482]
  Arena, the most popular benchmark of this type, ranks models by asking users to select the better response between two randomly selected models.<n>We show that an attacker can alter the leaderboard (to promote their favorite model or demote competitors) at the cost of roughly a thousand votes.<n>Our attack consists of two steps: first, we show how an attacker can determine which model was used to generate a given reply with more than $95%$ accuracy; and then, the attacker can use this information to consistently vote against a target model.
  arXiv  Detail & Related papers  (2025-01-13T17:12:38Z)
• A Hate Speech Moderated Chat Application: Use Case for GDPR and DSA Compliance [0.0]
  This research presents a novel application capable of implementing legal and ethical reasoning into the content moderation process. Two use cases fundamental to online communication are presented and implemented using technologies such as GPT-3.5, Solid Pods, and the rule language Prova. The work proposes a novel approach to reason within different legal and ethical definitions of hate speech and plan the fitting counter hate speech.
  arXiv  Detail & Related papers  (2024-10-10T08:28:38Z)
• SWE2: SubWord Enriched and Significant Word Emphasized Framework for Hate Speech Detection [3.0460060805145517]
  We propose a novel hate speech detection framework called SWE2, which only relies on the content of messages and automatically identifies hate speech. Experimental results show that our proposed model achieves 0.975 accuracy and 0.953 macro F1, outperforming 7 state-of-the-art baselines.
  arXiv  Detail & Related papers  (2024-09-25T07:05:44Z)
• ViTHSD: Exploiting Hatred by Targets for Hate Speech Detection on Vietnamese Social Media Texts [0.0]
  We first introduce the ViTHSD - a targeted hate speech detection dataset for Vietnamese Social Media Texts.<n>The dataset contains 10K comments, each comment is labeled to specific targets with three levels: clean, offensive, and hate.<n>The inter-annotator agreement obtained from the dataset is 0.45 by Cohen's Kappa index, which is indicated as a moderate level.
  arXiv  Detail & Related papers  (2024-04-30T04:16:55Z)
• CoSyn: Detecting Implicit Hate Speech in Online Conversations Using a Context Synergized Hyperbolic Network [52.85130555886915]
  CoSyn is a context-synergized neural network that explicitly incorporates user- and conversational context for detecting implicit hate speech in online conversations. We show that CoSyn outperforms all our baselines in detecting implicit hate speech with absolute improvements in the range of 1.24% - 57.8%.
  arXiv  Detail & Related papers  (2023-03-02T17:30:43Z)
• A Categorical Archive of ChatGPT Failures [47.64219291655723]
  ChatGPT, developed by OpenAI, has been trained using massive amounts of data and simulates human conversation. It has garnered significant attention due to its ability to effectively answer a broad range of human inquiries. However, a comprehensive analysis of ChatGPT's failures is lacking, which is the focus of this study.
  arXiv  Detail & Related papers  (2023-02-06T04:21:59Z)
• Countering Malicious Content Moderation Evasion in Online Social Networks: Simulation and Detection of Word Camouflage [64.78260098263489]
  Twisting and camouflaging keywords are among the most used techniques to evade platform content moderation systems. This article contributes significantly to countering malicious information by developing multilingual tools to simulate and detect new methods of evasion of content.
  arXiv  Detail & Related papers  (2022-12-27T16:08:49Z)
• Assessing the impact of contextual information in hate speech detection [0.48369513656026514]
  We provide a novel corpus for contextualized hate speech detection based on user responses to news posts from media outlets on Twitter. This corpus was collected in the Rioplatense dialectal variety of Spanish and focuses on hate speech associated with the COVID-19 pandemic.
  arXiv  Detail & Related papers  (2022-10-02T09:04:47Z)
• Why So Toxic? Measuring and Triggering Toxic Behavior in Open-Domain Chatbots [24.84440998820146]
  This paper presents a first-of-its-kind, large-scale measurement of toxicity in chatbots. We show that publicly available chatbots are prone to providing toxic responses when fed toxic queries. We then set out to design and experiment with an attack, ToxicBuddy, which relies on fine-tuning GPT-2 to generate non-toxic queries.
  arXiv  Detail & Related papers  (2022-09-07T20:45:41Z)
• Hate Speech Classification Using SVM and Naive BAYES [0.0]
  Many countries have developed laws to avoid online hate speech. But as online content continues to grow, so does the spread of hate speech. It is important to automatically process the online user contents to detect and remove hate speech.
  arXiv  Detail & Related papers  (2022-03-21T17:15:38Z)
• Speaker De-identification System using Autoencoders and Adversarial Training [58.720142291102135]
  We propose a speaker de-identification system based on adversarial training and autoencoders. Experimental results show that combining adversarial learning and autoencoders increase the equal error rate of a speaker verification system.
  arXiv  Detail & Related papers  (2020-11-09T19:22:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.