NoisyHate: Mining Online Human-Written Perturbations for Realistic Robustness Benchmarking of Content Moderation Models

• URL: http://arxiv.org/abs/2303.10430v2
• Date: Mon, 28 Apr 2025 15:25:39 GMT
• Title: NoisyHate: Mining Online Human-Written Perturbations for Realistic Robustness Benchmarking of Content Moderation Models
• Authors: Yiran Ye, Thai Le, Dongwon Lee,
• Abstract summary: We introduce a novel, high-quality dataset of human-written perturbations, named as NoisyHate.<n>We show that perturbations in NoisyHate have different characteristics than prior algorithm-generated toxic datasets show.
• Score: 13.887401380190335
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Online texts with toxic content are a clear threat to the users on social media in particular and society in general. Although many platforms have adopted various measures (e.g., machine learning-based hate-speech detection systems) to diminish their effect, toxic content writers have also attempted to evade such measures by using cleverly modified toxic words, so-called human-written text perturbations. Therefore, to help build automatic detection tools to recognize those perturbations, prior methods have developed sophisticated techniques to generate diverse adversarial samples. However, we note that these ``algorithms"-generated perturbations do not necessarily capture all the traits of ``human"-written perturbations. Therefore, in this paper, we introduce a novel, high-quality dataset of human-written perturbations, named as NoisyHate, that was created from real-life perturbations that are both written and verified by human-in-the-loop. We show that perturbations in NoisyHate have different characteristics than prior algorithm-generated toxic datasets show, and thus can be in particular useful to help develop better toxic speech detection solutions. We thoroughly validate NoisyHate against state-of-the-art language models, such as BERT and RoBERTa, and black box APIs, such as Perspective API, on two tasks, such as perturbation normalization and understanding.

Related papers

• A Knowledge-guided Adversarial Defense for Resisting Malicious Visual Manipulation [93.28532038721816]
  Malicious applications of visual manipulation have raised serious threats to the security and reputation of users in many fields. We propose a knowledge-guided adversarial defense (KGAD) to actively force malicious manipulation models to output semantically confusing samples.
  arXiv  Detail & Related papers  (2025-04-11T10:18:13Z)
• Evolving Hate Speech Online: An Adaptive Framework for Detection and Mitigation [18.459726677931023]
  We present an adaptive approach that uses word embeddings to update lexicons and develop a hybrid model that adjusts to emerging slurs and new linguistic patterns. Our hybrid model, which combines BERT with lexicon-based techniques, achieves an accuracy of 95% for most state-of-the-art datasets.
  arXiv  Detail & Related papers  (2025-02-15T22:46:50Z)
• Toxicity Detection towards Adaptability to Changing Perturbations [21.989281174371147]
  In this paper, we introduce a novel problem, i.e., continual learning jailbreak perturbation patterns, into the toxicity detection field. We first construct a new dataset generated by 9 types of perturbation patterns, 7 of them are summarized from prior work and 2 of them are developed by us. We then systematically validate the vulnerability of current methods on this new perturbation pattern-aware dataset.
  arXiv  Detail & Related papers  (2024-12-17T05:04:57Z)
• ToxiCloakCN: Evaluating Robustness of Offensive Language Detection in Chinese with Cloaking Perturbations [6.360597788845826]
  This study examines the limitations of state-of-the-art large language models (LLMs) in identifying offensive content within systematically perturbed data. Our work highlights the urgent need for more advanced techniques in offensive language detection to combat the evolving tactics used to evade detection mechanisms.
  arXiv  Detail & Related papers  (2024-06-18T02:44:56Z)
• Navigating the Shadows: Unveiling Effective Disturbances for Modern AI Content Detectors [24.954755569786396]
  AI-text detection has emerged to distinguish between human and machine-generated content. Recent research indicates that these detection systems often lack robustness and struggle to effectively differentiate perturbed texts. Our work simulates real-world scenarios in both informal and professional writing, exploring the out-of-the-box performance of current detectors.
  arXiv  Detail & Related papers  (2024-06-13T08:37:01Z)
• Humanizing Machine-Generated Content: Evading AI-Text Detection through Adversarial Attack [24.954755569786396]
  We propose a framework for a broader class of adversarial attacks, designed to perform minor perturbations in machine-generated content to evade detection. We consider two attack settings: white-box and black-box, and employ adversarial learning in dynamic scenarios to assess the potential enhancement of the current detection model's robustness. The empirical results reveal that the current detection models can be compromised in as little as 10 seconds, leading to the misclassification of machine-generated text as human-written content.
  arXiv  Detail & Related papers  (2024-04-02T12:49:22Z)
• Assaying on the Robustness of Zero-Shot Machine-Generated Text Detectors [57.7003399760813]
  We explore advanced Large Language Models (LLMs) and their specialized variants, contributing to this field in several ways. We uncover a significant correlation between topics and detection performance. These investigations shed light on the adaptability and robustness of these detection methods across diverse topics.
  arXiv  Detail & Related papers  (2023-12-20T10:53:53Z)
• DEMASQ: Unmasking the ChatGPT Wordsmith [63.8746084667206]
  We propose an effective ChatGPT detector named DEMASQ, which accurately identifies ChatGPT-generated content. Our method addresses two critical factors: (i) the distinct biases in text composition observed in human- and machine-generated content and (ii) the alterations made by humans to evade previous detection methods.
  arXiv  Detail & Related papers  (2023-11-08T21:13:05Z)
• Fine-Tuning Llama 2 Large Language Models for Detecting Online Sexual Predatory Chats and Abusive Texts [2.406214748890827]
  This paper proposes an approach to detection of online sexual predatory chats and abusive language using the open-source pretrained Llama 2 7B- parameter model. We fine-tune the LLM using datasets with different sizes, imbalance degrees, and languages (i.e., English, Roman Urdu and Urdu) Experimental results show a strong performance of the proposed approach, which performs proficiently and consistently across three distinct datasets.
  arXiv  Detail & Related papers  (2023-08-28T16:18:50Z)
• Watermarking Conditional Text Generation for AI Detection: Unveiling Challenges and a Semantic-Aware Watermark Remedy [52.765898203824975]
  We introduce a semantic-aware watermarking algorithm that considers the characteristics of conditional text generation and the input context. Experimental results demonstrate that our proposed method yields substantial improvements across various text generation models.
  arXiv  Detail & Related papers  (2023-07-25T20:24:22Z)
• MISMATCH: Fine-grained Evaluation of Machine-generated Text with Mismatch Error Types [68.76742370525234]
  We propose a new evaluation scheme to model human judgments in 7 NLP tasks, based on the fine-grained mismatches between a pair of texts. Inspired by the recent efforts in several NLP tasks for fine-grained evaluation, we introduce a set of 13 mismatch error types. We show that the mismatch errors between the sentence pairs on the held-out datasets from 7 NLP tasks align well with the human evaluation.
  arXiv  Detail & Related papers  (2023-06-18T01:38:53Z)
• Verifying the Robustness of Automatic Credibility Assessment [50.55687778699995]
  We show that meaning-preserving changes in input text can mislead the models. We also introduce BODEGA: a benchmark for testing both victim models and attack methods on misinformation detection tasks. Our experimental results show that modern large language models are often more vulnerable to attacks than previous, smaller solutions.
  arXiv  Detail & Related papers  (2023-03-14T16:11:47Z)
• Countering Malicious Content Moderation Evasion in Online Social Networks: Simulation and Detection of Word Camouflage [64.78260098263489]
  Twisting and camouflaging keywords are among the most used techniques to evade platform content moderation systems. This article contributes significantly to countering malicious information by developing multilingual tools to simulate and detect new methods of evasion of content.
  arXiv  Detail & Related papers  (2022-12-27T16:08:49Z)
• Combating high variance in Data-Scarce Implicit Hate Speech Classification [0.0]
  We develop a novel RoBERTa-based model that achieves state-of-the-art performance. In this paper, we explore various optimization and regularization techniques and develop a novel RoBERTa-based model that achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-08-29T13:45:21Z)
• Panning for gold: Lessons learned from the platform-agnostic automated detection of political content in textual data [48.7576911714538]
  We discuss how these techniques can be used to detect political content across different platforms. We compare the performance of three groups of detection techniques relying on dictionaries, supervised machine learning, or neural networks. Our results show the limited impact of preprocessing on model performance, with the best results for less noisy data being achieved by neural network- and machine-learning-based models.
  arXiv  Detail & Related papers  (2022-07-01T15:23:23Z)
• APEACH: Attacking Pejorative Expressions with Analysis on Crowd-Generated Hate Speech Evaluation Datasets [4.034948808542701]
  APEACH is a method that allows the collection of hate speech generated by unspecified users. By controlling the crowd-generation of hate speech and adding only a minimum post-labeling, we create a corpus that enables the generalizable and fair evaluation of hate speech detection.
  arXiv  Detail & Related papers  (2022-02-25T02:04:38Z)
• Deep Learning for Hate Speech Detection: A Comparative Study [54.42226495344908]
  We present here a large-scale empirical comparison of deep and shallow hate-speech detection methods. Our goal is to illuminate progress in the area, and identify strengths and weaknesses in the current state-of-the-art. In doing so we aim to provide guidance as to the use of hate-speech detection in practice, quantify the state-of-the-art, and identify future research directions.
  arXiv  Detail & Related papers  (2022-02-19T03:48:20Z)
• AES Systems Are Both Overstable And Oversensitive: Explaining Why And Proposing Defenses [66.49753193098356]
  We investigate the reason behind the surprising adversarial brittleness of scoring models. Our results indicate that autoscoring models, despite getting trained as "end-to-end" models, behave like bag-of-words models. We propose detection-based protection models that can detect oversensitivity and overstability causing samples with high accuracies.
  arXiv  Detail & Related papers  (2021-09-24T03:49:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.