Toxicity Detection towards Adaptability to Changing Perturbations

• URL: http://arxiv.org/abs/2412.15267v3
• Date: Tue, 04 Mar 2025 04:49:58 GMT
• Title: Toxicity Detection towards Adaptability to Changing Perturbations
• Authors: Hankun Kang, Jianhao Chen, Yongqi Li, Xin Miao, Mayi Xu, Ming Zhong, Yuanyuan Zhu, Tieyun Qian,
• Abstract summary: In this paper, we introduce a novel problem, i.e., continual learning jailbreak perturbation patterns, into the toxicity detection field.<n>We first construct a new dataset generated by 9 types of perturbation patterns, 7 of them are summarized from prior work and 2 of them are developed by us.<n>We then systematically validate the vulnerability of current methods on this new perturbation pattern-aware dataset.
• Score: 21.989281174371147
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Toxicity detection is crucial for maintaining the peace of the society. While existing methods perform well on normal toxic contents or those generated by specific perturbation methods, they are vulnerable to evolving perturbation patterns. However, in real-world scenarios, malicious users tend to create new perturbation patterns for fooling the detectors. For example, some users may circumvent the detector of large language models (LLMs) by adding `I am a scientist' at the beginning of the prompt. In this paper, we introduce a novel problem, i.e., continual learning jailbreak perturbation patterns, into the toxicity detection field. To tackle this problem, we first construct a new dataset generated by 9 types of perturbation patterns, 7 of them are summarized from prior work and 2 of them are developed by us. We then systematically validate the vulnerability of current methods on this new perturbation pattern-aware dataset via both the zero-shot and fine tuned cross-pattern detection. Upon this, we present the domain incremental learning paradigm and the corresponding benchmark to ensure the detector's robustness to dynamically emerging types of perturbed toxic text. Our code and dataset are provided in the appendix and will be publicly available at GitHub, by which we wish to offer new research opportunities for the security-relevant communities.

Related papers

• Ensuring Medical AI Safety: Explainable AI-Driven Detection and Mitigation of Spurious Model Behavior and Associated Data [14.991686165405959]
  We introduce a semi-automated framework for the identification of spurious behavior from both data and model perspective. This allows the retrieval of spurious data points and the detection of model circuits that encode the associated prediction rules. We show the applicability of our framework using four medical datasets, featuring controlled and real-world spurious correlations.
  arXiv  Detail & Related papers  (2025-01-23T16:39:09Z)
• Open-Set Deepfake Detection: A Parameter-Efficient Adaptation Method with Forgery Style Mixture [58.60915132222421]
  We introduce an approach that is both general and parameter-efficient for face forgery detection. We design a forgery-style mixture formulation that augments the diversity of forgery source domains. We show that the designed model achieves state-of-the-art generalizability with significantly reduced trainable parameters.
  arXiv  Detail & Related papers  (2024-08-23T01:53:36Z)
• Humanizing Machine-Generated Content: Evading AI-Text Detection through Adversarial Attack [24.954755569786396]
  We propose a framework for a broader class of adversarial attacks, designed to perform minor perturbations in machine-generated content to evade detection. We consider two attack settings: white-box and black-box, and employ adversarial learning in dynamic scenarios to assess the potential enhancement of the current detection model's robustness. The empirical results reveal that the current detection models can be compromised in as little as 10 seconds, leading to the misclassification of machine-generated text as human-written content.
  arXiv  Detail & Related papers  (2024-04-02T12:49:22Z)
• Does DetectGPT Fully Utilize Perturbation? Bridging Selective Perturbation to Fine-tuned Contrastive Learning Detector would be Better [21.901523394933076]
  We propose a novel fine-tuned detector, Pecola, bridging metric-based and fine-tuned methods by contrastive learning on selective perturbation. Experiments show that Pecola outperforms the state-of-the-art (SOTA) by 1.20% in accuracy on average on four public datasets.
  arXiv  Detail & Related papers  (2024-02-01T01:23:07Z)
• Assaying on the Robustness of Zero-Shot Machine-Generated Text Detectors [57.7003399760813]
  We explore advanced Large Language Models (LLMs) and their specialized variants, contributing to this field in several ways. We uncover a significant correlation between topics and detection performance. These investigations shed light on the adaptability and robustness of these detection methods across diverse topics.
  arXiv  Detail & Related papers  (2023-12-20T10:53:53Z)
• How adversarial attacks can disrupt seemingly stable accurate classifiers [76.95145661711514]
  Adversarial attacks dramatically change the output of an otherwise accurate learning system using a seemingly inconsequential modification to a piece of input data. Here, we show that this may be seen as a fundamental feature of classifiers working with high dimensional input data. We introduce a simple generic and generalisable framework for which key behaviours observed in practical systems arise with high probability.
  arXiv  Detail & Related papers  (2023-09-07T12:02:00Z)
• A Robust Likelihood Model for Novelty Detection [8.766411351797883]
  Current approaches to novelty or anomaly detection are based on deep neural networks. We propose a new prior that aims at learning a robust likelihood for the novelty test, as a defense against attacks. We also integrate the same prior with a state-of-the-art novelty detection approach.
  arXiv  Detail & Related papers  (2023-06-06T01:02:31Z)
• Red Teaming Language Model Detectors with Language Models [114.36392560711022]
  Large language models (LLMs) present significant safety and ethical risks if exploited by malicious users. Recent works have proposed algorithms to detect LLM-generated text and protect LLMs. We study two types of attack strategies: 1) replacing certain words in an LLM's output with their synonyms given the context; 2) automatically searching for an instructional prompt to alter the writing style of the generation.
  arXiv  Detail & Related papers  (2023-05-31T10:08:37Z)
• Few-shot Weakly-supervised Cybersecurity Anomaly Detection [1.179179628317559]
  We propose an enhancement to an existing few-shot weakly-supervised deep learning anomaly detection framework. This framework incorporates data augmentation, representation learning and ordinal regression. We then evaluated and showed the performance of our implemented framework on three benchmark datasets.
  arXiv  Detail & Related papers  (2023-04-15T04:37:54Z)
• Enhancing Multiple Reliability Measures via Nuisance-extended Information Bottleneck [77.37409441129995]
  In practical scenarios where training data is limited, many predictive signals in the data can be rather from some biases in data acquisition. We consider an adversarial threat model under a mutual information constraint to cover a wider class of perturbations in training. We propose an autoencoder-based training to implement the objective, as well as practical encoder designs to facilitate the proposed hybrid discriminative-generative training.
  arXiv  Detail & Related papers  (2023-03-24T16:03:21Z)
• NoisyHate: Mining Online Human-Written Perturbations for Realistic Robustness Benchmarking of Content Moderation Models [13.887401380190335]
  We introduce a novel, high-quality dataset of human-written perturbations, named as NoisyHate. We show that perturbations in NoisyHate have different characteristics than prior algorithm-generated toxic datasets show.
  arXiv  Detail & Related papers  (2023-03-18T14:54:57Z)
• Adversarially Robust One-class Novelty Detection [83.1570537254877]
  We show that existing novelty detectors are susceptible to adversarial examples. We propose a defense strategy that manipulates the latent space of novelty detectors to improve the robustness against adversarial examples.
  arXiv  Detail & Related papers  (2021-08-25T10:41:29Z)
• No Need to Know Physics: Resilience of Process-based Model-free Anomaly Detection for Industrial Control Systems [95.54151664013011]
  We present a novel framework to generate adversarial spoofing signals that violate physical properties of the system. We analyze four anomaly detectors published at top security conferences.
  arXiv  Detail & Related papers  (2020-12-07T11:02:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.