Watch Your Language: Investigating Content Moderation with Large Language Models

• URL: http://arxiv.org/abs/2309.14517v2
• Date: Wed, 17 Jan 2024 17:41:18 GMT
• Title: Watch Your Language: Investigating Content Moderation with Large Language Models
• Abstract summary: Large language models (LLMs) have exploded in popularity due to their ability to perform a wide array of natural language tasks. We evaluate a suite of commodity LLMs on two common content moderation tasks: rule-based community moderation and toxic content detection.
• Score: 5.306187298781318
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) have exploded in popularity due to their ability to perform a wide array of natural language tasks. Text-based content moderation is one LLM use case that has received recent enthusiasm, however, there is little research investigating how LLMs perform in content moderation settings. In this work, we evaluate a suite of commodity LLMs on two common content moderation tasks: rule-based community moderation and toxic content detection. For rule-based community moderation, we instantiate 95 subcommunity specific LLMs by prompting GPT-3.5 with rules from 95 Reddit subcommunities. We find that GPT-3.5 is effective at rule-based moderation for many communities, achieving a median accuracy of 64% and a median precision of 83%. For toxicity detection, we evaluate a suite of commodity LLMs (GPT-3, GPT-3.5, GPT-4, Gemini Pro, LLAMA 2) and show that LLMs significantly outperform currently widespread toxicity classifiers. However, recent increases in model size add only marginal benefit to toxicity detection, suggesting a potential performance plateau for LLMs on toxicity detection tasks. We conclude by outlining avenues for future work in studying LLMs and content moderation.

Related papers

• How Much Content Do LLMs Generate That Induces Cognitive Bias in Users? [13.872175096831343]
  Large language models (LLMs) are increasingly integrated into applications ranging from review summarization to medical diagnosis support.<n>We investigate when and how LLMs expose users to biased content and quantify its severity.<n>Our findings show that LLMs expose users to content that changes the sentiment of the context in 21.86% of the cases, hallucinates on post-knowledge-cutoff data questions in 57.33% of the cases, and primacy bias in 5.94% of the cases.
  arXiv  Detail & Related papers  (2025-07-03T21:56:44Z)
• An Empirical Study of Many-to-Many Summarization with Large Language Models [82.10000188179168]
  Large language models (LLMs) have shown strong multi-lingual abilities, giving them the potential to perform Many-to-many summarization (M2MS) in real applications.<n>This work presents a systematic empirical study on LLMs' M2MS ability.
  arXiv  Detail & Related papers  (2025-05-19T11:18:54Z)
• Dialectal Toxicity Detection: Evaluating LLM-as-a-Judge Consistency Across Language Varieties [23.777874316083984]
  There has been little systematic study on how dialectal differences affect toxicity detection by modern LLMs. We create a multi-dialect dataset through synthetic transformations and human-assisted translations, covering 10 language clusters and 60 varieties. We then evaluated three LLMs on their ability to assess toxicity across multilingual, dialectal, and LLM-human consistency.
  arXiv  Detail & Related papers  (2024-11-17T03:53:24Z)
• SLM-Mod: Small Language Models Surpass LLMs at Content Moderation [13.759689015188698]
  Large language models (LLMs) have shown promise in many natural language understanding tasks, including content moderation. We explore the use of open-source small language models (SLMs) for community-specific content moderation tasks. Using 150K comments from 15 popular Reddit communities, we find that SLMs outperform zero-shot LLMs at content moderation.
  arXiv  Detail & Related papers  (2024-10-17T02:16:37Z)
• GIVE: Structured Reasoning of Large Language Models with Knowledge Graph Inspired Veracity Extrapolation [108.2008975785364]
  Graph Inspired Veracity Extrapolation (GIVE) is a novel reasoning method that merges parametric and non-parametric memories to improve accurate reasoning with minimal external input. GIVE guides the LLM agent to select the most pertinent expert data (observe), engage in query-specific divergent thinking (reflect), and then synthesize this information to produce the final output (speak)
  arXiv  Detail & Related papers  (2024-10-11T03:05:06Z)
• Toxic Subword Pruning for Dialogue Response Generation on Large Language Models [51.713448010799986]
  We propose textbfToxic Subword textbfPruning (ToxPrune) to prune the subword contained by the toxic words from BPE in trained LLMs. ToxPrune simultaneously improves the toxic language model NSFW-3B on the task of dialogue response generation obviously.
  arXiv  Detail & Related papers  (2024-10-05T13:30:33Z)
• Testing and Evaluation of Large Language Models: Correctness, Non-Toxicity, and Fairness [30.632260870411177]
  Large language models (LLMs) have rapidly penetrated into people's work and daily lives over the past few years. This thesis focuses on the correctness, non-toxicity, and fairness of LLMs from both software testing and natural language processing perspectives.
  arXiv  Detail & Related papers  (2024-08-31T22:21:04Z)
• Pushing The Limit of LLM Capacity for Text Classification [27.684335455517417]
  We propose RGPT, an adaptive boosting framework tailored to produce a specialized text classification LLM. We show that RGPT significantly outperforms 8 SOTA PLMs and 7 SOTA LLMs on four benchmarks by 1.36% on average.
  arXiv  Detail & Related papers  (2024-02-12T08:14:03Z)
• Large Language Models: A Survey [69.72787936480394]
  Large Language Models (LLMs) have drawn a lot of attention due to their strong performance on a wide range of natural language tasks. LLMs' ability of general-purpose language understanding and generation is acquired by training billions of model's parameters on massive amounts of text data.
  arXiv  Detail & Related papers  (2024-02-09T05:37:09Z)
• Supervised Knowledge Makes Large Language Models Better In-context Learners [94.89301696512776]
  Large Language Models (LLMs) exhibit emerging in-context learning abilities through prompt engineering. The challenge of improving the generalizability and factuality of LLMs in natural language understanding and question answering remains under-explored. We propose a framework that enhances the reliability of LLMs as it: 1) generalizes out-of-distribution data, 2) elucidates how LLMs benefit from discriminative models, and 3) minimizes hallucinations in generative tasks.
  arXiv  Detail & Related papers  (2023-12-26T07:24:46Z)
• Unveiling the Implicit Toxicity in Large Language Models [77.90933074675543]
  The open-endedness of large language models (LLMs) combined with their impressive capabilities may lead to new safety issues when being exploited for malicious use. We show that LLMs can generate diverse implicit toxic outputs that are exceptionally difficult to detect via simply zero-shot prompting. We propose a reinforcement learning (RL) based attacking method to further induce the implicit toxicity in LLMs.
  arXiv  Detail & Related papers  (2023-11-29T06:42:36Z)
• LLMRefine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedback [65.84061725174269]
  Recent large language models (LLM) are leveraging human feedback to improve their generation quality. We propose LLMRefine, an inference time optimization method to refine LLM's output. We conduct experiments on three text generation tasks, including machine translation, long-form question answering (QA), and topical summarization. LLMRefine consistently outperforms all baseline approaches, achieving improvements up to 1.7 MetricX points on translation tasks, 8.1 ROUGE-L on ASQA, 2.2 ROUGE-L on topical summarization.
  arXiv  Detail & Related papers  (2023-11-15T19:52:11Z)
• Are Large Language Models Really Robust to Word-Level Perturbations? [68.60618778027694]
  We propose a novel rational evaluation approach that leverages pre-trained reward models as diagnostic tools. Longer conversations manifest the comprehensive grasp of language models in terms of their proficiency in understanding questions. Our results demonstrate that LLMs frequently exhibit vulnerability to word-level perturbations that are commonplace in daily language usage.
  arXiv  Detail & Related papers  (2023-09-20T09:23:46Z)
• You Only Prompt Once: On the Capabilities of Prompt Learning on Large Language Models to Tackle Toxic Content [13.600755614321493]
  We investigate how we can use large language models (LLMs) to tackle the problem of toxic content online. We focus on three tasks; 1) Toxicity Classification, 2) Toxic Span Detection, and 3) Detoxification. We find that prompt learning achieves around 10% improvement in the toxicity classification task compared to the baselines.
  arXiv  Detail & Related papers  (2023-08-10T14:14:13Z)
• Systematic Rectification of Language Models via Dead-end Analysis [34.37598463459319]
  Large language models (LLM) can be pushed to generate toxic discourses. Here, we center detoxification on the probability that the finished discourse is ultimately considered toxic. Our approach, called rectification, utilizes a separate but significantly smaller model for detoxification.
  arXiv  Detail & Related papers  (2023-02-27T17:47:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.