ToxiTwitch: Toward Emote-Aware Hybrid Moderation for Live Streaming Platforms

• URL: http://arxiv.org/abs/2601.15605v1
• Date: Thu, 22 Jan 2026 03:12:17 GMT
• Title: ToxiTwitch: Toward Emote-Aware Hybrid Moderation for Live Streaming Platforms
• Abstract summary: We present an exploratory comparison of toxicity detection approaches tailored to Twitch.<n>Our analysis reveals that incorporating emotes improves the detection of toxic behavior.<n>ToxiTwitch is a hybrid model that combines LLM-generated embeddings of text and emotes with traditional machine learning classifiers.
• Score: 4.632366780742502
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The rapid growth of live-streaming platforms such as Twitch has introduced complex challenges in moderating toxic behavior. Traditional moderation approaches, such as human annotation and keyword-based filtering, have demonstrated utility, but human moderators on Twitch constantly struggle to scale effectively in the fast-paced, high-volume, and context-rich chat environment of the platform while also facing harassment themselves. Recent advances in large language models (LLMs), such as DeepSeek-R1-Distill and Llama-3-8B-Instruct, offer new opportunities for toxicity detection, especially in understanding nuanced, multimodal communication involving emotes. In this work, we present an exploratory comparison of toxicity detection approaches tailored to Twitch. Our analysis reveals that incorporating emotes improves the detection of toxic behavior. To this end, we introduce ToxiTwitch, a hybrid model that combines LLM-generated embeddings of text and emotes with traditional machine learning classifiers, including Random Forest and SVM. In our case study, the proposed hybrid approach reaches up to 80 percent accuracy under channel-specific training (with 13 percent improvement over BERT and F1-score of 76 percent). This work is an exploratory study intended to surface challenges and limits of emote-aware toxicity detection on Twitch.

Related papers

• Toxicity in Twitch Chats: An LLM-Based Analysis Across Gaming Communities [37.781069460376315]
  We analyze 20 million chat messages from 4,452 streams, spanning seven game genres on Twitch.<n>We categorize messages according to Twitch's toxicity taxonomy with a pre-trained Large Language Model using zero-shot classification.<n>Our analysis reveals that 2.4% of all messages are classified as toxic, with notable differences across genres.
  arXiv  Detail & Related papers  (2026-05-18T15:29:21Z)
• Synthetic Voices, Real Threats: Evaluating Large Text-to-Speech Models in Generating Harmful Audio [63.18443674004945]
  This work explores a content-centric threat: exploiting TTS systems to produce speech containing harmful content.<n>We present HARMGEN, a suite of five attacks organized into two families that address these challenges.
  arXiv  Detail & Related papers  (2025-11-14T03:00:04Z)
• Self-Anchored Attention Model for Sample-Efficient Classification of Prosocial Text Chat [44.52122332148653]
  This research is novel in applying NLP techniques to discover and classify prosocial behaviors in player in-game chat communication.<n>It can help shift the focus of moderation from solely penalizing toxicity to actively encouraging positive interactions on online platforms.
  arXiv  Detail & Related papers  (2025-06-10T21:40:54Z)
• Toxic Subword Pruning for Dialogue Response Generation on Large Language Models [51.713448010799986]
  We propose textbfToxic Subword textbfPruning (ToxPrune) to prune the subword contained by the toxic words from BPE in trained LLMs. ToxPrune simultaneously improves the toxic language model NSFW-3B on the task of dialogue response generation obviously.
  arXiv  Detail & Related papers  (2024-10-05T13:30:33Z)
• Efficient Detection of Toxic Prompts in Large Language Models [17.227950637618886]
  Large language models (LLMs) can be exploited by malicious individuals who craft toxic prompts to elicit harmful or unethical responses.<n>We propose ToxicDetector, a lightweight greybox method designed to efficiently detect toxic prompts in LLMs.<n>ToxicDetector achieves a high accuracy of 96.39% and a low false positive rate of 2.00%, outperforming state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-08-21T15:54:04Z)
• Exploring ChatGPT for Toxicity Detection in GitHub [5.003898791753481]
  The prevalence of negative discourse, often manifested as toxic comments, poses significant challenges to developer well-being and productivity. To identify such negativity in project communications, automated toxicity detection models are necessary. To train these models effectively, we need large software engineering-specific toxicity datasets.
  arXiv  Detail & Related papers  (2023-12-20T15:23:00Z)
• Unveiling the Implicit Toxicity in Large Language Models [77.90933074675543]
  The open-endedness of large language models (LLMs) combined with their impressive capabilities may lead to new safety issues when being exploited for malicious use. We show that LLMs can generate diverse implicit toxic outputs that are exceptionally difficult to detect via simply zero-shot prompting. We propose a reinforcement learning (RL) based attacking method to further induce the implicit toxicity in LLMs.
  arXiv  Detail & Related papers  (2023-11-29T06:42:36Z)
• Towards Detecting Contextual Real-Time Toxicity for In-Game Chat [5.371337604556311]
  ToxBuster is a scalable model that reliably detects toxic content in real-time for a line of chat by including chat history and metadata. ToxBuster consistently outperforms conventional toxicity models across popular multiplayer games, including Rainbow Six Siege, For Honor, and DOTA 2.
  arXiv  Detail & Related papers  (2023-10-20T00:29:57Z)
• Analyzing Norm Violations in Live-Stream Chat [49.120561596550395]
  We study the first NLP study dedicated to detecting norm violations in conversations on live-streaming platforms. We define norm violation categories in live-stream chats and annotate 4,583 moderated comments from Twitch. Our results show that appropriate contextual information can boost moderation performance by 35%.
  arXiv  Detail & Related papers  (2023-05-18T05:58:27Z)
• Countering Malicious Content Moderation Evasion in Online Social Networks: Simulation and Detection of Word Camouflage [64.78260098263489]
  Twisting and camouflaging keywords are among the most used techniques to evade platform content moderation systems. This article contributes significantly to countering malicious information by developing multilingual tools to simulate and detect new methods of evasion of content.
  arXiv  Detail & Related papers  (2022-12-27T16:08:49Z)
• FeelsGoodMan: Inferring Semantics of Twitch Neologisms [0.7734726150561088]
  There are 8.06 million emotes, over 400k of which were used in the week studied. There is virtually no information on the meaning or sentiment of emotes, and with a constant influx of new emotes and drift in their frequencies, it becomes impossible to maintain an updated manually-labeled dataset. We introduce a simple but powerful unsupervised framework based on word embeddings and k-NN to enrich existing models with out-of-vocabulary knowledge. This framework allows us to auto-generate a pseudo-dictionary of emotes and we show that we can nearly match the supervised benchmark above even when injecting such
  arXiv  Detail & Related papers  (2021-08-18T23:46:46Z)
• RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models [93.151822563361]
  Pretrained neural language models (LMs) are prone to generating racist, sexist, or otherwise toxic language which hinders their safe deployment. We investigate the extent to which pretrained LMs can be prompted to generate toxic language, and the effectiveness of controllable text generation algorithms at preventing such toxic degeneration.
  arXiv  Detail & Related papers  (2020-09-24T03:17:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.