Towards Detecting Contextual Real-Time Toxicity for In-Game Chat

• URL: http://arxiv.org/abs/2310.18330v1
• Date: Fri, 20 Oct 2023 00:29:57 GMT
• Title: Towards Detecting Contextual Real-Time Toxicity for In-Game Chat
• Authors: Zachary Yang, Nicolas Grenan-Godbout, Reihaneh Rabbany
• Abstract summary: ToxBuster is a scalable model that reliably detects toxic content in real-time for a line of chat by including chat history and metadata. ToxBuster consistently outperforms conventional toxicity models across popular multiplayer games, including Rainbow Six Siege, For Honor, and DOTA 2.
• Score: 5.371337604556311
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Real-time toxicity detection in online environments poses a significant challenge, due to the increasing prevalence of social media and gaming platforms. We introduce ToxBuster, a simple and scalable model that reliably detects toxic content in real-time for a line of chat by including chat history and metadata. ToxBuster consistently outperforms conventional toxicity models across popular multiplayer games, including Rainbow Six Siege, For Honor, and DOTA 2. We conduct an ablation study to assess the importance of each model component and explore ToxBuster's transferability across the datasets. Furthermore, we showcase ToxBuster's efficacy in post-game moderation, successfully flagging 82.1% of chat-reported players at a precision level of 90.0%. Additionally, we show how an additional 6% of unreported toxic players can be proactively moderated.

Related papers

• Uncovering the Viral Nature of Toxicity in Competitive Online Video Games [0.4681661603096334]
  We analyze proprietary data from the free-to-play first-person action game Call of Duty: Warzone. All of a player's teammates engaging in toxic speech increases their probability of engaging in similar behavior by 26.1 to 30.3 times the average player's likelihood of engaging in toxic speech.
  arXiv  Detail & Related papers  (2024-10-01T18:07:06Z)
• Challenges for Real-Time Toxicity Detection in Online Games [1.2289361708127877]
  Toxic behaviour and malicious players can ruin the experience, reduce the player base and potentially harm the success of the game and the studio. This article will give a brief overview of the challenges faced in toxic content detection in terms of text, audio and image processing problems, and behavioural toxicity.
  arXiv  Detail & Related papers  (2024-07-05T09:38:58Z)
• Fine-Tuning Pre-trained Language Models to Detect In-Game Trash Talks [0.0]
  The study employs and evaluates the performance of pre-trained BERT and GPT language models in detecting toxicity within in-game chats. The study was able to collect around two thousand in-game chats to train and test BERT (Base-uncased), BERT (Large-uncased), and GPT-3 models.
  arXiv  Detail & Related papers  (2024-03-19T11:36:53Z)
• Unveiling the Implicit Toxicity in Large Language Models [77.90933074675543]
  The open-endedness of large language models (LLMs) combined with their impressive capabilities may lead to new safety issues when being exploited for malicious use. We show that LLMs can generate diverse implicit toxic outputs that are exceptionally difficult to detect via simply zero-shot prompting. We propose a reinforcement learning (RL) based attacking method to further induce the implicit toxicity in LLMs.
  arXiv  Detail & Related papers  (2023-11-29T06:42:36Z)
• Comprehensive Assessment of Toxicity in ChatGPT [49.71090497696024]
  We evaluate the toxicity in ChatGPT by utilizing instruction-tuning datasets. prompts in creative writing tasks can be 2x more likely to elicit toxic responses. Certain deliberately toxic prompts, designed in earlier studies, no longer yield harmful responses.
  arXiv  Detail & Related papers  (2023-11-03T14:37:53Z)
• ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation [43.356758428820626]
  We introduce ToxicChat, a novel benchmark based on real user queries from an open-source chatbots. Our systematic evaluation of models trained on existing toxicity datasets has shown their shortcomings when applied to this unique domain of ToxicChat. In the future, ToxicChat can be a valuable resource to drive further advancements toward building a safe and healthy environment for user-AI interactions.
  arXiv  Detail & Related papers  (2023-10-26T13:35:41Z)
• ToxBuster: In-game Chat Toxicity Buster with BERT [2.764897610820181]
  ToxBuster is a simple and scalable model trained on a relatively large dataset of 194k lines of game chat from Rainbow Six Siege and For Honor. Compared to the existing state-of-the-art, ToxBuster achieves 82.95% (+7) in precision and 83.56% (+57) in recall.
  arXiv  Detail & Related papers  (2023-05-21T18:53:26Z)
• Analyzing Norm Violations in Live-Stream Chat [49.120561596550395]
  We study the first NLP study dedicated to detecting norm violations in conversations on live-streaming platforms. We define norm violation categories in live-stream chats and annotate 4,583 moderated comments from Twitch. Our results show that appropriate contextual information can boost moderation performance by 35%.
  arXiv  Detail & Related papers  (2023-05-18T05:58:27Z)
• In-game Toxic Language Detection: Shared Task and Attention Residuals [1.9218741065333018]
  We describe how the in-game toxic language shared task has been established using the real-world in-game chat data. In addition, we propose and introduce the model/framework for toxic language token tagging (slot filling) from the in-game chat.
  arXiv  Detail & Related papers  (2022-11-11T04:33:45Z)
• Toxicity Detection can be Sensitive to the Conversational Context [64.28043776806213]
  We construct and publicly release a dataset of 10,000 posts with two kinds of toxicity labels. We introduce a new task, context sensitivity estimation, which aims to identify posts whose perceived toxicity changes if the context is also considered.
  arXiv  Detail & Related papers  (2021-11-19T13:57:26Z)
• RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models [93.151822563361]
  Pretrained neural language models (LMs) are prone to generating racist, sexist, or otherwise toxic language which hinders their safe deployment. We investigate the extent to which pretrained LMs can be prompted to generate toxic language, and the effectiveness of controllable text generation algorithms at preventing such toxic degeneration.
  arXiv  Detail & Related papers  (2020-09-24T03:17:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.