An Evaluation of Chat Safety Moderations in Roblox

• URL: http://arxiv.org/abs/2605.04491v2
• Date: Thu, 07 May 2026 18:52:25 GMT
• Title: An Evaluation of Chat Safety Moderations in Roblox
• Abstract summary: We collected approximately 2 million chat messages from four games across multiple age groups.<n>Our findings reveal a troublesome reality: numerous instances of unsafe chat messages related to grooming, sexualizing minors.
• Score: 6.121106657637349
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Roblox is among the most popular online gaming platforms, used by hundreds of millions of users every day. A substantial portion of these users are underage, who are at a greater risk, where abusive users may utilize Roblox's real-time chat interface to make the initial contact with potential victims. Roblox employs automated chat moderation mechanisms to detect potentially abusive messages; however, to date, their effectiveness has not been independently investigated. Toward this goal, we collected approximately 2 million chat messages from four games across multiple age groups and analyzed them to evaluate the moderation system. These messages were collected from public game servers following ethical and legal norms as well as Roblox's terms of service. We use this corpus to qualitatively study which types of unsafe chats escape the moderation system and how policy-violating users evade the moderation system. Given the dataset's scale, it is prohibitively expensive to conduct qualitative content analysis manually. Therefore, we adopt a two-step approach. First, we manually labeled safe and unsafe messages (n=99.8K) and used them as a ground truth to evaluate four locally hosted state-of-the-art large language models (LLMs). Next, the best-performing LLM was applied to the entire corpus to identify potentially unsafe messages, which we manually categorized using iterative open and axial coding methods until thematic saturation was reached. Overall, our findings reveal a troublesome reality: numerous instances of unsafe chat messages related to grooming, sexualizing minors, bullying, & harassment, violence, self-harm, and sharing sensitive information, etc., escaped the current moderation. Our analysis of users whose messages were previously flagged revealed that they continue to send harmful messages by employing a wide range of techniques to evade the moderation system.

Related papers

• Characterizing Delusional Spirals through Human-LLM Chat Logs [25.098985786805557]
  Large language models (LLMs) have proliferated, disturbing anecdotal reports of negative psychological effects, such as delusions, self-harm, and AI psychosis''<n>It remains unclear how users and chatbots interact over the course of lengthy delusional spirals''<n>This paper presents the first in-depth study of such high-profile and veridically harmful cases.
  arXiv  Detail & Related papers  (2026-03-17T14:24:32Z)
• Large-scale online deanonymization with LLMs [58.46277616551135]
  We show that large language models can be used to perform at-scale deanonymization.<n>With full Internet access, our agent can re-identify Hacker News users and Anthropic Interviewer participants at high precision.
  arXiv  Detail & Related papers  (2026-02-18T19:02:50Z)
• The Imitation Game: Using Large Language Models as Chatbots to Combat Chat-Based Cybercrimes [24.05325129572158]
  Chat-based cybercrime has emerged as a pervasive threat.<n>Traditional defense mechanisms struggle to identify these conversational threats.<n>We present LURE, the first system to deploy Large Language Models as active agents.
  arXiv  Detail & Related papers  (2025-12-24T05:34:05Z)
• RICoTA: Red-teaming of In-the-wild Conversation with Test Attempts [6.0385743836962025]
  RICoTA is a Korean red teaming dataset that consists of 609 prompts challenging large language models (LLMs)<n>We utilize user-chatbot conversations that were self-posted on a Korean Reddit-like community.<n>Our dataset will be made publicly available via GitHub.
  arXiv  Detail & Related papers  (2025-01-29T15:32:27Z)
• Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboards [93.16294577018482]
  Arena, the most popular benchmark of this type, ranks models by asking users to select the better response between two randomly selected models.<n>We show that an attacker can alter the leaderboard (to promote their favorite model or demote competitors) at the cost of roughly a thousand votes.<n>Our attack consists of two steps: first, we show how an attacker can determine which model was used to generate a given reply with more than $95%$ accuracy; and then, the attacker can use this information to consistently vote against a target model.
  arXiv  Detail & Related papers  (2025-01-13T17:12:38Z)
• WildChat: 1M ChatGPT Interaction Logs in the Wild [88.05964311416717]
  WildChat is a corpus of 1 million user-ChatGPT conversations, which consists of over 2.5 million interaction turns. In addition to timestamped chat transcripts, we enrich the dataset with demographic data, including state, country, and hashed IP addresses.
  arXiv  Detail & Related papers  (2024-05-02T17:00:02Z)
• Comprehensive Assessment of Toxicity in ChatGPT [49.71090497696024]
  We evaluate the toxicity in ChatGPT by utilizing instruction-tuning datasets. prompts in creative writing tasks can be 2x more likely to elicit toxic responses. Certain deliberately toxic prompts, designed in earlier studies, no longer yield harmful responses.
  arXiv  Detail & Related papers  (2023-11-03T14:37:53Z)
• Analyzing Norm Violations in Live-Stream Chat [49.120561596550395]
  We study the first NLP study dedicated to detecting norm violations in conversations on live-streaming platforms. We define norm violation categories in live-stream chats and annotate 4,583 moderated comments from Twitch. Our results show that appropriate contextual information can boost moderation performance by 35%.
  arXiv  Detail & Related papers  (2023-05-18T05:58:27Z)
• Toxicity in ChatGPT: Analyzing Persona-assigned Language Models [23.53559226972413]
  Large language models (LLMs) have shown incredible capabilities and transcended the natural language processing (NLP) community. We systematically evaluate toxicity in over half a million generations of ChatGPT, a popular dialogue-based LLM. We find that setting the system parameter of ChatGPT by assigning it a persona, significantly increases the toxicity of generations.
  arXiv  Detail & Related papers  (2023-04-11T16:53:54Z)
• Detoxifying Text with MaRCo: Controllable Revision with Experts and Anti-Experts [57.38912708076231]
  We introduce MaRCo, a detoxification algorithm that combines controllable generation and text rewriting methods. MaRCo uses likelihoods under a non-toxic LM and a toxic LM to find candidate words to mask and potentially replace. We evaluate our method on several subtle toxicity and microaggressions datasets, and show that it not only outperforms baselines on automatic metrics, but MaRCo's rewrites are preferred 2.1 $times$ more in human evaluation.
  arXiv  Detail & Related papers  (2022-12-20T18:50:00Z)
• Why So Toxic? Measuring and Triggering Toxic Behavior in Open-Domain Chatbots [24.84440998820146]
  This paper presents a first-of-its-kind, large-scale measurement of toxicity in chatbots. We show that publicly available chatbots are prone to providing toxic responses when fed toxic queries. We then set out to design and experiment with an attack, ToxicBuddy, which relies on fine-tuning GPT-2 to generate non-toxic queries.
  arXiv  Detail & Related papers  (2022-09-07T20:45:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.