Taming Toxicity or Fueling It? The Great Ban`s Role in Shifting Toxic User Behavior and Engagement

• URL: http://arxiv.org/abs/2411.04037v2
• Date: Thu, 07 Nov 2024 08:26:32 GMT
• Title: Taming Toxicity or Fueling It? The Great Ban`s Role in Shifting Toxic User Behavior and Engagement
• Authors: Lorenzo Cima, Benedetta Tessa, Stefano Cresci, Amaury Trujillo, Marco Avvenuti,
• Abstract summary: We evaluate the effectiveness of The Great Ban, one of the largest deplatforming interventions carried out by Reddit. We analyzed 53M comments shared by nearly 34K users. We found that 15.6% of the moderated users abandoned the platform while the remaining ones decreased their overall toxicity by 4.1%.
• Score: 0.6918368994425961
• License:
• Abstract: In today's online environments users experience harm and abuse on a daily basis. Therefore, content moderation is crucial to ensure their safety and well-being. However, the effectiveness of many moderation interventions is still uncertain. We evaluate the effectiveness of The Great Ban, one of the largest deplatforming interventions carried out by Reddit that affected almost 2,000 communities. We analyze 53M comments shared by nearly 34K users, providing in-depth results on both the intended and unintended consequences of this ban. We found that 15.6% of the moderated users abandoned the platform while the remaining ones decreased their overall toxicity by 4.1%. Nonetheless, a subset of those users increased their toxicity by 70% after the intervention. In any case, increases in toxicity did not lead to marked increases in activity or engagement, meaning that the most toxic users had overall a limited impact. Our findings bring to light new insights on the effectiveness of deplatforming. Furthermore, they also contribute to informing future content moderation strategies.

Related papers

• Tracking Patterns in Toxicity and Antisocial Behavior Over User Lifetimes on Large Social Media Platforms [0.2630859234884723]
  We analyze toxicity over a 14-year time span on nearly 500 million comments from Reddit and Wikipedia. We find that the most toxic behavior on Reddit exhibited in aggregate by the most active users, and the most toxic behavior on Wikipedia exhibited in aggregate by the least active users.
  arXiv  Detail & Related papers  (2024-07-12T15:45:02Z)
• The Great Ban: Efficacy and Unintended Consequences of a Massive Deplatforming Operation on Reddit [0.7422344184734279]
  We assess the effectiveness of The Great Ban, a massive deplatforming operation that affected nearly 2,000 communities on Reddit. By analyzing 16M comments posted by 17K users during 14 months, we provide nuanced results on the effects, both desired and otherwise.
  arXiv  Detail & Related papers  (2024-01-20T15:21:37Z)
• Unveiling the Implicit Toxicity in Large Language Models [77.90933074675543]
  The open-endedness of large language models (LLMs) combined with their impressive capabilities may lead to new safety issues when being exploited for malicious use. We show that LLMs can generate diverse implicit toxic outputs that are exceptionally difficult to detect via simply zero-shot prompting. We propose a reinforcement learning (RL) based attacking method to further induce the implicit toxicity in LLMs.
  arXiv  Detail & Related papers  (2023-11-29T06:42:36Z)
• Comprehensive Assessment of Toxicity in ChatGPT [49.71090497696024]
  We evaluate the toxicity in ChatGPT by utilizing instruction-tuning datasets. prompts in creative writing tasks can be 2x more likely to elicit toxic responses. Certain deliberately toxic prompts, designed in earlier studies, no longer yield harmful responses.
  arXiv  Detail & Related papers  (2023-11-03T14:37:53Z)
• Analyzing Norm Violations in Live-Stream Chat [49.120561596550395]
  We study the first NLP study dedicated to detecting norm violations in conversations on live-streaming platforms. We define norm violation categories in live-stream chats and annotate 4,583 moderated comments from Twitch. Our results show that appropriate contextual information can boost moderation performance by 35%.
  arXiv  Detail & Related papers  (2023-05-18T05:58:27Z)
• One of Many: Assessing User-level Effects of Moderation Interventions on r/The_Donald [1.1041211464412573]
  We evaluate the user level effects of the sequence of moderation interventions that targeted r/The_Donald on Reddit. We find that interventions having strong community level effects also cause extreme and diversified user level reactions. Our results highlight that platform and community level effects are not always representative of the underlying behavior of individuals or smaller user groups.
  arXiv  Detail & Related papers  (2022-09-19T07:46:18Z)
• Make Reddit Great Again: Assessing Community Effects of Moderation Interventions on r/The_Donald [1.1041211464412573]
  r/The_Donald was repeatedly denounced as a toxic and misbehaving online community. It was quarantined in June 2019, restricted in February 2020, and finally banned in June 2020, but the effects of this sequence of interventions are still unclear. We find that the interventions greatly reduced the activity of problematic users. However, the interventions also caused an increase in toxicity and led users to share more polarized and less factual news.
  arXiv  Detail & Related papers  (2022-01-17T15:09:51Z)
• Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection [75.54119209776894]
  We investigate the effect of annotator identities (who) and beliefs (why) on toxic language annotations. We consider posts with three characteristics: anti-Black language, African American English dialect, and vulgarity. Our results show strong associations between annotator identity and beliefs and their ratings of toxicity.
  arXiv  Detail & Related papers  (2021-11-15T18:58:20Z)
• News consumption and social media regulations policy [70.31753171707005]
  We analyze two social media that enforced opposite moderation methods, Twitter and Gab, to assess the interplay between news consumption and content regulation. Our results show that the presence of moderation pursued by Twitter produces a significant reduction of questionable content. The lack of clear regulation on Gab results in the tendency of the user to engage with both types of content, showing a slight preference for the questionable ones which may account for a dissing/endorsement behavior.
  arXiv  Detail & Related papers  (2021-06-07T19:26:32Z)
• Do Platform Migrations Compromise Content Moderation? Evidence from r/The_Donald and r/Incels [20.41491269475746]
  We report the results of a large-scale observational study of how problematic online communities progress following community-level moderation measures. Our results suggest that, in both cases, moderation measures significantly decreased posting activity on the new platform. In spite of that, users in one of the studied communities showed increases in signals associated with toxicity and radicalization.
  arXiv  Detail & Related papers  (2020-10-20T16:03:06Z)
• RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models [93.151822563361]
  Pretrained neural language models (LMs) are prone to generating racist, sexist, or otherwise toxic language which hinders their safe deployment. We investigate the extent to which pretrained LMs can be prompted to generate toxic language, and the effectiveness of controllable text generation algorithms at preventing such toxic degeneration.
  arXiv  Detail & Related papers  (2020-09-24T03:17:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.