Taming Toxic Talk: Using chatbots to intervene with users posting toxic comments

• URL: http://arxiv.org/abs/2601.20100v1
• Date: Tue, 27 Jan 2026 22:39:23 GMT
• Title: Taming Toxic Talk: Using chatbots to intervene with users posting toxic comments
• Authors: Jeremy Foote, Deepak Kumar, Bedadyuti Jha, Ryan Funkhouser, Loizos Bitsikokos, Hitesh Goel, Hsuen-Chi Chiu,
• Abstract summary: We explore the impact of rehabilitative conversations with generative AI chatbots on users who share toxic content online.<n>We conducted a large-scale field experiment with seven Reddit communities.<n>We did not observe a significant change in toxic behavior in the following month compared to a control group.
• Score: 3.1918086432069663
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generative AI chatbots have proven surprisingly effective at persuading people to change their beliefs and attitudes in lab settings. However, the practical implications of these findings are not yet clear. In this work, we explore the impact of rehabilitative conversations with generative AI chatbots on users who share toxic content online. Toxic behaviors -- like insults or threats of violence, are widespread in online communities. Strategies to deal with toxic behavior are typically punitive, such as removing content or banning users. Rehabilitative approaches are rarely attempted, in part due to the emotional and psychological cost of engaging with aggressive users. In collaboration with seven large Reddit communities, we conducted a large-scale field experiment (N=893) to invite people who had recently posted toxic content to participate in conversations with AI chatbots. A qualitative analysis of the conversations shows that many participants engaged in good faith and even expressed remorse or a desire to change. However, we did not observe a significant change in toxic behavior in the following month compared to a control group. We discuss possible explanations for our findings, as well as theoretical and practical implications based on our results.

Related papers

• Analyzing Toxicity in Deep Conversations: A Reddit Case Study [0.0]
  This work employs a tree-based approach to understand how users behave concerning toxicity in public conversation settings. We collect both the posts and the comment sections of the top 100 posts from 8 Reddit communities that allow profanity, totaling over 1 million responses. We find that toxic comments increase the likelihood of subsequent toxic comments being produced in online conversations.
  arXiv  Detail & Related papers  (2024-04-11T16:10:44Z)
• Comprehensive Assessment of Toxicity in ChatGPT [49.71090497696024]
  We evaluate the toxicity in ChatGPT by utilizing instruction-tuning datasets. prompts in creative writing tasks can be 2x more likely to elicit toxic responses. Certain deliberately toxic prompts, designed in earlier studies, no longer yield harmful responses.
  arXiv  Detail & Related papers  (2023-11-03T14:37:53Z)
• Understanding Multi-Turn Toxic Behaviors in Open-Domain Chatbots [8.763670548363443]
  A new attack, toxicbot, is developed to generate toxic responses in a multi-turn conversation. toxicbot can be used by both industry and researchers to develop methods for detecting and mitigating toxic responses in conversational dialogue.
  arXiv  Detail & Related papers  (2023-07-14T03:58:42Z)
• Analyzing Norm Violations in Live-Stream Chat [49.120561596550395]
  We study the first NLP study dedicated to detecting norm violations in conversations on live-streaming platforms. We define norm violation categories in live-stream chats and annotate 4,583 moderated comments from Twitch. Our results show that appropriate contextual information can boost moderation performance by 35%.
  arXiv  Detail & Related papers  (2023-05-18T05:58:27Z)
• User Engagement and the Toxicity of Tweets [1.1339580074756188]
  We analyze a random sample of more than 85,300 Twitter conversations to examine differences between toxic and non-toxic conversations. We find that toxic conversations, those with at least one toxic tweet, are longer but have fewer individual users contributing to the dialogue compared to the non-toxic conversations. We also find a relationship between the toxicity of the first reply to a toxic tweet and the toxicity of the conversation.
  arXiv  Detail & Related papers  (2022-11-07T20:55:22Z)
• Twitter Users' Behavioral Response to Toxic Replies [1.2387676601792899]
  We studied the impact of toxicity on users' online behavior on Twitter. We found that toxicity victims show a combination of the following behavioral reactions: avoidance, revenge, countermeasures, and negotiation. Our results can assist further studies in developing more effective detection and intervention methods for reducing the negative consequences of toxicity on social media.
  arXiv  Detail & Related papers  (2022-10-24T17:36:58Z)
• Why So Toxic? Measuring and Triggering Toxic Behavior in Open-Domain Chatbots [24.84440998820146]
  This paper presents a first-of-its-kind, large-scale measurement of toxicity in chatbots. We show that publicly available chatbots are prone to providing toxic responses when fed toxic queries. We then set out to design and experiment with an attack, ToxicBuddy, which relies on fine-tuning GPT-2 to generate non-toxic queries.
  arXiv  Detail & Related papers  (2022-09-07T20:45:41Z)
• Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection [75.54119209776894]
  We investigate the effect of annotator identities (who) and beliefs (why) on toxic language annotations. We consider posts with three characteristics: anti-Black language, African American English dialect, and vulgarity. Our results show strong associations between annotator identity and beliefs and their ratings of toxicity.
  arXiv  Detail & Related papers  (2021-11-15T18:58:20Z)
• CheerBots: Chatbots toward Empathy and Emotionusing Reinforcement Learning [60.348822346249854]
  This study presents a framework whereby several empathetic chatbots are based on understanding users' implied feelings and replying empathetically for multiple dialogue turns. We call these chatbots CheerBots. CheerBots can be retrieval-based or generative-based and were finetuned by deep reinforcement learning. To respond in an empathetic way, we develop a simulating agent, a Conceptual Human Model, as aids for CheerBots in training with considerations on changes in user's emotional states in the future to arouse sympathy.
  arXiv  Detail & Related papers  (2021-10-08T07:44:47Z)
• Put Chatbot into Its Interlocutor's Shoes: New Framework to Learn Chatbot Responding with Intention [55.77218465471519]
  This paper proposes an innovative framework to train chatbots to possess human-like intentions. Our framework included a guiding robot and an interlocutor model that plays the role of humans. We examined our framework using three experimental setups and evaluate the guiding robot with four different metrics to demonstrated flexibility and performance advantages.
  arXiv  Detail & Related papers  (2021-03-30T15:24:37Z)
• CASS: Towards Building a Social-Support Chatbot for Online Health Community [67.45813419121603]
  The CASS architecture is based on advanced neural network algorithms. It can handle new inputs from users and generate a variety of responses to them. With a follow-up field experiment, CASS is proven useful in supporting individual members who seek emotional support.
  arXiv  Detail & Related papers  (2021-01-04T05:52:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.