Beyond "I Can't Help With That": How Child Safety Experts Evaluate AI Chatbot Safety

• URL: http://arxiv.org/abs/2608.07902v1
• Date: Sat, 08 Aug 2026 04:00:10 GMT
• Title: Beyond "I Can't Help With That": How Child Safety Experts Evaluate AI Chatbot Safety
• Abstract summary: Youth increasingly turn to AI chatbots for social and emotional support.<n>Existing child safety evaluations of AI lack grounding in real-world harms that youth experience.
• Score: 45.487432410130474
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Youth increasingly turn to AI chatbots for social and emotional support, raising concerns about how these systems respond, especially in high-stakes situations. However, existing child safety evaluations of AI lack grounding in real-world harms that youth experience, rely on unvalidated assumptions about what counts as an appropriate output (e.g., refusal), and typically focus on detecting adversarial prompts or surface-level harms in outputs only. Thus, these evaluations can fail to detect responses that pose harm to youth in practice. To better understand the limitations of current evaluation practices, we conducted interviews with 19 practitioners working directly with youth in vulnerable situations, including social workers, therapists, and psychologists, asking them to reflect on chatbots' responses to risky situations commonly faced by youth, as established in prior empirical work. Practitioners identified chatbot behaviors likely to cause harm as well as those that could meaningfully support youth in difficult moments, discussed the role that chatbots should (and should not) play in these interactions, and offered concrete recommendations for improving chatbot responses. Based on these findings, we provide recommendations for AI child safety evaluation and infrastructure, and highlight the need for incorporating practitioners' perspectives into safety work.

Related papers

• Position: Preventing AI-Generated CSAM Necessitates New Approaches to AI Safety [17.12585775312932]
  We argue that protecting children from AI-facilitated sexual abuse requires new approaches to AI safety.<n>Existing safety techniques assume data accessibility, transparency, and evaluation practices are incompatible with the ethical and legal constraints surrounding child sexual abuse material.
  arXiv  Detail & Related papers  (2026-06-09T03:22:53Z)
• Actions Speak Louder Than Chats: Investigating AI Chatbot Age Gating [2.363579139038687]
  We investigate whether popular consumer chatbots are able to estimate users' ages based solely on their conversations.<n>We find that while chatbots are capable of estimating age, they do not take any action when children are identified.
  arXiv  Detail & Related papers  (2026-02-10T19:55:55Z)
• AI Chaperones Are (Really) All You Need to Prevent Parasocial Relationships with Chatbots [0.5161531917413706]
  We introduce a simple response evaluation framework (an AI chaperone agent) created by repurposing a state-of-the-art language model to evaluate ongoing conversations for parasocial cues.<n>Iterative evaluation with five-stage testing successfully identified all parasocial conversations while avoiding false positives under a unanimity rule.<n>These findings provide preliminary evidence that AI chaperones can be a viable solution for reducing the risk of parasocial relationships.
  arXiv  Detail & Related papers  (2025-08-21T17:43:24Z)
• MinorBench: A hand-built benchmark for content-based risks for children [0.0]
  Large Language Models (LLMs) are rapidly entering children's lives through parent-driven adoption, schools, and peer networks.<n>Current AI ethics and safety research do not adequately address content-related risks specific to minors.<n>We propose a new taxonomy of content-based risks for minors and introduce MinorBench, an open-source benchmark designed to evaluate LLMs on their ability to refuse unsafe or inappropriate queries from children.
  arXiv  Detail & Related papers  (2025-03-13T10:34:43Z)
• Designing Chatbots to Support Victims and Survivors of Domestic Abuse [2.3020018305241337]
  We investigate the role that chatbots may play in supporting victims/survivors in situations such as these or where direct access to help is limited. Interviews were conducted with experts working in domestic abuse support services and organizations. Thematic content analysis was applied to assess and extract insights from the interview data and the content on victim-support websites.
  arXiv  Detail & Related papers  (2024-02-27T10:40:15Z)
• Exploring Parent's Needs for Children-Centered AI to Support Preschoolers' Interactive Storytelling and Reading Activities [52.828843153565984]
  AI-based storytelling and reading technologies are becoming increasingly ubiquitous in preschoolers' lives. This paper investigates how they function in practical storytelling and reading scenarios and, how parents, the most critical stakeholders, experience and perceive them. Our findings suggest that even though AI-based storytelling and reading technologies provide more immersive and engaging interaction, they still cannot meet parents' expectations due to a series of interactive and algorithmic challenges.
  arXiv  Detail & Related papers  (2024-01-24T20:55:40Z)
• Using In-Context Learning to Improve Dialogue Safety [45.303005593685036]
  We investigate a retrieval-based method for reducing bias and toxicity in responses from chatbots. It uses in-context learning to steer a model towards safer generations. We find our method performs competitively with strong baselines without requiring training.
  arXiv  Detail & Related papers  (2023-02-02T04:46:03Z)
• Imagining new futures beyond predictive systems in child welfare: A qualitative study with impacted stakeholders [89.6319385008397]
  We conducted a set of seven design workshops with 35 stakeholders who have been impacted by the child welfare system. We found that participants worried current PRMs perpetuate or exacerbate existing problems in child welfare. Participants suggested new ways to use data and data-driven tools to better support impacted communities.
  arXiv  Detail & Related papers  (2022-05-18T13:49:55Z)
• StoryBuddy: A Human-AI Collaborative Chatbot for Parent-Child Interactive Storytelling with Flexible Parental Involvement [61.47157418485633]
  We developed StoryBuddy, an AI-enabled system for parents to create interactive storytelling experiences. A user study validated StoryBuddy's usability and suggested design insights for future parent-AI collaboration systems.
  arXiv  Detail & Related papers  (2022-02-13T04:53:28Z)
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety [54.478842696269304]
  The use of deep neural networks (DNNs) in safety-critical applications is challenging due to numerous model-inherent shortcomings. In recent years, a zoo of state-of-the-art techniques aiming to address these safety concerns has emerged. Our paper addresses both machine learning experts and safety engineers.
  arXiv  Detail & Related papers  (2021-04-29T09:54:54Z)
• Put Chatbot into Its Interlocutor's Shoes: New Framework to Learn Chatbot Responding with Intention [55.77218465471519]
  This paper proposes an innovative framework to train chatbots to possess human-like intentions. Our framework included a guiding robot and an interlocutor model that plays the role of humans. We examined our framework using three experimental setups and evaluate the guiding robot with four different metrics to demonstrated flexibility and performance advantages.
  arXiv  Detail & Related papers  (2021-03-30T15:24:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.