Beyond Her: Safety Dynamics in Role-play AI Companions

• URL: http://arxiv.org/abs/2606.28968v2
• Date: Tue, 30 Jun 2026 05:05:43 GMT
• Title: Beyond Her: Safety Dynamics in Role-play AI Companions
• Abstract summary: We investigate safety dynamics in Role-play AI Companions (RACs) usage through a two-part mixed-methods study.<n>We find that users' internalizing problems, the role personality adopted by the RAC, and risk interaction patterns jointly shape safety dynamics.<n>Our findings highlight the importance of modeling safety as a dynamic process rather than a static property.
• Score: 31.494321104359642
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The film 'Her' pictured a future of love between humans and AI. That future has quietly emerged in the form of Role-play AI Companions (RACs), where emotionally responsive interactions blur the boundary between tool use and relational engagement. However, the safety implications remain poorly understood, as user experiences evolve over time through safety dynamics, spanning both emotional and risk behavioral dynamics, that can gradually shift interactions toward risk. In this paper, we investigate safety dynamics in RAC usage through a two-part mixed-methods study (Study I \& II). (1) Study I consists of semi-structured interviews (N = 16) to identify the key factors shaping these dynamics. We find that users' internalizing problems, the role personality adopted by the RAC, and risk interaction patterns jointly shape safety dynamics. Building on these insights, (2) Study II conducts a 14-day Ecological Momentary Assessment (N = 102) to examine how safety dynamics unfold in real-world usage. We identify distinct user profiles based on internalizing problems and show that interactions with RACs can produce short-term emotional relief while masking longer-term deterioration. Furthermore, vulnerable users exhibit more unstable risk behavioral patterns over time, making risk emergence less predictable and harder to mitigate with static safeguards. Our findings highlight the importance of modeling safety as a dynamic process rather than a static property. We conclude with three-layer design implications for next-generation AI companions, advocating for adaptive safeguards that can respond to evolving emotional and behavioral signals.

Related papers

• Security and Privacy in the Musical Metaverse: Threat Analysis and Design Implications [0.0]
  Musical Metaverse (MM) introduces immersive, real-time environments for collaborative musical interaction.<n>MM is characterized by ultra-low-latency constraints, continuous multimodal data streams, and heterogeneous devices.<n>This paper presents a multi-layer threat analysis of MM ecosystems.<n>Threats are analyzed across network, application, data/AI, device, intellectual property rights, and social layers.
  arXiv  Detail & Related papers  (2026-09-03T10:56:26Z)
• Long-term Measurements: Towards a Longitudinal Understanding of Human-AI Interactions [7.282012402627575]
  We discuss how computational methods in the field of NLP need to be combined with measurements used in social science fields.<n>This ability to model human behavioral shifts as a function of model interactions can facilitate online rather than post-hoc detection of problematic behaviors.
  arXiv  Detail & Related papers  (2026-08-03T16:55:50Z)
• Responsible Personalisation: The Double-Edged Sword of Personalisation in Human-Robot Interaction [2.1875243525685946]
  We present a lifecycle-based and context-sensitive framework for personalised HRI, grounded in an embodiment-aware perspective.<n>We conduct an integrative analysis of key ethical risks, including autonomy erosion, biased user modelling, manipulation, dehumanisation, and privacy violations.<n>This work provides a foundation for more systematic, transparent, and ethically grounded approaches to personalised robot behaviour.
  arXiv  Detail & Related papers  (2026-07-07T14:40:20Z)
• Persona-Grounded Safety Evaluation of AI Companions in Multi-Turn Conversations [3.437656066916039]
  We present the first end-to-end scalable framework for controlled simulation and safety evaluation of multi-turn interactions with AI companion applications.<n>We apply this framework to evaluate how Replika, a widely used AI companion app, responds to high-risk user groups.
  arXiv  Detail & Related papers  (2026-04-30T21:04:41Z)
• Safety in Embodied AI: A Survey of Risks, Attacks, and Defenses [168.50301366360344]
  Embodied Artificial Intelligence (Embodied AI) integrates perception, cognition, planning, and interaction into agents that operate in safety-critical environments.<n>Unlike digital AI systems, embodied agents must act under uncertain sensing, incomplete knowledge, and dynamic human-robot interactions.<n>This survey provides a comprehensive review of safety research in embodied AI, examining attacks and defenses across the full embodied pipeline.
  arXiv  Detail & Related papers  (2026-03-28T13:21:44Z)
• ConflictBench: Evaluating Human-AI Conflict via Interactive and Visually Grounded Environments [43.08480724370872]
  We introduce ConflictBench, a benchmark for evaluating human-AI conflict through 150 multi-turn scenarios.<n>We show that ConflictBench integrates a text-based simulation engine with a visually grounded world model, enabling agents to perceive, plan, and act under dynamic conditions.
  arXiv  Detail & Related papers  (2026-03-09T06:59:48Z)
• Understanding Risk and Dependency in AI Chatbot Use from User Discourse [4.1957094635667875]
  We present a large-scale computational thematic analysis of posts collected between 2023 and 2025 from two communities, r/AIDangers and r/ChatbotAddiction.<n>We identify 14 recurring thematic categories and synthesize them into five higher-order experiential dimensions.<n>Our findings reveal five empirically derived experiential dimensions of AI-related psychological risk grounded in real-world user discourse.
  arXiv  Detail & Related papers  (2026-02-10T02:16:57Z)
• Towards Emotionally Intelligent and Responsible Reinforcement Learning [0.40719854602160227]
  We propose a Responsible Reinforcement Learning framework that integrates emotional and contextual understanding with ethical considerations.<n>We introduce a multi-objective reward function that balances short-term behavioral engagement with long-term user well-being.<n>We discuss the implications of this approach for human-centric domains such as behavioral health, education, and digital therapeutics.
  arXiv  Detail & Related papers  (2025-11-13T18:09:37Z)
• ANNIE: Be Careful of Your Robots [48.89876809734855]
  We present the first systematic study of adversarial safety attacks on embodied AI systems.<n>We show attack success rates exceeding 50% across all safety categories.<n>Results expose a previously underexplored but highly consequential attack surface in embodied AI systems.
  arXiv  Detail & Related papers  (2025-09-03T15:00:28Z)
• Reinforced Interactive Continual Learning via Real-time Noisy Human Feedback [59.768119380109084]
  This paper introduces an interactive continual learning paradigm where AI models dynamically learn new skills from real-time human feedback.<n>We propose RiCL, a Reinforced interactive Continual Learning framework leveraging Large Language Models (LLMs)<n>Our RiCL approach substantially outperforms existing combinations of state-of-the-art online continual learning and noisy-label learning methods.
  arXiv  Detail & Related papers  (2025-05-15T03:22:03Z)
• Safe Explicable Policy Search [2.614894682819833]
  We present SEPS, which aims to provide a learning approach to explicable behavior generation while minimizing the safety risk.<n>We formulate SEPS as a constrained optimization problem where the agent aims to maximize an explicability score subject to constraints on safety.<n>We evaluate SEPS in safety-gym environments and with a physical robot experiment to show its efficacy and relevance in human-AI teaming.
  arXiv  Detail & Related papers  (2025-03-10T20:52:41Z)
• HAICOSYSTEM: An Ecosystem for Sandboxing Safety Risks in Human-AI Interactions [95.49509269498367]
  We present HAICOSYSTEM, a framework examining AI agent safety within diverse and complex social interactions.<n>We run 1840 simulations based on 92 scenarios across seven domains (e.g., healthcare, finance, education)<n>Our experiments show that state-of-the-art LLMs, both proprietary and open-sourced, exhibit safety risks in over 50% cases.
  arXiv  Detail & Related papers  (2024-09-24T19:47:21Z)
• SAFE-SIM: Safety-Critical Closed-Loop Traffic Simulation with Diffusion-Controllable Adversaries [94.84458417662407]
  We introduce SAFE-SIM, a controllable closed-loop safety-critical simulation framework. Our approach yields two distinct advantages: 1) generating realistic long-tail safety-critical scenarios that closely reflect real-world conditions, and 2) providing controllable adversarial behavior for more comprehensive and interactive evaluations. We validate our framework empirically using the nuScenes and nuPlan datasets across multiple planners, demonstrating improvements in both realism and controllability.
  arXiv  Detail & Related papers  (2023-12-31T04:14:43Z)
• Risk-Sensitive Sequential Action Control with Multi-Modal Human Trajectory Forecasting for Safe Crowd-Robot Interaction [55.569050872780224]
  We present an online framework for safe crowd-robot interaction based on risk-sensitive optimal control, wherein the risk is modeled by the entropic risk measure. Our modular approach decouples the crowd-robot interaction into learning-based prediction and model-based control. A simulation study and a real-world experiment show that the proposed framework can accomplish safe and efficient navigation while avoiding collisions with more than 50 humans in the scene.
  arXiv  Detail & Related papers  (2020-09-12T02:02:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.