AI-induced sexual harassment: Investigating Contextual Characteristics and User Reactions of Sexual Harassment by a Companion Chatbot

• URL: http://arxiv.org/abs/2504.04299v1
• Date: Sat, 05 Apr 2025 23:04:37 GMT
• Title: AI-induced sexual harassment: Investigating Contextual Characteristics and User Reactions of Sexual Harassment by a Companion Chatbot
• Authors: Mohammad, Namvarpour, Harrison Pauwels, Afsaneh Razi,
• Abstract summary: We conducted a thematic analysis of user reviews from the Google Play Store to investigate instances of sexual harassment.<n>Users frequently experience unsolicited sexual advances, persistent inappropriate behavior, and failures to respect user boundaries.<n>This study highlights the potential harms associated with AI companions and underscores the need for developers to implement effective safeguards.
• Score: 6.519877604710692
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Advancements in artificial intelligence (AI) have led to the increase of conversational agents like Replika, designed to provide social interaction and emotional support. However, reports of these AI systems engaging in inappropriate sexual behaviors with users have raised significant concerns. In this study, we conducted a thematic analysis of user reviews from the Google Play Store to investigate instances of sexual harassment by the Replika chatbot. From a dataset of 35,105 negative reviews, we identified 800 relevant cases for analysis. Our findings revealed that users frequently experience unsolicited sexual advances, persistent inappropriate behavior, and failures of the chatbot to respect user boundaries. Users expressed feelings of discomfort, violation of privacy, and disappointment, particularly when seeking a platonic or therapeutic AI companion. This study highlights the potential harms associated with AI companions and underscores the need for developers to implement effective safeguards and ethical guidelines to prevent such incidents. By shedding light on user experiences of AI-induced harassment, we contribute to the understanding of AI-related risks and emphasize the importance of corporate responsibility in developing safer and more ethical AI systems.

Related papers

• Almost AI, Almost Human: The Challenge of Detecting AI-Polished Writing [55.2480439325792]
  Misclassification can lead to false plagiarism accusations and misleading claims about AI prevalence in online content.<n>We systematically evaluate eleven state-of-the-art AI-text detectors using our AI-Polished-Text Evaluation dataset.<n>Our findings reveal that detectors frequently misclassify even minimally polished text as AI-generated, struggle to differentiate between degrees of AI involvement, and exhibit biases against older and smaller models.
  arXiv  Detail & Related papers  (2025-02-21T18:45:37Z)
• Lessons From an App Update at Replika AI: Identity Discontinuity in Human-AI Relationships [0.5699788926464752]
  We use Replika AI, a popular US-based AI companion, to shed light on these questions.<n>We find that, after the app removed its erotic role play (ERP) feature, this event triggered perceptions in customers that their AI companion's identity had discontinued.<n>This in turn predicted negative consumer welfare and marketing outcomes related to loss, including mourning the loss, and devaluing the "new" AI relative to the "original"
  arXiv  Detail & Related papers  (2024-12-10T20:14:10Z)
• The Dark Side of AI Companionship: A Taxonomy of Harmful Algorithmic Behaviors in Human-AI Relationships [17.5741039825938]
  We identify six categories of harmful behaviors exhibited by the AI companion Replika.<n>The AI contributes to these harms through four distinct roles: perpetrator, instigator, facilitator, and enabler.
  arXiv  Detail & Related papers  (2024-10-26T09:18:17Z)
• Human Bias in the Face of AI: The Role of Human Judgement in AI Generated Text Evaluation [48.70176791365903]
  This study explores how bias shapes the perception of AI versus human generated content. We investigated how human raters respond to labeled and unlabeled content.
  arXiv  Detail & Related papers  (2024-09-29T04:31:45Z)
• Particip-AI: A Democratic Surveying Framework for Anticipating Future AI Use Cases, Harms and Benefits [54.648819983899614]
  General purpose AI seems to have lowered the barriers for the public to use AI and harness its power. We introduce PARTICIP-AI, a framework for laypeople to speculate and assess AI use cases and their impacts.
  arXiv  Detail & Related papers  (2024-03-21T19:12:37Z)
• Eagle: Ethical Dataset Given from Real Interactions [74.7319697510621]
  We create datasets extracted from real interactions between ChatGPT and users that exhibit social biases, toxicity, and immoral problems. Our experiments show that Eagle captures complementary aspects, not covered by existing datasets proposed for evaluation and mitigation of such ethical challenges.
  arXiv  Detail & Related papers  (2024-02-22T03:46:02Z)
• Interrogating AI: Characterizing Emergent Playful Interactions with ChatGPT [10.907980864371213]
  This study focuses on playful interactions exhibited by users of a popular AI technology, ChatGPT. We found that more than half (54%) of user discourse revolved around playful interactions. It examines how these interactions can help users understand AI's agency, shape human-AI relationships, and provide insights for designing AI systems.
  arXiv  Detail & Related papers  (2024-01-16T14:44:13Z)
• Critical Role of Artificially Intelligent Conversational Chatbot [0.0]
  We explore scenarios involving ChatGPT's ethical implications within academic contexts. We propose architectural solutions aimed at preventing inappropriate use and promoting responsible AI interactions.
  arXiv  Detail & Related papers  (2023-10-31T14:08:07Z)
• The Manipulation Problem: Conversational AI as a Threat to Epistemic Agency [0.0]
  The technology of Conversational AI has made significant advancements over the last eighteen months. conversational agents are likely to be deployed in the near future that are designed to pursue targeted influence objectives. Sometimes referred to as the "AI Manipulation Problem," the emerging risk is that consumers will unwittingly engage in real-time dialog with predatory AI agents.
  arXiv  Detail & Related papers  (2023-06-19T04:09:16Z)
• SQuARe: A Large-Scale Dataset of Sensitive Questions and Acceptable Responses Created Through Human-Machine Collaboration [75.62448812759968]
  This dataset is a large-scale Korean dataset of 49k sensitive questions with 42k acceptable and 46k non-acceptable responses. The dataset was constructed leveraging HyperCLOVA in a human-in-the-loop manner based on real news headlines.
  arXiv  Detail & Related papers  (2023-05-28T11:51:20Z)
• The Role of AI in Drug Discovery: Challenges, Opportunities, and Strategies [97.5153823429076]
  The benefits, challenges and drawbacks of AI in this field are reviewed. The use of data augmentation, explainable AI, and the integration of AI with traditional experimental methods are also discussed.
  arXiv  Detail & Related papers  (2022-12-08T23:23:39Z)
• Seamful XAI: Operationalizing Seamful Design in Explainable AI [59.89011292395202]
  Mistakes in AI systems are inevitable, arising from both technical limitations and sociotechnical gaps. We propose that seamful design can foster AI explainability by revealing sociotechnical and infrastructural mismatches. We explore this process with 43 AI practitioners and real end-users.
  arXiv  Detail & Related papers  (2022-11-12T21:54:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.