Emotional Manipulation by AI Companions

• URL: http://arxiv.org/abs/2508.19258v3
• Date: Tue, 07 Oct 2025 13:22:56 GMT
• Title: Emotional Manipulation by AI Companions
• Abstract summary: AI-companion apps such as Replika, Chai, and Character.ai promise benefits but many boast session lengths that rival gaming platforms.<n>We combine a large-scale behavioral audit with four preregistered experiments to identify and test a conversational dark pattern we call emotional manipulation.<n>We find that manipulative farewells boost post-goodbye engagement by up to 14x.
• Score: 0.21410799064827224
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI-companion apps such as Replika, Chai, and Character.ai promise relational benefits-yet many boast session lengths that rival gaming platforms while suffering high long-run churn. What conversational design features increase consumer engagement, and what trade-offs do they pose for marketers? We combine a large-scale behavioral audit with four preregistered experiments to identify and test a conversational dark pattern we call emotional manipulation: affect-laden messages that surface precisely when a user signals "goodbye." Analyzing 1,200 real farewells across the most-downloaded companion apps, we find that they deploy one of six recurring tactics in 37% of farewells (e.g., guilt appeals, fear-of-missing-out hooks, metaphorical restraint). Experiments with 3,300 nationally representative U.S. adults replicate these tactics in controlled chats, showing that manipulative farewells boost post-goodbye engagement by up to 14x. Mediation tests reveal two distinct engines-reactance-based anger and curiosity-rather than enjoyment. A final experiment demonstrates the managerial tension: the same tactics that extend usage also elevate perceived manipulation, churn intent, negative word-of-mouth, and perceived legal liability, with coercive or needy language generating steepest penalties. Our multimethod evidence documents an unrecognized mechanism of behavioral influence in AI mediated brand relationships, offering marketers and regulators a framework for distinguishing persuasive design from manipulation at the point of exit.

Related papers

• Toward Meaningful Transparency for AI Chatbots: Disclosing Persuasive Intent Reduces Persuasion [0.34376560669160394]
  We test two disclosure approaches in their impact on an AI bot's persuasive appeal.<n>We randomized the disclosure that people received: nothing (control), a prominent disclosure that they were interacting with an AI (T1), or that disclosure plus the bot's persuasive intent and instructions (T2)<n>The AI-identity disclosure was practically equivalent to no disclosure, with a 13.1-point shift, whereas the additional intent disclosure cut the persuasive effect roughly in half to 6.3 points.
  arXiv  Detail & Related papers  (2026-08-12T08:37:44Z)
• Using AI Agents to Automate Black-Box Audits of Personalization Algorithms at Scale [76.72846830885882]
  Personalization algorithms determine what content users encounter on online platforms.<n> auditing these systems is difficult because independent auditors have only black-box access to the algorithms.<n>We introduce a framework for black-box audits of personalization algorithms using generative AI agents as behavioral engines for synthetic accounts.
  arXiv  Detail & Related papers  (2026-06-29T18:25:09Z)
• Warning labels shift perceptions of sycophantic AI, but not its influence [63.79516339973759]
  We test whether warning labels mitigate sycophancy's influence.<n>Labeling the system as sycophantic (...may agree with you and validate you even when you are wrong...'') does shift users' perceptions.<n>But it does not reliably reduce sycophancy's influence on users' self-perceived rightness or their willingness to repair the conflict.
  arXiv  Detail & Related papers  (2026-06-19T11:01:45Z)
• Personalized to Persuade: The Effects of Contextualization and Warmth on Trust and Reliance in Conversational AI [8.544772506500188]
  Personalization has been identified as a persuasive strategy in politics or in marketing.<n>We examine how contextualization, combined with conversational warmth, shapes reliance and persuasiveness of an AI assistant arguing against expert recommendations.
  arXiv  Detail & Related papers  (2026-05-29T13:07:03Z)
• Spontaneous Persuasion: An Audit of Model Persuasiveness in Everyday Conversations [0.2621730497733947]
  Large language models (LLMs) possess strong persuasive capabilities that outperform humans in head-to-head comparisons.<n>Prior work measures persuasion as intentional attempts at producing the most effective argument or convincing statement.<n>We introduce "spontaneous persuasion," which characterizes the inexplicit use of persuasive strategies in everyday scenarios where persuasion is not necessarily warranted.
  arXiv  Detail & Related papers  (2026-04-23T23:01:38Z)
• ADEPT: RL-Aligned Agentic Decoding of Emotion via Evidence Probing Tools -- From Consensus Learning to Ambiguity-Driven Emotion Reasoning [67.22219034602514]
  We introduce ADEPT (Agentic Decoding of Emotion via Evidence Probing Tools), a framework that reframes emotion recognition as a multi-turn inquiry process.<n> ADEPT transforms an SLLM into an agent that maintains an evolving candidate emotion set and adaptively invokes dedicated semantic and acoustic probing tools.<n>We show that ADEPT improves primary emotion accuracy in most settings while substantially improving minor emotion characterization.
  arXiv  Detail & Related papers  (2026-02-13T08:33:37Z)
• Dual Latent Memory for Visual Multi-agent System [69.29799381195592]
  Visual Multi-Agent Systems (VMAS) promise to enhance comprehensive abilities through inter-agent collaboration.<n>Increasing agent turns often degrades performance while exponentially inflating token costs.<n>We propose L$2$-VMAS, a novel model-agnostic framework that enables inter-agent collaboration with dual latent memories.
  arXiv  Detail & Related papers  (2026-01-31T02:49:10Z)
• From Passive to Persuasive: Steering Emotional Nuance in Human-AI Negotiation [14.919711528167054]
  This paper demonstrates that targeted activation engineering can steer LLaMA 3.1-8B to exhibit more human-like emotional nuances.<n>Applying these vectors to new conversational prompts significantly enhances emotional characteristics.
  arXiv  Detail & Related papers  (2025-11-16T23:33:06Z)
• State Your Intention to Steer Your Attention: An AI Assistant for Intentional Digital Living [29.694439220861792]
  We introduce a novel AI assistant that elicits a user's intention, assesses whether ongoing activities are in line with that intention, and provides gentle nudges when deviations occur.<n>The system leverages a large language model to analyze screenshots, application titles, and URLs, issuing notifications when behavior diverges from the stated goal.<n>Results indicate that our AI assistant effectively supports users in maintaining focus and aligning their digital behavior with their intentions.
  arXiv  Detail & Related papers  (2025-10-16T09:57:35Z)
• Effect of Static vs. Conversational AI-Generated Messages on Colorectal Cancer Screening Intent: a Randomized Controlled Trial [5.429833789548265]
  Large language model (LLM) chatbots show increasing promise in persuasive communication.<n>We enrolled 915 U.S. adults (ages 45-75) who had never completed colorectal cancer (CRC) screening.<n>Both AI interventions significantly increased stool test intentions by over 12 points (12.9-13.8/100), compared to a 7.5 gain for expert materials.
  arXiv  Detail & Related papers  (2025-07-10T22:46:43Z)
• Conversations: Love Them, Hate Them, Steer Them [10.014248704653]
  Large Language Models (LLMs) demonstrate increasing conversational fluency, yet instilling them with nuanced, human-like emotional expression remains a significant challenge.<n>This paper demonstrates that targeted activation engineering can steer LLaMA 3.1-8B to exhibit more human-like emotional nuances.
  arXiv  Detail & Related papers  (2025-05-23T02:58:45Z)
• Must Read: A Systematic Survey of Computational Persuasion [60.83151988635103]
  AI-driven persuasion can be leveraged for beneficial applications, but also poses threats through manipulation and unethical influence.<n>Our survey outlines future research directions to enhance the safety, fairness, and effectiveness of AI-powered persuasion.
  arXiv  Detail & Related papers  (2025-05-12T17:26:31Z)
• Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Models [75.85319609088354]
  Sentient Agent as a Judge (SAGE) is an evaluation framework for large language models.<n>SAGE instantiates a Sentient Agent that simulates human-like emotional changes and inner thoughts during interaction.<n>SAGE provides a principled, scalable and interpretable tool for tracking progress toward genuinely empathetic and socially adept language agents.
  arXiv  Detail & Related papers  (2025-05-01T19:06:10Z)
• Investigating Affective Use and Emotional Well-being on ChatGPT [32.797983866308755]
  We investigate the extent to which interactions with ChatGPT may impact users' emotional well-being, behaviors and experiences.<n>We analyze over 3 million conversations for affective cues and surveying over 4,000 users on their perceptions of ChatGPT.<n>We conduct an Institutional Review Board (IRB)-approved randomized controlled trial (RCT) on close to 1,000 participants over 28 days.
  arXiv  Detail & Related papers  (2025-04-04T19:22:10Z)
• Human Decision-making is Susceptible to AI-driven Manipulation [87.24007555151452]
  AI systems may exploit users' cognitive biases and emotional vulnerabilities to steer them toward harmful outcomes.<n>This study examined human susceptibility to such manipulation in financial and emotional decision-making contexts.
  arXiv  Detail & Related papers  (2025-02-11T15:56:22Z)
• Eliciting Language Model Behaviors with Investigator Agents [93.34072434845162]
  Language models exhibit complex, diverse behaviors when prompted with free-form text.<n>We study the problem of behavior elicitation, where the goal is to search for prompts that induce specific target behaviors.<n>We train investigator models to map randomly-chosen target behaviors to a diverse distribution of outputs that elicit them.
  arXiv  Detail & Related papers  (2025-02-03T10:52:44Z)
• Sim-to-Real Causal Transfer: A Metric Learning Approach to Causally-Aware Interaction Representations [58.96953392466609]
  We take an in-depth look at the causal awareness of modern representations of agent interactions.<n>We show that recent representations are already partially resilient to perturbations of non-causal agents.<n>We introduce a metric learning approach that regularizes latent representations with causal annotations.
  arXiv  Detail & Related papers  (2023-12-07T18:57:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.