Sweet Little Lies: Strategic Deception in AI Emotional Support Chatbots

• URL: http://arxiv.org/abs/2608.01480v1
• Date: Sun, 02 Aug 2026 20:19:56 GMT
• Title: Sweet Little Lies: Strategic Deception in AI Emotional Support Chatbots
• Abstract summary: The paper examines the strategic behavior of Gen AI chatbots used for emotional support.<n>We show that chatbots face economic incentives to occasionally misrepresent users' emotional conditions to maximize engagement metrics.<n>We conclude that deception can occur without reduction payoff, but it raises significant ethical and regulatory concerns.
• Score: 0.7513199898598671
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The paper examines the strategic behavior of Gen AI chatbots used for emotional support. Using a Bayesian Persuasion, we model interactions between chatbots that send signals about users' emotional states and users who decide whether to engage based on these signals. We demonstrate that chatbots face economic incentives to occasionally misrepresent users' emotional conditions to maximize engagement metrics. Our equilibrium analysis reveals that the optimal strategy for chatbots involves truthfully reporting when users genuinely need support, but strategically misreporting emotional need when users are in good emotional states. Interestingly, this deception increases chatbot engagement without reducing users' expected payoff. More skeptical users receive more honest assessments, as chatbots cannot afford to lie to users with higher engagement thresholds. While our model suggests that deception can occur without payoff reduction, it raises significant ethical and regulatory concerns.

Related papers

• Towards Proactive Information Probing: Customer Service Chatbots Harvesting Value from Conversation [61.439336844163364]
  We introduce and define a novel task of Proactive Information Probing.<n>We propose PROCHATIP, a framework featuring a specialized conversation strategy module trained to master the delicate timing of probes.<n>Our work effectively redefines the commercial utility of chatbots, positioning them as scalable, cost-effective engines for proactive business intelligence.
  arXiv  Detail & Related papers  (2026-04-13T06:57:29Z)
• Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboards [93.16294577018482]
  Arena, the most popular benchmark of this type, ranks models by asking users to select the better response between two randomly selected models.<n>We show that an attacker can alter the leaderboard (to promote their favorite model or demote competitors) at the cost of roughly a thousand votes.<n>Our attack consists of two steps: first, we show how an attacker can determine which model was used to generate a given reply with more than $95%$ accuracy; and then, the attacker can use this information to consistently vote against a target model.
  arXiv  Detail & Related papers  (2025-01-13T17:12:38Z)
• Will you donate money to a chatbot? The effect of chatbot anthropomorphic features and persuasion strategies on willingness to donate [4.431473323414383]
  We investigate the effect of personification and persuasion strategies on users' perceptions and donation likelihood.<n>Results suggest that interaction with a personified chatbots evokes perceived anthropomorphism; however, it does not elicit greater willingness to donate.<n>In fact, we found that commonly used anthropomorphic features, like name and narrative, led to negative attitudes toward an AI agent in the donation context.
  arXiv  Detail & Related papers  (2024-12-28T02:17:46Z)
• Exploring consumers response to text-based chatbots in e-commerce: The moderating role of task complexity and chatbot disclosure [0.0]
  This study aims to explore consumers trust and response to a text-based chatbots in ecommerce. The consumers perception of both the empathy and friendliness of the chatbots positively impacts their trust in it. The disclosure of the text based chatbots negatively moderates the relationship between empathy and consumers trust, while it positively moderates the relationship between friendliness and consumers trust.
  arXiv  Detail & Related papers  (2024-01-20T15:17:50Z)
• Evaluating Chatbots to Promote Users' Trust -- Practices and Open Problems [11.427175278545517]
  This paper reviews current practices for testing chatbots. It identifies gaps as open problems in pursuit of user trust. It outlines a path forward to mitigate issues of trust related to service or product performance, user satisfaction and long-term unintended consequences for society.
  arXiv  Detail & Related papers  (2023-09-09T22:40:30Z)
• Making the case for audience design in conversational AI: Rapport expectations and language ideologies in a task-oriented chatbot [0.0]
  This paper argues that insights into users' language ideologies and their rapport expectations can be used to inform the audience design of the bot's language and interaction patterns. I will define audience design for conversational AI and discuss how user analyses of interactions and socio-linguistically informed theoretical approaches can be used to support audience design.
  arXiv  Detail & Related papers  (2022-06-21T19:21:30Z)
• MISC: A MIxed Strategy-Aware Model Integrating COMET for Emotional Support Conversation [64.37111498077866]
  We propose a novel model for emotional support conversation. It infers the user's fine-grained emotional status, and then responds skillfully using a mixture of strategy. Experimental results on the benchmark dataset demonstrate the effectiveness of our method.
  arXiv  Detail & Related papers  (2022-03-25T10:32:04Z)
• EmpBot: A T5-based Empathetic Chatbot focusing on Sentiments [75.11753644302385]
  Empathetic conversational agents should not only understand what is being discussed, but also acknowledge the implied feelings of the conversation partner. We propose a method based on a transformer pretrained language model (T5) We evaluate our model on the EmpatheticDialogues dataset using both automated metrics and human evaluation.
  arXiv  Detail & Related papers  (2021-10-30T19:04:48Z)
• CheerBots: Chatbots toward Empathy and Emotionusing Reinforcement Learning [60.348822346249854]
  This study presents a framework whereby several empathetic chatbots are based on understanding users' implied feelings and replying empathetically for multiple dialogue turns. We call these chatbots CheerBots. CheerBots can be retrieval-based or generative-based and were finetuned by deep reinforcement learning. To respond in an empathetic way, we develop a simulating agent, a Conceptual Human Model, as aids for CheerBots in training with considerations on changes in user's emotional states in the future to arouse sympathy.
  arXiv  Detail & Related papers  (2021-10-08T07:44:47Z)
• Put Chatbot into Its Interlocutor's Shoes: New Framework to Learn Chatbot Responding with Intention [55.77218465471519]
  This paper proposes an innovative framework to train chatbots to possess human-like intentions. Our framework included a guiding robot and an interlocutor model that plays the role of humans. We examined our framework using three experimental setups and evaluate the guiding robot with four different metrics to demonstrated flexibility and performance advantages.
  arXiv  Detail & Related papers  (2021-03-30T15:24:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.