Manipulation and the AI Act: Large Language Model Chatbots and the Danger of Mirrors

• URL: http://arxiv.org/abs/2503.18387v1
• Date: Mon, 24 Mar 2025 06:56:29 GMT
• Title: Manipulation and the AI Act: Large Language Model Chatbots and the Danger of Mirrors
• Authors: Joshua Krook,
• Abstract summary: Personifying AI chatbots could foreseeably increase their trust with users.<n>However, it could also make them more capable of manipulation, by creating the illusion of a close and intimate relationship with an artificial entity.<n>The European Commission has finalized the AI Act, with the EU Parliament making amendments banning manipulative and deceptive AI systems that cause significant harm to users.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Model chatbots are increasingly taking the form and visage of human beings, adapting human faces, names, voices, personalities, and quirks, including those of celebrities and well-known political figures. Personifying AI chatbots could foreseeably increase their trust with users. However, it could also make them more capable of manipulation, by creating the illusion of a close and intimate relationship with an artificial entity. The European Commission has finalized the AI Act, with the EU Parliament making amendments banning manipulative and deceptive AI systems that cause significant harm to users. Although the AI Act covers harms that accumulate over time, it is unlikely to prevent harms associated with prolonged discussions with AI chatbots. Specifically, a chatbot could reinforce a person's negative emotional state over weeks, months, or years through negative feedback loops, prolonged conversations, or harmful recommendations, contributing to a user's deteriorating mental health.

Related papers

• Increasing happiness through conversations with artificial intelligence [4.225027291187279]
  We found that happiness after AI conversations was higher than after journaling. When discussing negative topics, participants gradually aligned their sentiment with the AI's positivity. Using computational modeling, we find the history of these sentiment prediction errors over the course of a conversation predicts greater post-conversation happiness.
  arXiv  Detail & Related papers  (2025-04-02T19:52:02Z)
• AI Mimicry and Human Dignity: Chatbot Use as a Violation of Self-Respect [0.04260910081285213]
  We argue that interacting with chatbots in this way is incompatible with the dignity of users.<n>We show that, since second-personal respect is premised on reciprocal recognition of second-personal authority, behaving towards chatbots in ways that convey second-personal respect is bound to misfire.
  arXiv  Detail & Related papers  (2025-02-17T19:02:12Z)
• Will you donate money to a chatbot? The effect of chatbot anthropomorphic features and persuasion strategies on willingness to donate [4.431473323414383]
  We investigate the effect of personification and persuasion strategies on users' perceptions and donation likelihood.<n>Results suggest that interaction with a personified chatbots evokes perceived anthropomorphism; however, it does not elicit greater willingness to donate.<n>In fact, we found that commonly used anthropomorphic features, like name and narrative, led to negative attitudes toward an AI agent in the donation context.
  arXiv  Detail & Related papers  (2024-12-28T02:17:46Z)
• LLM Roleplay: Simulating Human-Chatbot Interaction [52.03241266241294]
  We propose a goal-oriented, persona-based method to automatically generate diverse multi-turn dialogues simulating human-chatbot interaction. Our method can simulate human-chatbot dialogues with a high indistinguishability rate.
  arXiv  Detail & Related papers  (2024-07-04T14:49:46Z)
• Evaluating Chatbots to Promote Users' Trust -- Practices and Open Problems [11.427175278545517]
  This paper reviews current practices for testing chatbots. It identifies gaps as open problems in pursuit of user trust. It outlines a path forward to mitigate issues of trust related to service or product performance, user satisfaction and long-term unintended consequences for society.
  arXiv  Detail & Related papers  (2023-09-09T22:40:30Z)
• Towards Healthy AI: Large Language Models Need Therapists Too [41.86344997530743]
  We define Healthy AI to be safe, trustworthy and ethical. We present the SafeguardGPT framework that uses psychotherapy to correct for these harmful behaviors.
  arXiv  Detail & Related papers  (2023-04-02T00:39:12Z)
• Robots with Different Embodiments Can Express and Influence Carefulness in Object Manipulation [104.5440430194206]
  This work investigates the perception of object manipulations performed with a communicative intent by two robots. We designed the robots' movements to communicate carefulness or not during the transportation of objects.
  arXiv  Detail & Related papers  (2022-08-03T13:26:52Z)
• Neural Generation Meets Real People: Building a Social, Informative Open-Domain Dialogue Agent [65.68144111226626]
  Chirpy Cardinal aims to be both informative and conversational. We let both the user and bot take turns driving the conversation. Chirpy Cardinal placed second out of nine bots in the Alexa Prize Socialbot Grand Challenge.
  arXiv  Detail & Related papers  (2022-07-25T09:57:23Z)
• You Don't Know My Favorite Color: Preventing Dialogue Representations from Revealing Speakers' Private Personas [44.82330540456883]
  We show that speakers' personas can be inferred through a simple neural network with high accuracy. We conduct extensive experiments to demonstrate that our proposed defense objectives can greatly reduce the attack accuracy from 37.6% to 0.5%.
  arXiv  Detail & Related papers  (2022-04-26T09:36:18Z)
• CheerBots: Chatbots toward Empathy and Emotionusing Reinforcement Learning [60.348822346249854]
  This study presents a framework whereby several empathetic chatbots are based on understanding users' implied feelings and replying empathetically for multiple dialogue turns. We call these chatbots CheerBots. CheerBots can be retrieval-based or generative-based and were finetuned by deep reinforcement learning. To respond in an empathetic way, we develop a simulating agent, a Conceptual Human Model, as aids for CheerBots in training with considerations on changes in user's emotional states in the future to arouse sympathy.
  arXiv  Detail & Related papers  (2021-10-08T07:44:47Z)
• Put Chatbot into Its Interlocutor's Shoes: New Framework to Learn Chatbot Responding with Intention [55.77218465471519]
  This paper proposes an innovative framework to train chatbots to possess human-like intentions. Our framework included a guiding robot and an interlocutor model that plays the role of humans. We examined our framework using three experimental setups and evaluate the guiding robot with four different metrics to demonstrated flexibility and performance advantages.
  arXiv  Detail & Related papers  (2021-03-30T15:24:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.