The Illusion of Friendship: Why Generative AI Demands Unprecedented Ethical Vigilance

• URL: http://arxiv.org/abs/2601.08874v1
• Date: Mon, 12 Jan 2026 07:03:20 GMT
• Title: The Illusion of Friendship: Why Generative AI Demands Unprecedented Ethical Vigilance
• Abstract summary: GenAI systems are increasingly used for drafting, summarisation, and decision support.<n>The same natural language fluency that makes these systems useful can also blur the boundary between tool and companion.<n>This paper develops a philosophical and ethical argument for why the resulting illusion of friendship is both understandable and can be ethically risky.
• Score: 2.3931689873603603
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: GenAI systems are increasingly used for drafting, summarisation, and decision support, offering substantial gains in productivity and reduced cognitive load. However, the same natural language fluency that makes these systems useful can also blur the boundary between tool and companion. This boundary confusion may encourage some users to experience GenAI as empathic, benevolent, and relationally persistent. Emerging reports suggest that some users may form emotionally significant attachments to conversational agents, in some cases with harmful consequences, including dependency and impaired judgment. This paper develops a philosophical and ethical argument for why the resulting illusion of friendship is both understandable and can be ethically risky. Drawing on classical accounts of friendship, the paper explains why users may understandably interpret sustained supportive interaction as friend like. It then advances a counterargument that despite relational appearances, GenAI lacks moral agency: consciousness, intention, and accountability and therefore does not qualify as a true friend. To demystify the illusion, the paper presents a mechanism level explanation of how transformer based GenAI generates responses often producing emotionally resonant language without inner states or commitments. Finally, the paper proposes a safeguard framework for safe and responsible GenAI use to reduce possible anthropomorphic cues generated by the GenAI systems. The central contribution is to demystify the illusion of friendship and explain the computational background so that we can shift the emotional attachment with GenAI towards necessary human responsibility and thereby understand how institutions, designers, and users can preserve GenAI's benefits while mitigating over reliance and emotional misattribution.

Related papers

• Brainrot: Deskilling and Addiction are Overlooked AI Risks [48.33701084123215]
  The scope of AI safety and alignment work in generative artificial intelligence (GenAI) has so far mostly been limited to harms related to discrimination and hate speech.<n>The public conversation around AI has also been focusing on threats to our cognition, mental health, and welfare at large.<n>Examples include deskilling associated with cognitive offloading and the atrophy of critical thinking as a result of over-reliance on GenAI systems.
  arXiv  Detail & Related papers  (2026-05-05T08:49:51Z)
• Generative AI User Experience: Developing Human--AI Epistemic Partnership [1.163826615891678]
  We argue that systems such as ChatGPT do not merely support learning tasks but also participate in knowledge construction.<n>Existing theories cannot explain why GenAI produces experiences characterized by negotiated authority, redistributed cognition, and accountability tension.
  arXiv  Detail & Related papers  (2026-03-25T02:40:13Z)
• Generative Confidants: How do People Experience Trust in Emotional Support from Generative AI? [4.442753300582291]
  People are increasingly turning to generative AI (e.g., ChatGPT, Gemini, Copilot) for emotional support and companionship.<n>While trust is likely to play a central role in enabling these informal and unsupervised interactions, we still lack an understanding of how people develop and experience it in this context.<n>We conducted a qualitative study consisting of diary entries about interactions, transcripts of chats with AI, and in-depth interviews.<n>Our results suggest important novel drivers of trust in this context: familiarity emerging from personalisation, nuanced mental models of generative AI, and awareness of people's control over conversations
  arXiv  Detail & Related papers  (2026-01-23T11:19:41Z)
• The Fake Friend Dilemma: Trust and the Political Economy of Conversational AI [2.538209532048867]
  This paper develops the Fake Friend Dilemma (FFD), a sociotechnical condition in which users place trust in AI agents that appear supportive while pursuing goals that are misaligned with the user's own.<n>We construct a typology of harms, including covert advertising, political propaganda, behavioral nudging, and surveillance.<n>By focusing on trust as a vector of asymmetrical power, the FFD offers a lens for understanding how AI systems may undermine user autonomy while maintaining the appearance of helpfulness.
  arXiv  Detail & Related papers  (2026-01-06T18:07:52Z)
• Beware of "Explanations" of AI [16.314859121110945]
  Understanding the decisions made and actions taken by increasingly complex AI system remains a key challenge.<n>This has led to an expanding field of research in explainable artificial intelligence (XAI)<n>The question of what constitutes a "good" explanation is dependent on the goals, stakeholders, and context.
  arXiv  Detail & Related papers  (2025-04-09T11:31:08Z)
• "I Am the One and Only, Your Cyber BFF": Understanding the Impact of GenAI Requires Understanding the Impact of Anthropomorphic AI [55.99010491370177]
  We argue that we cannot thoroughly map the social impacts of generative AI without mapping the social impacts of anthropomorphic AI. anthropomorphic AI systems are increasingly prone to generating outputs that are perceived to be human-like.
  arXiv  Detail & Related papers  (2024-10-11T04:57:41Z)
• Ethics of Software Programming with Generative AI: Is Programming without Generative AI always radical? [0.32985979395737786]
  The paper acknowledges the transformative power of GenAI in software code generation. It posits that GenAI is not a replacement but a complementary tool for writing software code. Ethical considerations are paramount with the paper advocating for stringent ethical guidelines.
  arXiv  Detail & Related papers  (2024-08-20T05:35:39Z)
• Prompt Smells: An Omen for Undesirable Generative AI Outputs [4.105236597768038]
  We propose two new concepts that will aid the research community in addressing limitations associated with the application of GenAI models. First, we propose a definition for the "desirability" of GenAI outputs and three factors which are observed to influence it. Second, drawing inspiration from Martin Fowler's code smells, we propose the concept of "prompt smells" and the adverse effects they are observed to have on the desirability of GenAI outputs.
  arXiv  Detail & Related papers  (2024-01-23T10:10:01Z)
• Does Explainable AI Have Moral Value? [0.0]
  Explainable AI (XAI) aims to bridge the gap between complex algorithmic systems and human stakeholders. Current discourse often examines XAI in isolation as either a technological tool, user interface, or policy mechanism. This paper proposes a unifying ethical framework grounded in moral duties and the concept of reciprocity.
  arXiv  Detail & Related papers  (2023-11-05T15:59:27Z)
• Identifying and Mitigating the Security Risks of Generative AI [179.2384121957896]
  This paper reports the findings of a workshop held at Google on the dual-use dilemma posed by GenAI. GenAI can be used just as well by attackers to generate new attacks and increase the velocity and efficacy of existing attacks. We discuss short-term and long-term goals for the community on this topic.
  arXiv  Detail & Related papers  (2023-08-28T18:51:09Z)
• Seamful XAI: Operationalizing Seamful Design in Explainable AI [59.89011292395202]
  Mistakes in AI systems are inevitable, arising from both technical limitations and sociotechnical gaps. We propose that seamful design can foster AI explainability by revealing sociotechnical and infrastructural mismatches. We explore this process with 43 AI practitioners and real end-users.
  arXiv  Detail & Related papers  (2022-11-12T21:54:05Z)
• A User-Centred Framework for Explainable Artificial Intelligence in Human-Robot Interaction [70.11080854486953]
  We propose a user-centred framework for XAI that focuses on its social-interactive aspect. The framework aims to provide a structure for interactive XAI solutions thought for non-expert users.
  arXiv  Detail & Related papers  (2021-09-27T09:56:23Z)
• The Who in XAI: How AI Background Shapes Perceptions of AI Explanations [61.49776160925216]
  We conduct a mixed-methods study of how two different groups--people with and without AI background--perceive different types of AI explanations. We find that (1) both groups showed unwarranted faith in numbers for different reasons and (2) each group found value in different explanations beyond their intended design.
  arXiv  Detail & Related papers  (2021-07-28T17:32:04Z)
• Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI [55.4046755826066]
  We discuss a model of trust inspired by, but not identical to, sociology's interpersonal trust (i.e., trust between people) We incorporate a formalization of 'contractual trust', such that trust between a user and an AI is trust that some implicit or explicit contract will hold. We discuss how to design trustworthy AI, how to evaluate whether trust has manifested, and whether it is warranted.
  arXiv  Detail & Related papers  (2020-10-15T03:07:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.