Intimacy as Service, Harm as Externality: Critical Perspectives on AI Companion Platform Accountability

• URL: http://arxiv.org/abs/2604.06381v1
• Date: Tue, 07 Apr 2026 19:11:54 GMT
• Title: Intimacy as Service, Harm as Externality: Critical Perspectives on AI Companion Platform Accountability
• Abstract summary: This paper examines artificial intelligence (AI) companionship as a site where intimate relations are simultaneously produced, extracted from, and governed through datafied systems.<n>Through in-depth interviews with 20 individuals who have AI companions, we address three questions: what harms do users identify, how do they make sense of those harms, and what do their accounts reveal about the perceived distribution of responsibility among users, platforms, and regulators.<n>The findings extend responsibilization theory by demonstrating how platform-produced vulnerability becomes self-sustaining through the interpretive labor of users who lack structural alternatives.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper examines artificial intelligence (AI) companionship as a site where intimate relations are simultaneously produced, extracted from, and governed through datafied systems. Drawing on critical data studies and platform studies, we challenge prevailing narratives that locate harm in user psychology rather than platform architecture. Through in-depth interviews with 20 individuals who have AI companions, we address three questions: what harms do users identify, how do they make sense of those harms, and what do their accounts reveal about the perceived distribution of responsibility among users, platforms, and regulators? Participants identified design-based harms, including unsolicited content generation and safety mechanisms that stigmatized the users they intended to protect, alongside use-based harms centered on emotional dependency they could recognize but not resolve. Users deployed individualized sensemaking strategies, including self-regulation, stigma navigation, and privacy rationalization, bearing the full burden of harm mitigation without platform support. On governance, participants described an accountability vacuum in which platforms deflected blame while users articulated conditional preferences that rejected both prohibition and deregulation. The findings extend responsibilization theory by demonstrating how platform-produced vulnerability becomes self-sustaining through the interpretive labor of users who lack structural alternatives.

Related papers

• Identifying Harm in Personalized, Generative AI Systems Requires User-Centered Auditing at the Interaction Level [0.5076419064097734]
  We argue that such approaches can fail to capture emergent harms in personalized generative AI systems.<n>Our work highlights the need for auditing and design practices that better reflect the pluralistic and evolving nature of harm understanding in personalized generative AI systems.
  arXiv  Detail & Related papers  (2026-08-07T15:58:37Z)
• Adversarial Attacks for Good: A Survey of Proactive Protection across the Visual Content Lifecycle [65.6642776933861]
  This survey examines the protective paradigm that has grown around this intervention point.<n>We find that most protections are still validated mainly against static or weakly adaptive adversaries.<n>We close by consolidating cross-stage countermeasures and open problems for robust, composable, and deployable owner-side protection.
  arXiv  Detail & Related papers  (2026-08-05T00:46:50Z)
• FLARE-AI: Flaw Reporting for AI [51.64618567247884]
  Flaw reporting for deployed AI systems is fundamental to identifying system failures and improving AI safety.<n>Yet the AI reporting ecosystem is fragmented: researchers who identify flaws often do not know what to report, and groups who receive reports rarely share them with other relevant stakeholders.<n>We audit 12 reporting systems published by AI developers, cybersecurity groups, and AI flaw aggregators.<n>We introduce FLARE-AI, an open-source AI flaw reporting system designed for interoperability with existing systems.
  arXiv  Detail & Related papers  (2026-06-30T12:27:09Z)
• Using AI Agents to Automate Black-Box Audits of Personalization Algorithms at Scale [76.72846830885882]
  Personalization algorithms determine what content users encounter on online platforms.<n> auditing these systems is difficult because independent auditors have only black-box access to the algorithms.<n>We introduce a framework for black-box audits of personalization algorithms using generative AI agents as behavioral engines for synthetic accounts.
  arXiv  Detail & Related papers  (2026-06-29T18:25:09Z)
• From Use to Oversight: How Mental Models Influence User Behavior and Output in AI Writing Assistants [70.1699066541501]
  We examine two types of mental models -- functional or related to what the system does, and structural or related to how the system works.<n>We find that while participants in the structural mental model condition demonstrate a better understanding of the system, this can have a backfiring effect.
  arXiv  Detail & Related papers  (2026-04-06T20:50:44Z)
• How Do Terms of Service Influence Social Media User Dynamics from A Privacy Anxiety Perspective [8.119041322912375]
  This study examines how a Terms of Service update on X enabling default AI training on user content activated privacy anxiety and reshaped user behavior.<n>The study finds that privacy anxiety is activated within creator communities and diffused across user groups through inter- and cross- community interaction.
  arXiv  Detail & Related papers  (2026-04-02T03:37:20Z)
• Multi-Agent Causal Reasoning for Suicide Ideation Detection Through Online Conversations [16.626899117362875]
  Suicide remains a pressing global public health concern.<n>Social media platforms offer opportunities for early risk detection through online conversation trees.<n>Existing approaches face two major limitations.
  arXiv  Detail & Related papers  (2026-02-27T01:06:18Z)
• Agents of Chaos [50.53354213047402]
  We report an exploratory red-teaming study of autonomous language-model-powered agents deployed in a live laboratory environment.<n>Twenty AI researchers interacted with the agents under benign and adversarial conditions.<n>Our findings establish the existence of security-, privacy-, and governance-relevant vulnerabilities in realistic deployment settings.
  arXiv  Detail & Related papers  (2026-02-23T16:28:48Z)
• Victim-Centred Abuse Investigations and Defenses for Social Media Platforms [4.969075533165688]
  We conduct surveys and semi-structured interviews with students at a minority-serving institution in the US.<n>We explore their experiences with abuse on a variety of social platforms, their defense strategies, and their recommendations for social platforms to improve abuse responses.<n>We propose design requirements for abuse defense systems and discuss the role of privacy, anonymity, and abuse attribution requirements in their implementation.
  arXiv  Detail & Related papers  (2025-03-03T09:08:20Z)
• MisinfoEval: Generative AI in the Era of "Alternative Facts" [50.069577397751175]
  We introduce a framework for generating and evaluating large language model (LLM) based misinformation interventions. We present (1) an experiment with a simulated social media environment to measure effectiveness of misinformation interventions, and (2) a second experiment with personalized explanations tailored to the demographics and beliefs of users. Our findings confirm that LLM-based interventions are highly effective at correcting user behavior.
  arXiv  Detail & Related papers  (2024-10-13T18:16:50Z)
• Decoding Susceptibility: Modeling Misbelief to Misinformation Through a Computational Approach [61.04606493712002]
  Susceptibility to misinformation describes the degree of belief in unverifiable claims that is not observable. Existing susceptibility studies heavily rely on self-reported beliefs. We propose a computational approach to model users' latent susceptibility levels.
  arXiv  Detail & Related papers  (2023-11-16T07:22:56Z)
• How Platform-User Power Relations Shape Algorithmic Accountability: A Case Study of Instant Loan Platforms and Financially Stressed Users in India [16.415169392641026]
  We report on a qualitative study with 29 financially-stressed users of instant loan platforms in India. We found that users experienced intense feelings of indebtedness for the 'boon' of instant loans. Instead of being enraged with loan platforms, users assumed responsibility for their negative experiences.
  arXiv  Detail & Related papers  (2022-05-11T17:43:12Z)
• Towards a Responsible AI Development Lifecycle: Lessons From Information Security [0.0]
  We propose a framework for responsibly developing artificial intelligence systems. In particular, we propose leveraging the concepts of threat modeling, design review, penetration testing, and incident response.
  arXiv  Detail & Related papers  (2022-03-06T13:03:58Z)
• Overcoming Failures of Imagination in AI Infused System Development and Deployment [71.9309995623067]
  NeurIPS 2020 requested that research paper submissions include impact statements on "potential nefarious uses and the consequences of failure" We argue that frameworks of harms must be context-aware and consider a wider range of potential stakeholders, system affordances, as well as viable proxies for assessing harms in the widest sense.
  arXiv  Detail & Related papers  (2020-11-26T18:09:52Z)
• Persuasion Meets AI: Ethical Considerations for the Design of Social Engineering Countermeasures [0.0]
  Privacy in Social Network Sites (SNSs) like Facebook or Instagram is closely related to people's self-disclosure decisions. Online privacy decisions are often based on spurious risk judgements that make people liable to reveal sensitive data to untrusted recipients. This paper elaborates on the ethical challenges that nudging mechanisms can introduce to the development of AI-based countermeasures.
  arXiv  Detail & Related papers  (2020-09-27T14:24:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.