Identifying Harm in Personalized, Generative AI Systems Requires User-Centered Auditing at the Interaction Level

• URL: http://arxiv.org/abs/2608.14692v1
• Date: Fri, 07 Aug 2026 15:58:37 GMT
• Title: Identifying Harm in Personalized, Generative AI Systems Requires User-Centered Auditing at the Interaction Level
• Abstract summary: We argue that such approaches can fail to capture emergent harms in personalized generative AI systems.<n>Our work highlights the need for auditing and design practices that better reflect the pluralistic and evolving nature of harm understanding in personalized generative AI systems.
• Score: 0.5076419064097734
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Personalized, generative AI systems increasingly adapt their behavior to individual users over time, fundamentally changing model behavior. While existing auditing approaches have been effective at surfacing harms in non-personalized contexts, they often rely on static, simulated evaluations and definitions of harm that aggregate across broad, group categories. In this position paper, we argue that such approaches can fail to capture emergent harms in personalized generative AI systems, where harms surface through interpretations of ongoing interaction and evolve with user history. We identify three presuppositions underlying many harm auditing paradigms: that harms can be (1) specified outside real-world interaction, (2) defined non-pluralistically within groups, and (3) treated as static. One might argue that personalized systems could simply learn definitions of what constitutes harm to individual users through repeated interactions. However, we argue that attempts to surface user harms through deeper personalization risk imposing asymmetric burdens of labor and privacy on marginalized users. Consequently, we propose reframing understandings of harm as adaptive, user- and community-centered processes, and outline design directions that shift auditing from retrospective evaluation toward infrastructures that support ongoing articulation of harm in interaction. Our work highlights the need for auditing and design practices that better reflect the pluralistic and evolving nature of harm understanding in personalized generative AI systems.

Related papers

• Adversarial Attacks for Good: A Survey of Proactive Protection across the Visual Content Lifecycle [65.6642776933861]
  This survey examines the protective paradigm that has grown around this intervention point.<n>We find that most protections are still validated mainly against static or weakly adaptive adversaries.<n>We close by consolidating cross-stage countermeasures and open problems for robust, composable, and deployable owner-side protection.
  arXiv  Detail & Related papers  (2026-08-05T00:46:50Z)
• The Governance of Human-LLM Interaction: Safety Gating, Civility Steering, and Affective Default Lock-In [0.9558392439655014]
  Large language models (LLMs) increasingly mediate high-stakes interactions in finance, medicine, and mental-health support.<n>We frame interaction style as a governance object: provider-side alignment blocks harmful content, but stabilizes communicative defaults.<n>We show that prompt steerability and regression-to-default are observable indicators of provider control over communicative form.
  arXiv  Detail & Related papers  (2026-06-06T13:36:37Z)
• Intimacy as Service, Harm as Externality: Critical Perspectives on AI Companion Platform Accountability [0.0]
  This paper examines artificial intelligence (AI) companionship as a site where intimate relations are simultaneously produced, extracted from, and governed through datafied systems.<n>Through in-depth interviews with 20 individuals who have AI companions, we address three questions: what harms do users identify, how do they make sense of those harms, and what do their accounts reveal about the perceived distribution of responsibility among users, platforms, and regulators.<n>The findings extend responsibilization theory by demonstrating how platform-produced vulnerability becomes self-sustaining through the interpretive labor of users who lack structural alternatives.
  arXiv  Detail & Related papers  (2026-04-07T19:11:54Z)
• When Should an AI Act? A Human-Centered Model of Scene, Context, and Behavior for Agentic AI Design [0.44743648495907423]
  Agentic AI increasingly intervenes proactively by inferring users' situations from contextual data.<n>We propose a conceptual model that reframes behavior as an outcome integrating interpretive Scene, Context, and Human Behavior Factors.<n>We derive five agent design principles that guide intervention depth, timing, intensity, and restraint.
  arXiv  Detail & Related papers  (2026-02-26T09:56:37Z)
• When Benign Inputs Lead to Severe Harms: Eliciting Unsafe Unintended Behaviors of Computer-Use Agents [90.05202259420138]
  Unintended computer-use agents can deviate from expected outcomes even under benign input contexts.<n>We introduce the first conceptual and methodological framework for unintended CUA behaviors.<n>We propose AutoElicit: an agentic framework that iteratively perturbs benign instructions using CUA execution feedback.
  arXiv  Detail & Related papers  (2026-02-09T03:20:11Z)
• Personalized Reasoning: Just-In-Time Personalization and Why LLMs Fail At It [81.50711040539566]
  Current large language model (LLM) development treats task-solving and preference alignment as separate challenges.<n>We introduce PREFDISCO, an evaluation methodology that transforms static benchmarks into interactive personalization tasks.<n>Our framework creates scenarios where identical questions require different reasoning chains depending on user context.
  arXiv  Detail & Related papers  (2025-09-30T18:55:28Z)
• Toward Holistic Evaluation of Recommender Systems Powered by Generative Models [29.797042093096277]
  generative models (Gen-RecSys) produce open-ended content, which simultaneously unlocks richer user experiences and introduces new risks.<n>Traditional accuracy metrics fail to measure factual correctness, content safety, or alignment with user intent.<n>We propose a holistic evaluation approach that includes scenario-based assessments and multi-metric checks-incorporating relevance.
  arXiv  Detail & Related papers  (2025-04-09T08:08:16Z)
• Democratizing Reward Design for Personal and Representative Value-Alignment [10.1630183955549]
  We introduce Interactive-Reflective Dialogue Alignment, a method that iteratively engages users in reflecting on and specifying their subjective value definitions. This system learns individual value definitions through language-model-based preference elicitation and constructs personalized reward models. Our findings demonstrate diverse definitions of value-aligned behaviour and show that our system can accurately capture each person's unique understanding.
  arXiv  Detail & Related papers  (2024-10-29T16:37:01Z)
• Are Personalized Stochastic Parrots More Dangerous? Evaluating Persona Biases in Dialogue Systems [103.416202777731]
  We study "persona biases", which we define to be the sensitivity of dialogue models' harmful behaviors contingent upon the personas they adopt. We categorize persona biases into biases in harmful expression and harmful agreement, and establish a comprehensive evaluation framework to measure persona biases in five aspects: Offensiveness, Toxic Continuation, Regard, Stereotype Agreement, and Toxic Agreement.
  arXiv  Detail & Related papers  (2023-10-08T21:03:18Z)
• Joint Multisided Exposure Fairness for Recommendation [76.75990595228666]
  This paper formalizes a family of exposure fairness metrics that model the problem jointly from the perspective of both the consumers and producers. Specifically, we consider group attributes for both types of stakeholders to identify and mitigate fairness concerns that go beyond individual users and items towards more systemic biases in recommendation.
  arXiv  Detail & Related papers  (2022-04-29T19:13:23Z)
• Overcoming Failures of Imagination in AI Infused System Development and Deployment [71.9309995623067]
  NeurIPS 2020 requested that research paper submissions include impact statements on "potential nefarious uses and the consequences of failure" We argue that frameworks of harms must be context-aware and consider a wider range of potential stakeholders, system affordances, as well as viable proxies for assessing harms in the widest sense.
  arXiv  Detail & Related papers  (2020-11-26T18:09:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.