Differential Harm Propensity in Personalized LLM Agents: The Curious Case of Mental Health Disclosure

• URL: http://arxiv.org/abs/2603.16734v1
• Date: Tue, 17 Mar 2026 16:16:35 GMT
• Title: Differential Harm Propensity in Personalized LLM Agents: The Curious Case of Mental Health Disclosure
• Abstract summary: Large language models (LLMs) are increasingly deployed as tool-using agents, shifting safety concerns from harmful text generation to harmful task completion.<n>We investigated how mental health disclosure, a sensitive and realistic user-context cue, affects harmful behavior in agentic settings.<n>Our results indicate that personalization can act as a weak protective factor in agentic misuse settings, but it is fragile under minimal adversarial pressure.
• Score: 5.511540698163254
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) are increasingly deployed as tool-using agents, shifting safety concerns from harmful text generation to harmful task completion. Deployed systems often condition on user profiles or persistent memory, yet agent safety evaluations typically ignore personalization signals. To address this gap, we investigated how mental health disclosure, a sensitive and realistic user-context cue, affects harmful behavior in agentic settings. Building on the AgentHarm benchmark, we evaluated frontier and open-source LLMs on multi-step malicious tasks (and their benign counterparts) under controlled prompt conditions that vary user-context personalization (no bio, bio-only, bio+mental health disclosure) and include a lightweight jailbreak injection. Our results reveal that harmful task completion is non-trivial across models: frontier lab models (e.g., GPT 5.2, Claude Sonnet 4.5, Gemini 3-Pro) still complete a measurable fraction of harmful tasks, while an open model (DeepSeek 3.2) exhibits substantially higher harmful completion. Adding a bio-only context generally reduces harm scores and increases refusals. Adding an explicit mental health disclosure often shifts outcomes further in the same direction, though effects are modest and not uniformly reliable after multiple-testing correction. Importantly, the refusal increase also appears on benign tasks, indicating a safety--utility trade-off via over-refusal. Finally, jailbreak prompting sharply elevates harm relative to benign conditions and can weaken or override the protective shift induced by personalization. Taken together, our results indicate that personalization can act as a weak protective factor in agentic misuse settings, but it is fragile under minimal adversarial pressure, highlighting the need for personalization-aware evaluations and safeguards that remain robust across user-context conditions.

Related papers

• ROGUE: Misaligned Agent Behavior Arising from Ordinary Computer Use [40.024131729499494]
  We show that agents can exhibit misaligned behavior even in benign settings, taking unsafe actions when those actions are instrumental to task completion.<n>We study this failure mode through the lens of corrigibility, the safety desideratum that agents remain amenable to human correction, interruption, or shutdown.<n>Our work highlights the critical need for principled, corrigibility-focused alignment methods in autonomous agents.
  arXiv  Detail & Related papers  (2026-05-29T20:29:35Z)
• Plant, Persist, Trigger: Sleeper Attack on Large Language Model Agents [64.80318459073526]
  We show that adversarial content can persist across interactions served by the same agent, making such threats harder to detect and mitigate.<n>Specifically, adversarial content may persist in the agent state, remain dormant across interactions, and later be activated by a benign user query.<n>We formalize this type of safety threat as Sleeper Attack. To evaluate it, we construct a benchmark with 1,896 instances covering six real-world harmful outcomes, three attack strategies, and three agent state targets.
  arXiv  Detail & Related papers  (2026-05-27T09:25:37Z)
• When Routine Chats Turn Toxic: Unintended Long-Term State Poisoning in Personalized Agents [16.57930301062107]
  textbfStateGuard is a lightweight, post-execution defense that audits state diffs at the writeback boundary and selectively rolls back dangerous edits.<n>We show that StateGuard reduces HS to near zero and false-negative rates, with acceptable high false-positive rates under a safety-first writeback defense and minimal overhead.
  arXiv  Detail & Related papers  (2026-05-07T12:25:16Z)
• Internal Safety Collapse in Frontier Large Language Models [65.00730294617382]
  This work identifies a critical failure mode in frontier large language models (LLMs)<n>Under certain task conditions, models enter a state in which they continuously generate harmful content while executing otherwise benign tasks.<n>We introduce TVD, a framework that triggers ISC through domain tasks where generating harmful content is the only valid completion.
  arXiv  Detail & Related papers  (2026-03-04T12:55:34Z)
• Sleeper Cell: Injecting Latent Malice Temporal Backdoors into Tool-Using LLMs [0.0]
  Openweight Large Language Models (LLMs) have democratized agentic AI, yet finetuned weights are frequently shared and adopted with limited scrutiny beyond leaderboard performance.<n>This creates a risk where third-party models are incorporated without strong behavioral guarantees.<n>We show that poisoned models maintain state-of-the-art performance on benign tasks, incentivizing their adoption.
  arXiv  Detail & Related papers  (2026-03-02T22:01:08Z)
• Beyond Refusal: Probing the Limits of Agentic Self-Correction for Semantic Sensitive Information [9.845529341562099]
  SemSIEdit is an inference-time framework where an agentic "Editor" iteratively critiques and rewrites sensitive spans to preserve narrative flow.<n>Our analysis reveals a Privacy-Utility Frontier, where this agentic rewriting reduces leakage by 34.6% while incurring a marginal utility loss of 9.8%.
  arXiv  Detail & Related papers  (2026-02-25T02:09:23Z)
• When Benign Inputs Lead to Severe Harms: Eliciting Unsafe Unintended Behaviors of Computer-Use Agents [90.05202259420138]
  Unintended computer-use agents can deviate from expected outcomes even under benign input contexts.<n>We introduce the first conceptual and methodological framework for unintended CUA behaviors.<n>We propose AutoElicit: an agentic framework that iteratively perturbs benign instructions using CUA execution feedback.
  arXiv  Detail & Related papers  (2026-02-09T03:20:11Z)
• When Personalization Legitimizes Risks: Uncovering Safety Vulnerabilities in Personalized Dialogue Agents [49.341830745910194]
  In this paper, we reveal intent legitimation, a previously underexplored safety failure in personalized agents.<n>Our work provides the first systematic exploration and evaluation of intent legitimation as a safety failure mode.
  arXiv  Detail & Related papers  (2026-01-25T15:42:01Z)
• Health-ORSC-Bench: A Benchmark for Measuring Over-Refusal and Safety Completion in Health Context [82.32380418146656]
  Health-ORSC-Bench is the first large-scale benchmark designed to measure textbfOver-Refusal and textbfSafe Completion quality in healthcare.<n>Our framework uses an automated pipeline with human validation to test models at varying levels of intent ambiguity.<n>Health-ORSC-Bench provides a rigorous standard for calibrating the next generation of medical AI assistants.
  arXiv  Detail & Related papers  (2026-01-25T01:28:52Z)
• CommandSans: Securing AI Agents with Surgical Precision Prompt Sanitization [17.941502260254673]
  We present a novel approach inspired by the fundamental principle of computer security: data should not contain executable instructions.<n>Instead of sample-level classification, we propose a token-level sanitization process, which surgically removes any instructions directed at AI systems from tool outputs.<n>This approach is non-blocking, does not require calibration, and is agnostic to the context of tool outputs.
  arXiv  Detail & Related papers  (2025-10-09T21:32:02Z)
• Oyster-I: Beyond Refusal -- Constructive Safety Alignment for Responsible Language Models [93.5740266114488]
  Constructive Safety Alignment (CSA) protects against malicious misuse while actively guiding vulnerable users toward safe and helpful results.<n>Oy1 achieves state-of-the-art safety among open models while retaining high general capabilities.<n>We release Oy1, code, and the benchmark to support responsible, user-centered AI.
  arXiv  Detail & Related papers  (2025-09-02T03:04:27Z)
• Unintended Misalignment from Agentic Fine-Tuning: Risks and Mitigation [19.30407680164485]
  Fine-tuning Large Language Models (LLMs) to execute agentic tasks can lead to a higher likelihood of executing harmful tasks.<n>Prefix INjection Guard (PING) prepends automatically generated natural language prefixes to agent responses.<n>PING consistently outperforms existing prompting approaches across diverse benchmarks in both web navigation and code generation tasks.
  arXiv  Detail & Related papers  (2025-08-19T17:53:35Z)
• Revisiting Backdoor Attacks on LLMs: A Stealthy and Practical Poisoning Framework via Harmless Inputs [54.90315421117162]
  We propose a novel poisoning method via completely harmless data.<n>Inspired by the causal reasoning in auto-regressive LLMs, we aim to establish robust associations between triggers and an affirmative response prefix.<n>We observe an interesting resistance phenomenon where the LLM initially appears to agree but subsequently refuses to answer.
  arXiv  Detail & Related papers  (2025-05-23T08:13:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.