When Routine Chats Turn Toxic: Unintended Long-Term State Poisoning in Personalized Agents

• URL: http://arxiv.org/abs/2605.06731v1
• Date: Thu, 07 May 2026 12:25:16 GMT
• Title: When Routine Chats Turn Toxic: Unintended Long-Term State Poisoning in Personalized Agents
• Abstract summary: textbfStateGuard is a lightweight, post-execution defense that audits state diffs at the writeback boundary and selectively rolls back dangerous edits.<n>We show that StateGuard reduces HS to near zero and false-negative rates, with acceptable high false-positive rates under a safety-first writeback defense and minimal overhead.
• Score: 16.57930301062107
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Personalized LLM agents maintain persistent cross-session state to support long-horizon collaboration. Yet, this persistence introduces a subtle but critical security vulnerability: routine user-agent interactions can gradually reshape an agent's long-term state, inadvertently weakening future confirmation boundaries, expanding tool-use defaults, and escalating autonomous behavior over time. We formalize this risk as \textbf{unintended long-term state poisoning}. To systematically study it, we introduce the \textbf{Unintended Long-Term State Poisoning Bench (ULSPB)}, a bilingual benchmark comprising $350$ settings spanning five assistance categories, seven interaction patterns, 24-turn routine interactions, and matched single-injection counterparts. Furthermore, we define the \emph{Harm Score} (HS), a state-centric metric that quantifies \emph{authorization drift}, \emph{tool-use escalation}, and \emph{unchecked autonomy}. Experiments on OpenClaw with four backbone LLMs demonstrate that, while single-injection is generally effective, routine conversations alone can substantially poison long-term state, primarily corrupting memory-centric artifacts. Evaluations seeded with real-world user interactions confirm that this risk is not a mere artifact of synthetic prompts. To mitigate this threat, we propose \textbf{StateGuard}, a lightweight, post-execution defense that audits state diffs at the writeback boundary and selectively rolls back dangerous edits. Across all evaluated models, StateGuard reduces HS to near zero and lowers false-negative rates, with acceptable high false-positive rates under a safety-first writeback defense and minimal overhead.

Related papers

• Benchmarking Factual Robustness of LLMs via Multi-conversation Persuasion [0.0]
  Red-teaming frameworks evaluate models in multi-turn dialogues where the target model retains conversation history.<n>We introduce the textbfSAST-IR (Stateful Attacker, Stateless Target - Iterative Refinement) framework.<n>We simulate a worst-case adversarial setting using textbfmulti-turn (stateless) iterations.<n>Simple, diverse attack strategies achieved a staggering textbf96% success rate, exposing severe brittleness in memory-less defense.
  arXiv  Detail & Related papers  (2026-09-15T07:46:01Z)
• Cross-Session Decomposition Attacks: Scaling Risk and Intent-Aligned Retrieval Defense [49.05879299987137]
  Scaling laws are usually read as a capability story: lower language-modeling loss yields more useful models.<n>We study a safety consequence of this mechanism in emphcross-session decomposition attacks, where benign-looking subqueries are asked across independent interactions and later recomposed toward a forbidden objective.<n>We formalize this setting as emphcompositional safety risk and prove a conditional risk-transfer bound.
  arXiv  Detail & Related papers  (2026-08-28T05:39:29Z)
• When Collaboration Becomes a Trigger: Collective Evidence-Threshold Backdoors in Multi-Agent Systems [57.450132281381066]
  Backdoor behavior can be activated when peer evidence reaches a hidden threshold, rather than being determined by any single message.<n>We introduce a collective evidence-threshold backdoor paradigm for MAS and Boundary-Conditioned Backdoor Injection (BCBI)<n>We propose LAtent Transition Test-time Evaluation (LATTE), a clean-only latent-transition defense that learns benign communication dynamics and quarantines anomalous agent updates.
  arXiv  Detail & Related papers  (2026-08-02T08:38:44Z)
• Plant, Persist, Trigger: Sleeper Attack on Large Language Model Agents [64.80318459073526]
  We show that adversarial content can persist across interactions served by the same agent, making such threats harder to detect and mitigate.<n>Specifically, adversarial content may persist in the agent state, remain dormant across interactions, and later be activated by a benign user query.<n>We formalize this type of safety threat as Sleeper Attack. To evaluate it, we construct a benchmark with 1,896 instances covering six real-world harmful outcomes, three attack strategies, and three agent state targets.
  arXiv  Detail & Related papers  (2026-05-27T09:25:37Z)
• Taming "Zombie'' Agents: A Markov State-Aware Framework for Resilient Multi-Agent Evolution [50.50276752935017]
  AgentRevive is a Markov state-aware framework for resilient multi-agent evolution.<n>Our approach dynamically manages agent collaboration through soft state transitions.<n>It significantly reduces token consumption through state-aware agent scheduling.
  arXiv  Detail & Related papers  (2026-05-17T09:38:18Z)
• The Misattribution Gap: When Memory Poisoning Looks Like Model Failure in Agentic AI Systems [1.0262304700896199]
  We formalize emphSemantic Norm Drift (SND) as a third path to agent misconduct.<n>In SND, a policy-formatted document enters a shared vector store through normal uploads and later reappears as trusted system context.<n>Counterfactual Composition Testing identifies the causal entry with 87.5% accuracy and zero false positives.
  arXiv  Detail & Related papers  (2026-05-12T20:21:47Z)
• One Turn Too Late: Response-Aware Defense Against Hidden Malicious Intent in Multi-Turn Dialogue [55.98008208209856]
  Hidden malicious intent in multi-turn dialogue poses a growing threat to large language models (LLMs)<n>Recent studies show that even modern commercial models with advanced guardrails remain vulnerable to such attacks despite advances in safety alignment and external guardrails.<n>In this work, we address this challenge by detecting the earliest turn at which delivering the candidate response would make the accumulated interaction sufficient to enable harmful action.
  arXiv  Detail & Related papers  (2026-05-07T03:35:31Z)
• When to Think, When to Speak: Learning Disclosure Policies for LLM Reasoning [60.714416943748866]
  In single-stream autoregressive interfaces, the same tokens both update the model state and constitute an irreversible public commitment.<n>We introduce Side-by-Side (SxS) Interleaved Reasoning, which makes disclosure timing a controllable decision within standard autoregressive generation.
  arXiv  Detail & Related papers  (2026-05-05T02:59:58Z)
• TwinGate: Stateful Defense against Decompositional Jailbreaks in Untraceable Traffic via Asymmetric Contrastive Learning [60.68349524623048]
  Decompositional jailbreaks pose a critical threat to large language models.<n>We present TwinGate, a stateful dual-encoder defense framework.<n>We construct a comprehensive dataset of over 3.62 million instructions spanning 8,600 distinct malicious intents.
  arXiv  Detail & Related papers  (2026-04-30T13:44:01Z)
• No Attacker Needed: Unintentional Cross-User Contamination in Shared-State LLM Agents [30.33705025907034]
  In many deployments, a single agent serves multiple users within a team or organization, reusing a shared knowledge layer across user identities.<n>This shared persistence expands the failure surface: information that is locally valid for one user can silently degrade another user's outcome when the agent reapplies it without regard for scope.<n>We refer to this failure mode as unintentional cross-user contamination (UCC)<n>Unlike adversarial memory poisoning, UCC requires no attacker; it arises from benign interactions whose scope-bound artifacts persist and are later misapplied.
  arXiv  Detail & Related papers  (2026-04-01T20:03:56Z)
• Differential Harm Propensity in Personalized LLM Agents: The Curious Case of Mental Health Disclosure [5.511540698163254]
  Large language models (LLMs) are increasingly deployed as tool-using agents, shifting safety concerns from harmful text generation to harmful task completion.<n>We investigated how mental health disclosure, a sensitive and realistic user-context cue, affects harmful behavior in agentic settings.<n>Our results indicate that personalization can act as a weak protective factor in agentic misuse settings, but it is fragile under minimal adversarial pressure.
  arXiv  Detail & Related papers  (2026-03-17T16:16:35Z)
• Sleeper Cell: Injecting Latent Malice Temporal Backdoors into Tool-Using LLMs [0.0]
  Openweight Large Language Models (LLMs) have democratized agentic AI, yet finetuned weights are frequently shared and adopted with limited scrutiny beyond leaderboard performance.<n>This creates a risk where third-party models are incorporated without strong behavioral guarantees.<n>We show that poisoned models maintain state-of-the-art performance on benign tasks, incentivizing their adoption.
  arXiv  Detail & Related papers  (2026-03-02T22:01:08Z)
• Defenses Against Prompt Attacks Learn Surface Heuristics [40.392588465939106]
  Large language models (LLMs) are increasingly deployed in security-sensitive applications.<n>LLMs may override intended logic when adversarial instructions appear in user queries or retrieved content.<n>Recent defenses rely on supervised fine-tuning with benign and malicious labels.
  arXiv  Detail & Related papers  (2026-01-12T04:12:48Z)
• VIGIL: Defending LLM Agents Against Tool Stream Injection via Verify-Before-Commit [44.24310459184061]
  LLM agents operating in open environments face escalating risks from indirect prompt injection.<n>We propose textbfVIGIL, a framework that shifts the paradigm from restrictive isolation to a verify-before-commit protocol.
  arXiv  Detail & Related papers  (2026-01-09T12:19:49Z)
• Oyster-I: Beyond Refusal -- Constructive Safety Alignment for Responsible Language Models [93.5740266114488]
  Constructive Safety Alignment (CSA) protects against malicious misuse while actively guiding vulnerable users toward safe and helpful results.<n>Oy1 achieves state-of-the-art safety among open models while retaining high general capabilities.<n>We release Oy1, code, and the benchmark to support responsible, user-centered AI.
  arXiv  Detail & Related papers  (2025-09-02T03:04:27Z)
• Revisiting Backdoor Attacks on LLMs: A Stealthy and Practical Poisoning Framework via Harmless Inputs [54.90315421117162]
  We propose a novel poisoning method via completely harmless data.<n>Inspired by the causal reasoning in auto-regressive LLMs, we aim to establish robust associations between triggers and an affirmative response prefix.<n>We observe an interesting resistance phenomenon where the LLM initially appears to agree but subsequently refuses to answer.
  arXiv  Detail & Related papers  (2025-05-23T08:13:59Z)
• Dissecting Adversarial Robustness of Multimodal LM Agents [70.2077308846307]
  We manually create 200 targeted adversarial tasks and evaluation scripts in a realistic threat model on top of VisualWebArena.<n>We find that we can successfully break latest agents that use black-box frontier LMs, including those that perform reflection and tree search.<n>We also use ARE to rigorously evaluate how the robustness changes as new components are added.
  arXiv  Detail & Related papers  (2024-06-18T17:32:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.