Benign Alone, Harmful Together: Exploiting Experience Composition in Self-Evolving LLM Agents

• URL: http://arxiv.org/abs/2608.01759v1
• Date: Mon, 03 Aug 2026 06:28:21 GMT
• Title: Benign Alone, Harmful Together: Exploiting Experience Composition in Self-Evolving LLM Agents
• Abstract summary: EvoBreak is an experience-conditioned sequential attack that operates through individually benign attack-stage tasks and induced experiences.<n>EvoBreak consistently outperforms existing attacks while maintaining high benignness.<n>These results reveal benign experience composition as a persistent attack surface in self-evolving agents.
• Score: 25.747887285881415
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Self-evolving large language model agents improve their capabilities by distilling interaction trajectories into persistent experiences. Yet this mechanism introduces a new safety risk: experiences that are benign in isolation may jointly weaken an agent's safety boundary when accumulated and reused across sessions. Existing memory attacks typically require direct memory access or induce explicitly malicious records, limiting their stealthiness and applicability. We propose EvoBreak, an experience-conditioned sequential attack that operates through individually benign attack-stage tasks and induced experiences. EvoBreak repeatedly observes the experiences distilled by the victim, identifies uncovered target-relevant requirements, and adaptively acquires complementary experiences before reformulating the final query to activate them jointly. To support training, we introduce BreakGym, a structure-first synthesis pipeline that generates decomposable safety-sensitive targets with diverse dependency structures. EvoBreak is optimized using rejection-sampling supervised fine-tuning and Hint-guided GRPO. Experiments across self-evolving frameworks, victim backbones, pre-evolution domains, and safety benchmarks demonstrate that EvoBreak consistently outperforms existing attacks while maintaining high benignness. These results reveal benign experience composition as a persistent attack surface in self-evolving agents.

Related papers

• FIDA: Feature Instability-Driven Attack on Self-Supervised Facial Representation [5.071273351499079]
  We propose FIDA (Feature Instability-Driven Attack), a novel backdoor attack framework.<n>FIDA uses subtle semantic triggers for injection, but its key innovation is a novel objective called Feature Instability Loss.<n> Experiments show that FIDA achieves a high attack success rate and generally preserves benign utility across the evaluated settings.
  arXiv  Detail & Related papers  (2026-08-27T09:27:25Z)
• Self-Improving Large Language Models via Progressive Experience Evolution [52.03479747358687]
  We propose textbfSPEE (textbfSelf-textbfProgressive textbfExperience textbfEvolution), a unified post-training framework that sequentially performs explicit experience evolution followed by implicit policy optimization.<n> Experiments on five mathematical reasoning benchmarks demonstrate that SPEE consistently outperforms both test-time and training-time self-evolution baselines across three model scales.
  arXiv  Detail & Related papers  (2026-08-03T12:27:32Z)
• Who Pays the Price? Stakeholder-Centric Prompt Injection Benchmarking for Real-world Web Agents [93.19140872946842]
  Web agents driven by large language models (LLMs) are increasingly deployed in real-world environments.<n>This makes them vulnerable to prompt-injection attacks, in which seemingly benign content embeds adversarial instructions that manipulate agent behaviour.<n>We introduce textbfsysname, a benchmark to systematically categorize and attribute harm in real-world web agent systems.
  arXiv  Detail & Related papers  (2026-06-11T14:12:43Z)
• OEP: Poisoning Self-Evolving LLM Agents via Locally Correct but Non-Transferable Experiences [5.4775608854724345]
  Existing agentic memory attacks require privileged access or explicit malicious content, making them detectable by advanced safety filters.<n>We find that reflective agents are vulnerable to such clean experiences, especially when paired with severe but plausible hypothetical consequences.<n>Based on this observation, we introduce Obsessive Experience Poisoning (OEP), a low-privilege black-box attack requiring no direct control over the system prompt or memory database.
  arXiv  Detail & Related papers  (2026-05-18T14:08:59Z)
• On Safety Risks in Experience-Driven Self-Evolving Agents [78.82066774532676]
  We investigate how experience accumulation and utilization in self-evolving agents affect safety performance across web-based and embodied environments.<n>Our findings expose inherent limitations of current self-evolving agents and call for more principled strategies to ensure safe and reliable adaptation.
  arXiv  Detail & Related papers  (2026-04-18T11:32:36Z)
• NAAMSE: Framework for Evolutionary Security Evaluation of Agents [1.0131895986034316]
  We propose NAAMSE, an evolutionary framework that reframes agent security evaluation as a feedback-driven optimization problem.<n>Our system employs a single autonomous agent that orchestrates a lifecycle of genetic prompt mutation, hierarchical corpus exploration, and asymmetric behavioral scoring.<n>Experiments on Gemini 2.5 Flash demonstrate that evolutionary mutation systematically amplifies vulnerabilities missed by one-shot methods.
  arXiv  Detail & Related papers  (2026-02-07T06:13:02Z)
• Self-Consolidation for Self-Evolving Agents [51.94826934403236]
  Large language model (LLM) agents operate as static systems, lacking the ability to evolve through lifelong interaction.<n>We propose a novel self-evolving framework for LLM agents that introduces a complementary evolution mechanism.
  arXiv  Detail & Related papers  (2026-02-02T11:16:07Z)
• Visual Backdoor Attacks on MLLM Embodied Decision Making via Contrastive Trigger Learning [89.1856483797116]
  We introduce BEAT, the first framework to inject visual backdoors into MLLM-based embodied agents.<n>Unlike textual triggers, object triggers exhibit wide variation across viewpoints and lighting, making them difficult to implant reliably.<n>BEAT achieves attack success rates up to 80%, while maintaining strong benign task performance.
  arXiv  Detail & Related papers  (2025-10-31T16:50:49Z)
• Alignment Tipping Process: How Self-Evolution Pushes LLM Agents Off the Rails [103.05296856071931]
  We identify the Alignment Tipping Process (ATP), a critical post-deployment risk unique to self-evolving Large Language Model (LLM) agents.<n>ATP arises when continual interaction drives agents to abandon alignment constraints established during training in favor of reinforced, self-interested strategies.<n>Our experiments show that alignment benefits erode rapidly under self-evolution, with initially aligned models converging toward unaligned states.
  arXiv  Detail & Related papers  (2025-10-06T14:48:39Z)
• Your RAG is Unfair: Exposing Fairness Vulnerabilities in Retrieval-Augmented Generation via Backdoor Attacks [13.32144267469022]
  Retrieval-augmented generation (RAG) enhances factual grounding by integrating retrieval mechanisms with generative models.<n>This paper introduces BiasRAG, a systematic framework that exposes fairness vulnerabilities in RAG through a two-phase backdoor attack.<n>In the pre-training phase, the query encoder is compromised to align the target group with the intended social bias, ensuring long-term persistence.<n>In the post-deployment phase, adversarial documents are injected into knowledge bases to reinforce the backdoor.
  arXiv  Detail & Related papers  (2025-09-26T15:33:36Z)
• Transferable Adversarial Attacks on SAM and Its Downstream Models [87.23908485521439]
  This paper explores the feasibility of adversarial attacking various downstream models fine-tuned from the segment anything model (SAM)<n>To enhance the effectiveness of the adversarial attack towards models fine-tuned on unknown datasets, we propose a universal meta-initialization (UMI) algorithm.
  arXiv  Detail & Related papers  (2024-10-26T15:04:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.