Safety in Self-Evolving LLM Agent Systems: Threats, Amplification, and Case Studies

• URL: http://arxiv.org/abs/2606.23075v1
• Date: Mon, 22 Jun 2026 09:23:50 GMT
• Title: Safety in Self-Evolving LLM Agent Systems: Threats, Amplification, and Case Studies
• Abstract summary: Self-evolving agent systems introduce a qualitatively new threat landscape in which adversarial influences become permanently encoded.<n>We present a systematic security and privacy analysis organized around the Module-Lifecycle Attack Surface matrix.<n>Our findings establish that self-evolution converts every known attack category from session-bounded to lineage-persistent.
• Score: 62.65225104423404
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Self-evolving LLM agent systems, which autonomously update their model parameters, memory, tools, and architectures, introduce a qualitatively new threat landscape in which adversarial influences become permanently encoded, self-amplify across generations, and propagate through populations without sustained attacker access. We present a systematic security and privacy analysis organized around the Module-Lifecycle Attack Surface (MLAS) matrix, which decomposes the attack surface into five functional modules (Brain, Cognitive Resource, Execution, Self-Design, Collective) $\times$ five lifecycle stages (Bootstrap, Propose, Evaluate, Commit, Serve). Analysis of the resulting 25 cells reveals that 17 face critical threats for which no effective partial mitigation. We identify seven cross-cutting amplification effects that interact synergistically and cannot be addressed by securing individual modules in isolation. Comparative case studies of two open-source frameworks demonstrate that evolution-native design activates $3.5\times$ more attack surface cells and achieves a 100% attack persistence rate (40/40 payloads across all CIA+Privacy categories), while co-located security scanners block only 2.5% of attacks. Our findings establish that self-evolution converts every known attack category from session-bounded to lineage-persistent, gives rise to entirely new attack classes, and renders static defenses structurally inadequate, motivating evolution-aware security frameworks and formal verification for self-modifying systems.

Related papers

• A Self-Evolving Multi-Agent Framework Defense against LLM Jailbreak Attacks [53.12252890906191]
  Large language models (LLMs) remain vulnerable to jailbreak attacks.<n>We propose a self-evolving test-time defense built around a persistent, cross-interaction rule memory.
  arXiv  Detail & Related papers  (2026-08-26T16:52:06Z)
• When Agents Act on Web3: An Attack-Surface Survey of MCP, Skills, and Tool Calling [9.566535604762787]
  This survey argues that four properties of that layer (irreversibility, signing authority, continuous autonomy, and sequence-level composition) qualitatively change the threat model.<n>We organize the fragmented MCP-security literature into an attack-surface taxonomy, then contribute a Web3 risk-mapping matrix.<n>We synthesize defenses, including emerging blockchain-based mechanisms, and find them improving but insufficient.
  arXiv  Detail & Related papers  (2026-08-18T02:04:58Z)
• Stateful Cooperative Agents Safeguarding LLMs Against Evolving Multi-Turn Attacks [56.45762972211923]
  We present a proactive defense framework for securing LLMs against evolving multi-turn adversarial attacks.<n>It employs a cooperative multi-agent architecture in which specialized agents execute complementary defense strategies.<n>We present the EMRA dataset designed to simulate evolving strategies across multi-turn attacks.
  arXiv  Detail & Related papers  (2026-07-31T14:04:49Z)
• Belief-Space Quantum-Inspired Reinforcement Learning for Partially Observable Autonomous Cyber Defense in the Internet of Vehicles [1.1411682827396572]
  Internet of Vehicles (IoV) faces a dynamic, adversarial security environment where attackers adapt to defenses.<n>We formulate IoV security as a sequential attacker-defender interaction and model defense as a reinforcement learning problem under partial observability.<n>We propose Quantum Belief-Integrated Reinforcement Defense (Q-BIRD) using quantum-inspired belief representation to encode defender uncertainty about hidden attacker intent.
  arXiv  Detail & Related papers  (2026-06-05T19:20:38Z)
• AI Model Extraction Attacks: Bypassing Single-Client Assumptions in Defenses [1.247917923987062]
  Model Extraction Attacks (MEAs) enable adversaries to replicate proprietary models, compromise protected information, and prepare offline adversarial attacks.<n>Current defense strategies predominantly rely on the Single Client Assumption (SCA), which is the implicit assumption that attacks originate from isolated identities.<n>We introduce a modular, open-source framework called CerberusAI for reproducible model-stealing research, and use it to simulate distributed attack scenarios.
  arXiv  Detail & Related papers  (2026-06-02T09:25:29Z)
• From Stateless Queries to Autonomous Actions: A Layered Security Framework for Agentic AI Systems [0.0]
  Agentic AI systems plan across extended horizons, maintain persistent memory, invoke external tools, and coordinate with peer agents.<n>Existing security analyses organize threats by attack type (prompt injection, jailbreaking), but provide no principled model of which architectural component is vulnerable.<n>We introduce the Layered Attack Surface Model (LASM), a seven-layer framework that maps threats to distinct architectural components.
  arXiv  Detail & Related papers  (2026-04-25T14:57:15Z)
• Poisoning the Pixels: Revisiting Backdoor Attacks on Semantic Segmentation [28.208102468217394]
  This work systematically examines backdoor attacks tailored to semantic segmentation.<n>We introduce BADSEG, a unified framework that optimize trigger designs and apply label manipulation strategies to maximize attack performance.<n>Our work reveals previously overlooked security vulnerabilities in semantic segmentation, and motivates the development of defenses tailored to segmentation-specific threat models.
  arXiv  Detail & Related papers  (2026-03-17T11:42:17Z)
• ClawWorm: Self-Propagating Attacks Across LLM Agent Ecosystems [18.180880919825082]
  We present ClawWorm, the first self-replicating worm attack against a production-scale agent framework.<n>We demonstrate a 64.5% aggregate attack success rate, sustained multi-hop propagation, and reveal stark divergences in model security postures.
  arXiv  Detail & Related papers  (2026-03-16T17:55:43Z)
• Adversarial Attack-Defense Co-Evolution for LLM Safety Alignment via Tree-Group Dual-Aware Search and Optimization [51.12422886183246]
  Large Language Models (LLMs) have developed rapidly in web services, delivering unprecedented capabilities while amplifying societal risks.<n>Existing works tend to focus on either isolated jailbreak attacks or static defenses, neglecting the dynamic interplay between evolving threats and safeguards in real-world web contexts.<n>We propose ACE-Safety, a novel framework that jointly optimize attack and defense models by seamlessly integrating two key innovative procedures.
  arXiv  Detail & Related papers  (2025-11-24T15:23:41Z)
• AegisLLM: Scaling Agentic Systems for Self-Reflective Defense in LLM Security [74.22452069013289]
  AegisLLM is a cooperative multi-agent defense against adversarial attacks and information leakage.<n>We show that scaling agentic reasoning system at test-time substantially enhances robustness without compromising model utility.<n> Comprehensive evaluations across key threat scenarios, including unlearning and jailbreaking, demonstrate the effectiveness of AegisLLM.
  arXiv  Detail & Related papers  (2025-04-29T17:36:05Z)
• Fixed Points in Cyber Space: Rethinking Optimal Evasion Attacks in the Age of AI-NIDS [70.60975663021952]
  We study blackbox adversarial attacks on network classifiers. We argue that attacker-defender fixed points are themselves general-sum games with complex phase transitions. We show that a continual learning approach is required to study attacker-defender dynamics.
  arXiv  Detail & Related papers  (2021-11-23T23:42:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.