On Safety Risks in Experience-Driven Self-Evolving Agents

• URL: http://arxiv.org/abs/2604.16968v1
• Date: Sat, 18 Apr 2026 11:32:36 GMT
• Title: On Safety Risks in Experience-Driven Self-Evolving Agents
• Abstract summary: We investigate how experience accumulation and utilization in self-evolving agents affect safety performance across web-based and embodied environments.<n>Our findings expose inherent limitations of current self-evolving agents and call for more principled strategies to ensure safe and reliable adaptation.
• Score: 78.82066774532676
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Experience-driven self-evolution has emerged as a promising paradigm for improving the autonomy of large language model agents, yet its reliance on self-curated experience introduces underexplored safety risks. In this study, we investigate how experience accumulation and utilization in self-evolving agents affect safety performance across web-based and embodied environments. Notably, experience gathered solely from benign tasks can still compromise safety in high-risk scenarios. Further analysis attributes this degradation to the execution-oriented nature of accumulated experience, which reinforces agents' tendency to act rather than refuse. In more realistic settings where agents encounter both benign and harmful tasks, refusal-related experience mitigates safety decline but induces over-refusal, revealing a fundamental safety-utility trade-off. Overall, our findings expose inherent limitations of current self-evolving agents and call for more principled strategies to ensure safe and reliable adaptation.

Related papers

• Self-Aware Active Learning Enables Continual Improvement in Autonomous Driving [12.669051115615176]
  Self-aware guided exploration (SAGE) is an active learning framework for post-training adaptation in autonomous driving.<n>Curiosity adaptively calibrates the intervention threshold for fear, allowing the agent to regulate risk in a context-dependent manner.<n>Fear is integrated into policy optimization and evaluation as a safety-oriented constraint to reduce performance regressions.
  arXiv  Detail & Related papers  (2026-08-30T13:01:32Z)
• Benign Alone, Harmful Together: Exploiting Experience Composition in Self-Evolving LLM Agents [25.747887285881415]
  EvoBreak is an experience-conditioned sequential attack that operates through individually benign attack-stage tasks and induced experiences.<n>EvoBreak consistently outperforms existing attacks while maintaining high benignness.<n>These results reveal benign experience composition as a persistent attack surface in self-evolving agents.
  arXiv  Detail & Related papers  (2026-08-03T06:28:21Z)
• Enhancing Agent Safety Judgment: Controlled Benchmark Rewriting and Analogical Reasoning for Deceptive Out-of-Distribution Scenarios [0.2578242050187029]
  ROME is a controlled benchmark-construction pipeline that rewrites known unsafe trajectories into deceptive evaluation instances.<n>ARISE is a retrieval-guided inference-time enhancement that retrieves ReAct-style analogical safety trajectories from an external analogical base and injects them as structured reasoning exemplars.
  arXiv  Detail & Related papers  (2026-05-05T00:21:00Z)
• Risky-Bench: Probing Agentic Safety Risks under Real-World Deployment [64.36422334429228]
  Large Language Models (LLMs) are increasingly deployed as agents that operate in real-world environments.<n>Existing agent safety evaluations rely on risk-oriented tasks tailored to specific agent settings.<n>We propose Risky-Bench, a framework that enables systematic agent safety evaluation grounded in real-world deployment.
  arXiv  Detail & Related papers  (2026-02-03T04:44:11Z)
• Self-Guard: Defending Large Reasoning Models via enhanced self-reflection [54.775612141528164]
  Self-Guard is a lightweight safety defense framework for Large Reasoning Models.<n>It bridges the awareness-compliance gap, achieving robust safety performance without compromising model utility.<n>Self-Guard exhibits strong generalization across diverse unseen risks and varying model scales.
  arXiv  Detail & Related papers  (2026-01-31T13:06:11Z)
• BeSafe-Bench: Unveiling Behavioral Safety Risks of Situated Agents in Functional Environments [22.32760987248309]
  We present BeSafe-Bench (BSB), a benchmark for exposing behavioral safety risks of situated agents in functional environments.<n>BSB covers four representative domains: Web, Mobile, Embodied VLM, and Embodied VLA.<n>Using functional environments, we construct a diverse instruction space by augmenting tasks with nine categories of safety-critical risks.
  arXiv  Detail & Related papers  (2026-01-30T03:41:57Z)
• Large Language Model Agents Are Not Always Faithful Self-Evolvers [84.08646612111092]
  Self-evolving large language model (LLM) agents continually improve by accumulating and reusing past experience.<n>We present the first systematic investigation of experience faithfulness, the causal dependence of an agent's decisions on the experience it is given.
  arXiv  Detail & Related papers  (2026-01-30T01:05:15Z)
• SafeEvalAgent: Toward Agentic and Self-Evolving Safety Evaluation of LLMs [37.82193156438782]
  This paper introduces a new paradigm of agentic safety evaluation, reframing evaluation as a continuous and self-evolving process.<n>We propose a novel multi-agent framework SafeEvalAgent, which autonomously ingests unstructured policy documents to generate and perpetually evolve a comprehensive safety benchmark.<n>Our experiments demonstrate the effectiveness of SafeEvalAgent, showing a consistent decline in model safety as the evaluation hardens.
  arXiv  Detail & Related papers  (2025-09-30T11:20:41Z)
• IS-Bench: Evaluating Interactive Safety of VLM-Driven Embodied Agents in Daily Household Tasks [30.535665641990114]
  We present IS-Bench, the first multi-modal benchmark designed for interactive safety.<n>It features 161 challenging scenarios with 388 unique safety risks instantiated in a high-fidelity simulator.<n>It facilitates a novel process-oriented evaluation that verifies whether risk mitigation actions are performed before/after specific risk-prone steps.
  arXiv  Detail & Related papers  (2025-06-19T15:34:46Z)
• SafeAgent: Safeguarding LLM Agents via an Automated Risk Simulator [77.86600052899156]
  Large Language Model (LLM)-based agents are increasingly deployed in real-world applications.<n>We propose AutoSafe, the first framework that systematically enhances agent safety through fully automated synthetic data generation.<n>We show that AutoSafe boosts safety scores by 45% on average and achieves a 28.91% improvement on real-world tasks.
  arXiv  Detail & Related papers  (2025-05-23T10:56:06Z)
• Information Retrieval Induced Safety Degradation in AI Agents [52.15553901577888]
  This study investigates how expanding retrieval access affects model reliability, bias propagation, and harmful content generation.<n>Retrieval-enabled agents built on aligned LLMs often behave more unsafely than uncensored models without retrieval.<n>These findings underscore the need for robust mitigation strategies to ensure fairness and reliability in retrieval-enabled and increasingly autonomous AI systems.
  arXiv  Detail & Related papers  (2025-05-20T11:21:40Z)
• AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detection [47.83354878065321]
  We propose AGrail, a lifelong guardrail to enhance agent safety.<n>AGrail features adaptive safety check generation, effective safety check optimization, and tool compatibility and flexibility.
  arXiv  Detail & Related papers  (2025-02-17T05:12:33Z)
• Agent-SafetyBench: Evaluating the Safety of LLM Agents [72.92604341646691]
  We introduce Agent-SafetyBench, a benchmark designed to evaluate the safety of large language models (LLMs)<n>Agent-SafetyBench encompasses 349 interaction environments and 2,000 test cases, evaluating 8 categories of safety risks and covering 10 common failure modes frequently encountered in unsafe interactions.<n>Our evaluation of 16 popular LLM agents reveals a concerning result: none of the agents achieves a safety score above 60%.
  arXiv  Detail & Related papers  (2024-12-19T02:35:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.