Agent Security is a Systems Problem

• URL: http://arxiv.org/abs/2605.18991v2
• Date: Wed, 20 May 2026 17:25:38 GMT
• Title: Agent Security is a Systems Problem
• Abstract summary: We take the position that agent security must be approached as a systems problem.<n>We articulate a set of core principles that provide a foundation for designing agentic systems with predictable guarantees.
• Score: 39.04908287507772
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We take the position that agent security must be approached as a systems problem: the AI model powering the agent must be treated as an untrusted component, and security invariants must be enforced at the system level. Through this lens, efforts to increase model robustness (the dominant viewpoint in the community) are insufficient on their own. Instead, we must complement existing efforts with techniques from the systems security domain. Based on our experience as cybersecurity researchers in operating systems, networks, formal methods, and adversarial machine learning, we articulate a set of core principles, grounded in decades of systems security research, that provide a foundation for designing agentic systems with predictable guarantees. As evidence, we analyze eleven representative real-world attacks on agents and discuss how systems principles, if realized, could have prevented these attacks. We also identify the research challenges that stand in the way of implementing these principles in agents.

Related papers

• Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security [57.35851886874902]
  Agentic AI systems execute complex tasks autonomously, but their multi-step trajectories introduce new failure modes that challenge trustworthiness.<n>This survey provides a focused examination of trustworthy agentic AI through two core dimensions that are critical for high-risk deployments.<n>For each dimension, we clarify key concepts, identify where risks emerge along the agent workflow, and summarize stage-targeted mitigation strategies.
  arXiv  Detail & Related papers  (2026-05-17T10:26:37Z)
• The Attack and Defense Landscape of Agentic AI: A Comprehensive Survey [46.278608570660005]
  AI agents that combine large language models with non-AI system components are rapidly emerging in real-world applications.<n>This paper presents the first systematic and comprehensive survey of AI agent security.
  arXiv  Detail & Related papers  (2026-03-11T05:25:55Z)
• Extending the Formalism and Theoretical Foundations of Cryptography to AI [18.724847875398435]
  Recent progress in (Large) Language Models has enabled the development of autonomous LM-based agents.<n>One emerging direction to mitigate security risks is to constrain agent behaviours via access control and permissioning mechanisms.<n>We first systematize the landscape by constructing an attack taxonomy tailored to language models.<n>We then develop a formal treatment of agentic access control by defining an AIOracle algorithmically and introducing a security-game framework.
  arXiv  Detail & Related papers  (2026-03-03T04:11:21Z)
• Systems Security Foundations for Agentic Computing [21.600611396864114]
  This paper examines end-to-end security properties of entire systems, rather than AI models in isolation.<n>A key goal is to examine where research challenges arise when applying traditional security principles in the context of AI agents.
  arXiv  Detail & Related papers  (2025-12-01T05:28:59Z)
• Perspectives on a Reliability Monitoring Framework for Agentic AI Systems [5.539407031861404]
  We derive the main reliability challenges of agentic AI systems during operation based on their characteristics.<n>We propose a two-layered reliability monitoring framework for agentic AI systems.
  arXiv  Detail & Related papers  (2025-11-12T10:19:17Z)
• OS-Sentinel: Towards Safety-Enhanced Mobile GUI Agents via Hybrid Validation in Realistic Workflows [77.95511352806261]
  Computer-using agents powered by Vision-Language Models (VLMs) have demonstrated human-like capabilities in operating digital environments like mobile platforms.<n>We propose OS-Sentinel, a novel hybrid safety detection framework that combines a Formal Verifier for detecting explicit system-level violations with a Contextual Judge for assessing contextual risks and agent actions.
  arXiv  Detail & Related papers  (2025-10-28T13:22:39Z)
• A Comprehensive Survey of Self-Evolving AI Agents: A New Paradigm Bridging Foundation Models and Lifelong Agentic Systems [53.37728204835912]
  Most existing AI systems rely on manually crafted configurations that remain static after deployment.<n>Recent research has explored agent evolution techniques that aim to automatically enhance agent systems based on interaction data and environmental feedback.<n>This survey aims to provide researchers and practitioners with a systematic understanding of self-evolving AI agents.
  arXiv  Detail & Related papers  (2025-08-10T16:07:32Z)
• LLM Agents Should Employ Security Principles [60.03651084139836]
  This paper argues that the well-established design principles in information security should be employed when deploying Large Language Model (LLM) agents at scale.<n>We introduce AgentSandbox, a conceptual framework embedding these security principles to provide safeguards throughout an agent's life-cycle.
  arXiv  Detail & Related papers  (2025-05-29T21:39:08Z)
• Safeguarding AI Agents: Developing and Analyzing Safety Architectures [0.0]
  This paper addresses the need for safety measures in AI systems that collaborate with human teams.<n>We propose and evaluate three frameworks to enhance safety protocols in AI agent systems.<n>We conclude that these frameworks can significantly strengthen the safety and security of AI agent systems.
  arXiv  Detail & Related papers  (2024-09-03T10:14:51Z)
• Dos and Don'ts of Machine Learning in Computer Security [74.1816306998445]
  Despite great potential, machine learning in security is prone to subtle pitfalls that undermine its performance. We identify common pitfalls in the design, implementation, and evaluation of learning-based security systems. We propose actionable recommendations to support researchers in avoiding or mitigating the pitfalls where possible.
  arXiv  Detail & Related papers  (2020-10-19T13:09:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.