A Survey on AgentOps: Categorization, Challenges, and Future Directions

• URL: http://arxiv.org/abs/2508.02121v1
• Date: Mon, 04 Aug 2025 06:59:36 GMT
• Title: A Survey on AgentOps: Categorization, Challenges, and Future Directions
• Authors: Zexin Wang, Jingjing Li, Quan Zhou, Haotian Si, Yuanhao Liu, Jianhui Li, Gaogang Xie, Fei Sun, Dan Pei, Changhua Pei,
• Abstract summary: This paper introduces a novel and comprehensive operational framework for agent systems, dubbed Agent System Operations (AgentOps)<n>We provide detailed definitions and explanations of its four key stages: monitoring, anomaly detection, root cause analysis, and resolution.
• Score: 25.00082531560766
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As the reasoning capabilities of Large Language Models (LLMs) continue to advance, LLM-based agent systems offer advantages in flexibility and interpretability over traditional systems, garnering increasing attention. However, despite the widespread research interest and industrial application of agent systems, these systems, like their traditional counterparts, frequently encounter anomalies. These anomalies lead to instability and insecurity, hindering their further development. Therefore, a comprehensive and systematic approach to the operation and maintenance of agent systems is urgently needed. Unfortunately, current research on the operations of agent systems is sparse. To address this gap, we have undertaken a survey on agent system operations with the aim of establishing a clear framework for the field, defining the challenges, and facilitating further development. Specifically, this paper begins by systematically defining anomalies within agent systems, categorizing them into intra-agent anomalies and inter-agent anomalies. Next, we introduce a novel and comprehensive operational framework for agent systems, dubbed Agent System Operations (AgentOps). We provide detailed definitions and explanations of its four key stages: monitoring, anomaly detection, root cause analysis, and resolution.

Related papers

• SafeMobile: Chain-level Jailbreak Detection and Automated Evaluation for Multimodal Mobile Agents [58.21223208538351]
  This work explores the security issues surrounding mobile multimodal agents.<n>It attempts to construct a risk discrimination mechanism by incorporating behavioral sequence information.<n>It also designs an automated assisted assessment scheme based on a large language model.
  arXiv  Detail & Related papers  (2025-07-01T15:10:00Z)
• Deep Research Agents: A Systematic Examination And Roadmap [79.04813794804377]
  Deep Research (DR) agents are designed to tackle complex, multi-turn informational research tasks.<n>In this paper, we conduct a detailed analysis of the foundational technologies and architectural components that constitute DR agents.
  arXiv  Detail & Related papers  (2025-06-22T16:52:48Z)
• SentinelAgent: Graph-based Anomaly Detection in Multi-Agent Systems [11.497269773189254]
  We present a system-level anomaly detection framework tailored for large language model (LLM)-based multi-agent systems (MAS)<n>We propose a graph-based framework that models agent interactions as dynamic execution graphs, enabling semantic anomaly detection at node, edge, and path levels.<n>Second, we introduce a pluggable SentinelAgent, an LLM-powered oversight agent that observes, analyzes, and intervenes in MAS execution based on security policies and contextual reasoning.
  arXiv  Detail & Related papers  (2025-05-30T04:25:19Z)
• Inherent and emergent liability issues in LLM-based agentic systems: a principal-agent perspective [0.0]
  Agentic systems powered by large language models (LLMs) are becoming progressively more complex and capable.<n>Their increasing agency and expanding deployment settings attract growing attention to effective governance policies, monitoring, and control protocols.<n>We analyze potential liability issues arising from the delegated use of LLM agents and their extended systems through a principal-agent perspective.
  arXiv  Detail & Related papers  (2025-04-04T08:10:02Z)
• Multi-Agent Risks from Advanced AI [90.74347101431474]
  Multi-agent systems of advanced AI pose novel and under-explored risks.<n>We identify three key failure modes based on agents' incentives, as well as seven key risk factors.<n>We highlight several important instances of each risk, as well as promising directions to help mitigate them.
  arXiv  Detail & Related papers  (2025-02-19T23:03:21Z)
• Agent-as-a-Judge: Evaluate Agents with Agents [61.33974108405561]
  We introduce the Agent-as-a-Judge framework, wherein agentic systems are used to evaluate agentic systems. This is an organic extension of the LLM-as-a-Judge framework, incorporating agentic features that enable intermediate feedback for the entire task-solving process. We present DevAI, a new benchmark of 55 realistic automated AI development tasks.
  arXiv  Detail & Related papers  (2024-10-14T17:57:02Z)
• Preparing for Super-Reactivity: Early Fault-Detection in the Development of Exceedingly Complex Reactive Systems [1.6298172960110866]
  We introduce the term Super-Reactive Systems to refer to reactive systems whose construction and behavior are complex, constantly changing and evolving. Finding hidden faults in such systems early in planning and development is critical for human safety, the environment, society and the economy. We propose an architecture for models and tools to overcome barriers and enable simulation, systematic analysis, and fault detection and handling.
  arXiv  Detail & Related papers  (2024-10-03T16:08:30Z)
• EvoAgent: Towards Automatic Multi-Agent Generation via Evolutionary Algorithms [55.77492625524141]
  EvoAgent is a generic method to automatically extend specialized agents to multi-agent systems.<n>We show that EvoAgent can significantly enhance the task-solving capability of LLM-based agents.
  arXiv  Detail & Related papers  (2024-06-20T11:49:23Z)
• A Survey on Context-Aware Multi-Agent Systems: Techniques, Challenges and Future Directions [1.0488897291370285]
  Research interest in autonomous agents is on the rise as an emerging topic.<n>The challenge lies in enabling these agents to learn, reason, and navigate uncertainties in dynamic environments.<n>Context awareness emerges as a pivotal element in fortifying multi-agent systems.
  arXiv  Detail & Related papers  (2024-02-03T00:27:22Z)
• A Hierarchical Framework with Spatio-Temporal Consistency Learning for Emergence Detection in Complex Adaptive Systems [41.055298739292695]
  Emergence, a global property of complex adaptive systems, is prevalent in real-world dynamic systems, e.g., network-level traffic congestions. This paper proposes a hierarchical framework with CAS-temporal consistency to solve these two problems by learning the system representation and agent representations. Our method achieves more detection than traditional methods and deep learning methods on three datasets with well-known yet hard-to-detect emergent behaviors.
  arXiv  Detail & Related papers  (2024-01-18T08:55:05Z)
• Interactive System-wise Anomaly Detection [66.3766756452743]
  Anomaly detection plays a fundamental role in various applications. It is challenging for existing methods to handle the scenarios where the instances are systems whose characteristics are not readily observed as data. We develop an end-to-end approach which includes an encoder-decoder module that learns system embeddings.
  arXiv  Detail & Related papers  (2023-04-21T02:20:24Z)
• Harms from Increasingly Agentic Algorithmic Systems [21.613581713046464]
  Research in Fairness, Accountability, Transparency, and Ethics (FATE) has established many sources and forms of algorithmic harm. Despite ongoing harms, new systems are being developed and deployed which threaten the perpetuation of the same harms.
  arXiv  Detail & Related papers  (2023-02-20T21:42:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.