Monitoring Agentic Systems Before They're Reliable

• URL: http://arxiv.org/abs/2606.02494v1
• Date: Mon, 01 Jun 2026 17:01:53 GMT
• Title: Monitoring Agentic Systems Before They're Reliable
• Abstract summary: Agentic systems entering production typically operate as partially integrated assemblies where structural defects, not task-level errors, dominate the failure landscape.<n>We present a monitoring and triage methodology that decomposes agentic system evaluation into three dimensions (quality, suitability, efficiency) at three monitoring scopes.<n>Findings are routed through severity classification adapted from FMEA, concentrating human attention on the subset that warrants investigation.
• Score: 0.7422381022274628
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agentic systems entering production typically operate as partially integrated assemblies where structural defects, not task-level errors, dominate the failure landscape. At this maturity level, task-level error detection may be infeasible: structural failure modes mask the signal that task-level monitors are designed to detect.We present a monitoring and triage methodology that decomposes agentic system evaluation into three dimensions (quality, suitability, efficiency) at three monitoring scopes (within-run, cross-run, structural), using variance as a characterization signal. Findings are routed through severity classification adapted from FMEA, concentrating human attention on the subset that warrants investigation. We evaluate on a synthetic testbed of 220 runs across 120 document bundles with controlled error injection.Three results emerge. Monitor scope determines failure type: within-run monitors surface deterministic stage defects (CV = 0.02), cross-run monitors surface stochastic integration consequences (CV = 1.25, 24% at L2), and a structural monitor identifies an integration gap with perfect consistency (CV = 0.00). Injected task-level errors are indistinguishable from clean baselines, confirming structural defects mask task-level signal. Deterministic triage routes 97% of findings to automated tracking, leaving the 2% reflecting variable behavior for human investigation.We propose, on Stage 1 evidence, a maturity-staging model in which monitoring transitions from structural characterization to error detection to reliability tracking as integration defects resolve. The taxonomy, CV-based scope characterization, and severity model transfer architecturally to document-driven, multi-stage agentic workflows in regulated industries; specific calibrations are domain-specific. Deploy monitoring early: the first thing it finds is the most important thing to fix.

Related papers

• Can Large Language Models Generate Observability-Aware Code? [51.30941152293301]
  We present a systematic study of observability in agent-generated systems.<n>Our results reveal a consistent gap between diagnostic semantics at the source level and fault signals at runtime.<n>We introduce an observability-oriented skill, which can serve as a guidance to improve both diagnostic semantics and fault-signal exposure.
  arXiv  Detail & Related papers  (2026-07-07T03:20:52Z)
• FoodMonitor: Benchmarking MLLMs for Explainable Compliance Analysis [29.80090524705393]
  FoodMonitor is a benchmark for explainable compliance analysis in commercial kitchen surveillance.<n>The best-performing model achieves only 0.360 $C_textscore$, with spatial localization and fine-grained rule understanding emerging as the primary bottlenecks.
  arXiv  Detail & Related papers  (2026-05-23T10:19:41Z)
• Trace-Level Analysis of Information Contamination in Multi-Agent Systems [8.829951903444352]
  Reasoning over heterogeneous artifacts increasingly occurs within structured agent.<n>We study this phenomenon by treating uncertainty as a controlled variable.<n>We characterize three manifestation types: silent semantic corruption, behavioral detours with recovery, and combined structural disruption and control-flow signatures.
  arXiv  Detail & Related papers  (2026-04-30T08:39:42Z)
• Automated Self-Testing as a Quality Gate: Evidence-Driven Release Management for LLM Applications [51.56484100374058]
  We present an automated self-testing framework that introduces quality gates with evidence-based release decisions.<n>We evaluate the framework through a longitudinal case study of an internally deployed multi-agent conversational AI system.
  arXiv  Detail & Related papers  (2026-03-13T20:44:15Z)
• World Model Failure Classification and Anomaly Detection for Autonomous Inspection [23.48742973289626]
  We propose a hybrid framework that combines supervised failure classification with anomaly detection.<n>Our approach uses a world model backbone with compressed video inputs.<n>Experiments show over 90% accuracy in distinguishing between successes, failures, and OOD cases.
  arXiv  Detail & Related papers  (2026-02-18T04:41:14Z)
• VSCOUT: A Hybrid Variational Autoencoder Approach to Outlier Detection in High-Dimensional Retrospective Monitoring [0.0]
  VSCOUT is a distribution-free framework designed for retrospective (Phase I) monitoring in high-dimensional settings.<n>VSCOUT achieves superior sensitivity to special-cause structure while maintaining controlled false alarms.<n>Its scalability, distributional flexibility, and resilience position VSCOUT as a practical and effective method for retrospective modeling and anomaly detection in AI-enabled environments.
  arXiv  Detail & Related papers  (2026-01-28T18:30:48Z)
• Noise & pattern: identity-anchored Tikhonov regularization for robust structural anomaly detection [58.535473924035365]
  Anomaly detection plays a pivotal role in automated industrial inspection, aiming to identify subtle or rare defects in otherwise uniform visual patterns.<n>We tackle structural anomaly detection using a self-supervised autoencoder that learns to repair corrupted inputs.<n>We introduce a corruption model that injects artificial disruptions into training images to mimic structural defects.
  arXiv  Detail & Related papers  (2025-11-10T15:48:50Z)
• Metacognitive Self-Correction for Multi-Agent System via Prototype-Guided Next-Execution Reconstruction [58.51530390018909]
  Large Language Model based multi-agent systems excel at collaborative problem solving but remain brittle to cascading errors.<n>We present MASC, a metacognitive framework that endows MAS with real-time, unsupervised, step-level error detection and self-correction.
  arXiv  Detail & Related papers  (2025-10-16T05:35:37Z)
• Dual-Mode Deep Anomaly Detection for Medical Manufacturing: Structural Similarity and Feature Distance [2.467339701756281]
  We propose two attention-guided autoencoder architectures for deep anomaly detection.<n>The first employs a structural similarity-based scoring approach that enables lightweight, real-time defect detection.<n>The second applies a feature distance-based strategy using Mahalanobis scoring on reduced latent features.
  arXiv  Detail & Related papers  (2025-09-06T18:17:40Z)
• Backdoor Cleaning without External Guidance in MLLM Fine-tuning [76.82121084745785]
  Believe Your Eyes (BYE) is a data filtering framework that leverages attention entropy patterns as self-supervised signals to identify and filter backdoor samples.<n>It achieves near-zero attack success rates while maintaining clean-task performance.
  arXiv  Detail & Related papers  (2025-05-22T17:11:58Z)
• GeneralAD: Anomaly Detection Across Domains by Attending to Distorted Features [68.14842693208465]
  GeneralAD is an anomaly detection framework designed to operate in semantic, near-distribution, and industrial settings. We propose a novel self-supervised anomaly generation module that employs straightforward operations like noise addition and shuffling to patch features. We extensively evaluated our approach on ten datasets, achieving state-of-the-art results in six and on-par performance in the remaining.
  arXiv  Detail & Related papers  (2024-07-17T09:27:41Z)
• A Comparison of Residual-based Methods on Fault Detection [6.675805308519987]
  In this study, we compare two residual-based approaches to detect faults in industrial systems. The performance evaluation focuses on three tasks: health indicator construction, fault detection, and health indicator interpretation. The detection results reveal that both models are capable of detecting faults with an average delay of around 20 cycles and maintain a low false positive rate.
  arXiv  Detail & Related papers  (2023-09-05T14:39:27Z)
• Self-Supervised Masked Convolutional Transformer Block for Anomaly Detection [122.4894940892536]
  We present a novel self-supervised masked convolutional transformer block (SSMCTB) that comprises the reconstruction-based functionality at a core architectural level. In this work, we extend our previous self-supervised predictive convolutional attentive block (SSPCAB) with a 3D masked convolutional layer, a transformer for channel-wise attention, as well as a novel self-supervised objective based on Huber loss.
  arXiv  Detail & Related papers  (2022-09-25T04:56:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.