When Agentic Executions Fail: Detecting and Localizing Runtime Faults from Telemetry
- URL: http://arxiv.org/abs/2608.14680v1
- Date: Tue, 04 Aug 2026 20:47:37 GMT
- Title: When Agentic Executions Fail: Detecting and Localizing Runtime Faults from Telemetry
- Abstract summary: AGENTCHAOSBENCH is a benchmark for detecting and localizing runtime faults in agentic systems from execution telemetry.<n>We run five applications that coordinate agents over the Agent-to-Agent protocol and call tools through the Model Context Protocol.<n>The resulting dataset contains 2759.2% traces: 250 faulty executions spanning ten fault types and 25 no-fault controls.
- Score: 12.471487731560643
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Reliability in LLM-based agentic systems is a property of the whole execution (its tool calls, model calls, guardrails, and inter-agent messages), not of the final answer alone, yet evaluating only task outcomes reveals little about how or why a run fails. We present AGENTCHAOSBENCH, a benchmark for detecting and localizing runtime faults in agentic systems from their execution telemetry. We run five heterogeneous applications that coordinate agents over the Agent-to-Agent protocol and call tools through the Model Context Protocol, and inject ten types of operational fault (unavailable or slow tools, corrupted or oversized responses, and delayed, looped, or misrouted delegations and bypassed guardrails) at their tool, model, guardrail, and inter-agent boundaries, alongside a no-fault control. The resulting dataset contains 275 sanitized traces: 250 faulty executions spanning ten fault types and 25 no-fault controls. Each faulty trace is aligned with the no-fault execution of the same input; fault-type labels and, where applicable, location labels are held out from diagnosis. On structured single-trace inputs, a first set of zero-shot LLM baselines shows the task is far from solved: local detectors up to 14B parameters reach only 13.6-19.2% top-1 fault-type accuracy and the frontier DeepSeek-v4-pro only 24.8%, while jointly identifying the fault type and its location tops out at 22%; reference-dependent faults (above all a bypassed guardrail) stay near-unsolved from a single trace. An aligned reference improves selected relative faults but does not resolve guardrail bypass. The held-out labels and compact prediction format support reproducible comparison of LLM-based and non-LLM diagnosis methods.
Related papers
- Calibration is the Bottleneck: An Action-Class Diagnostic of Multi-Turn Tool-Calling [65.03665015319457]
We propose an action-class-oriented diagnostic framework that decomposes multi-turn failures into two modes: action-class miscalibration and action-execution failure.<n>We validate it on a panel of tool-calling models across multiple multi-turn benchmarks.
arXiv Detail & Related papers (2026-09-01T09:08:15Z) - AgentChaos: Chaos Engineering for Agent Systems via Programmatic Fault Injection [16.28121266059156]
Evaluations across agent systems, benchmarks, and backbones under 65 fault configurations show that all systems degrade under fault injection.<n>We propose AgentChaos, a chaos engineering framework for controlled, runtime, non-intrusive fault injection.
arXiv Detail & Related papers (2026-08-07T04:24:29Z) - PhyAgentOS: A Self-Evolving Operating System for Embodied Agents with Decoupled Cognitive Planning and Physical Execution [49.776611937968]
We present PhyAgentOS, a foundation delivering scheduling, verification, memory, benchmarking, and safety as system-level services.<n>Its Session-Centered treats a session, not an action, as the minimum unit of scheduling, compatibility, supervised execution, evidence collection, and acceptance.<n>SessionVerifier distinguishes execution termination from semantic task completion via evidence-grounded verdicts of success, failure, or replan.<n> Benchmarking reuses the deployment session and verification path, so results trace to real execution.
arXiv Detail & Related papers (2026-07-18T04:46:53Z) - AutoTrace: From Patches to Triggers via Agentic Interprocedural Exploration [6.39392731596656]
Given a vulnerability-fixing commit, trigger localization asks which specific statement turns the vulnerable program state into a concrete unsafe operation.<n>We present AutoTrace, an agentic pipeline that localizes vulnerability triggers by exploring a code property graph layer by layer.<n>Agents never accept a trigger on their own authority; every reported trigger is backed by explicit evidence drawn from the graph.
arXiv Detail & Related papers (2026-07-13T18:21:42Z) - What Resolve Rate Hides: Trajectory Structure Diagnostics for Coding Agents [14.988334585821939]
Coding agents are ranked almost entirely by resolve rate.<n>A single pass/fail label says nothing about why a run failed or why an accepted run spent extra steps, time, or tokens.<n>We present TraceProbe, a trajectory-diagnostic framework that recovers what resolve rate hides.
arXiv Detail & Related papers (2026-07-07T12:09:46Z) - Early Diagnosis of Wasted Computation in Multi-Agent LLM Systems via Failure-Aware Observability [8.036549927091286]
This paper introduces a failure-aware observability framework for diagnosing wasted computation in multi-agent LLM traces.<n>We instantiate the framework in a three- agent question-answering system and evaluate it on 165 GAIA validation traces under identical execution caps.
arXiv Detail & Related papers (2026-05-31T17:50:11Z) - VerifyMAS: Hypothesis Verification for Failure Attribution in LLM Multi-Agent Systems [79.51005192758262]
Large language model-driven multi-agent systems excel at complex tasks.<n>However, unreliable agents remain a key bottleneck to system-level reliability.<n>We propose a hypothesis verification framework for agent failure attribution.
arXiv Detail & Related papers (2026-05-17T14:09:35Z) - Arbiter: Detecting Interference in LLM Agent System Prompts [0.0]
Arbiter is a framework combining formal evaluation rules with multi-model LLM scouring to detect interference patterns in system prompts.<n> Applied to three major coding agent system prompts: Claude Code (Anthropic), Codex CLI (OpenAI), and Gemini CLI (Google)
arXiv Detail & Related papers (2026-03-09T22:29:47Z) - DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems [48.971606069204825]
DoVer is an intervention-driven debug framework for large language model (LLM)-based multi-agent systems.<n>It augments hypothesis generation with active verification through targeted interventions.<n>DoVer flips 18-28% of failed trials into successes, achieves up to 16% milestone progress, and validates or refutes 30-60% of failure hypotheses.
arXiv Detail & Related papers (2025-12-07T09:23:48Z) - Where LLM Agents Fail and How They can Learn From Failures [62.196870049524364]
Large Language Model (LLM) agents have shown promise in solving complex, multi-step tasks.<n>They amplify vulnerability to cascading failures, where a single root-cause error propagates through subsequent decisions.<n>Current systems lack a framework that can comprehensively understand agent error in a modular and systemic way.<n>We introduce the AgentErrorTaxonomy, a modular classification of failure modes spanning memory, reflection, planning, action, and system-level operations.
arXiv Detail & Related papers (2025-09-29T18:20:27Z) - Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems [50.29939179830491]
Failure attribution in LLM multi-agent systems remains underexplored and labor-intensive.<n>We develop and evaluate three automated failure attribution methods, summarizing their corresponding pros and cons.<n>The best method achieves 53.5% accuracy in identifying failure-responsible agents but only 14.2% in pinpointing failure steps.
arXiv Detail & Related papers (2025-04-30T23:09:44Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.