Real-Time Detection and Repair of LLM Agent Failures

• URL: http://arxiv.org/abs/2608.02464v1
• Date: Mon, 03 Aug 2026 16:34:46 GMT
• Title: Real-Time Detection and Repair of LLM Agent Failures
• Abstract summary: LLM agents fail mid-episode loop, cascade tool errors, drift off goal, fabricate results, or silently absorb corrupted content.<n>We ask how much detection is achievable from observable step alone, using monitors costing microseconds per step and trained only on healthy runs.<n>System runs at 200 microseconds per step, three orders of magnitude below a judge call.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM agents fail mid-episode -- they loop, cascade tool errors, drift off goal, fabricate results, or silently absorb corrupted content -- and the standard remedy, judging every step with a second LLM, costs more than the agent itself. We ask how much detection is achievable from observable step telemetry alone, using monitors costing microseconds per step and trained only on healthy runs. On 2,823 committed agent episodes across three frameworks, three local models (qwen2.5 7b/3b, llama3.1 8b) and a commercial API (gemini-2.5-flash), a one-class echo-state-network ensemble with CUSUM alarms detects 0.71 of failures at a 5% false-alarm budget (AUROC 0.872). Its advantage over a memoryless baseline is a monotone function of post-onset horizon (+0.09 at <=3 steps, +0.40 at >=9), predicting its own failure region out of sample on AFTraj-2K. Ranking transfers with no retraining to two corpora from other groups (AFTraj-2K 0.745, ATBench 0.779). Monitors carry two burdens: a per-deployment healthy null (they do not transfer -- AUROC 0.527 cold against 0.885 recalibrated) and a residual false-alarm rate. We add a layer carrying neither: deterministic verification, which recomputes a run's stated total from the tool results it actually received and confirms every required call was made. Head-to-head it catches 60% of failures (96% with the coverage check) at 0 of 63 false positives against the monitor's 54% at 17%, transfers unchanged to llama3.1:8b (110 of 110 at 0 of 10), and trips on 0 of 1825 healthy episodes. Detection is then closed into repair: each flagged run is rolled back and re-run live, recovering 45% of failures against a 16% resampling control (p=0.0005) and lifting task success from 52% to 73% for about one extra model call per run. The system runs at ~200 microseconds per step, three orders of magnitude below a judge call. Code, traces and results are released.

Related papers

• ReDraft, Don't Just Distill: Reference-Driven Revision for Continual VLLM Post-Training [66.72708893922605]
  Post-training of large multimodal models should add new capabilities while preserving those from pre-training.<n>SFT gives explicit target supervision that learns a task from near-zero accuracy, but its off-policy targets move the model far enough to cause forgetting.<n>We introduce ReDraft, which obtains both from the model's own failures.
  arXiv  Detail & Related papers  (2026-09-15T04:59:03Z)
• From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix [55.08143827481755]
  We consolidate traffic from over 200 internal applications onto a single model.<n>We train a separate GRPO expert per axis and merge them via two-stage SLERP.<n>The model absorbs 50% of platform traffic, 116M requests per month, at a fraction of the serving cost.
  arXiv  Detail & Related papers  (2026-09-01T17:39:26Z)
• A Calibration Audit of Confidence in Feed-Forward 3D Reconstruction [45.496856835431885]
  3D reconstruction models emit per-pixel confidence that downstream systems read as a reliability signal.<n>We score the confidence on four properties, how well it ranks error, whether its level is right on average, whether it holds across the confidence range, and whether its intervals cover the truth.<n>The confidence ranks error well, but the predicted uncertainty is too low when it is read under conditions that are not exactly those of training.
  arXiv  Detail & Related papers  (2026-08-30T10:21:34Z)
• CURA: Certified Runtime Alarms for Computer-Use Agents [6.896160570537455]
  Self-report is the cheapest oversight channel a deployer has, and on capable computer-use agents it fails precisely where oversight matters.<n>On 361 OSWorld tasks our pipeline, a read-only feasibility gate, a planner, and an executor, reaches a mean task score of 82.9, above the 72.4 human reference.<n>64 of its 71 failures (90%) end with a success claim, acknowledging no explicit failure affordance is never used in roughly 9,100 calls.<n>Alarm-gated mid-execution oversight recovers 23 of 70 failures while spending a frontier overseer on 38, giving a deployable
  arXiv  Detail & Related papers  (2026-08-28T00:50:47Z)
• Invocation-Level Reliability of Tool-Using Agents [0.7874708385247353]
  We measure a correct-invocation rate that separates the two, under both a clean teacher-forced context and the model's own free-running context.<n>By depth 6, roughly 70% of a model's own clean-context capability is lost to its own earlier mistakes.
  arXiv  Detail & Related papers  (2026-08-23T09:04:59Z)
• Located but Not Releasable: Silent Gate Inversion and Bounded Linear Release [0.0]
  We submit the complete pipeline -- detect, localize, and release -- to a fully preregistered stress test on a 25.7M transformer.<n> Interventions at observation-evidence channels of mid layers restore target behavior on otherwise-suppressed worlds.<n>Gating fails out of distribution: a detector calibrated to trigger on zero out-of-distribution calibration worlds triggers on 6.9-7.3% of held-out in-distribution generations and on zero of the 2,400 held-out generations that actually need it.
  arXiv  Detail & Related papers  (2026-08-12T09:04:54Z)
• AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agents [83.37969790806069]
  LLM agent failures are difficult to debug because the step where an error surfaces is often not the one that caused it.<n>We present Agent DebuggerX, an open-source framework that organizes debug as a closed loop of Detect, Attribute, Recover, and Rerun.<n>At its core, Deep Debugger performs multi-turn root-cause diagnosis through global trajectory understanding.
  arXiv  Detail & Related papers  (2026-07-21T06:21:13Z)
• Zero2Skill: Bootstrapping Robot Skills through Autonomous Data Collection, Training, and Deployment [53.913175773174636]
  Existing pipelines reduce human effort via self-resetting, VLM verification, or language-guided correction.<n>We present Zero2Skill, a human-robot symbiotic agentic system in which corrections are retained and reused across rounds.
  arXiv  Detail & Related papers  (2026-07-15T17:16:24Z)
• Doomed from the Start: Early Abort of LLM Agent Episodes via a Recall-Controlled Probe Cascade [14.63102636181275]
  Large language model (LLM) agents often waste inference compute by continuing multi-step trajectories that are already doomed to fail.<n>We study early failure prediction and inference-time early stopping for LLM agents using hidden-state probes.<n>We turn this signal into a recall-controlled abort cascade for reducing LLM agent inference costs.
  arXiv  Detail & Related papers  (2026-07-07T17:03:55Z)
• Catching One in Five: LLM-as-Judge Blind Spots in Production Multi-Turn Transaction Agents [45.148328075418156]
  We study a deployed multi-turn food-and-beverage ordering agent and measure how many genuine quality problems it catches.<n>Our blind-spot taxonomy shows the failure is structured, not random.<n>For production multi-turn agents, automated judging is a regression floor, not a substitute for human review.
  arXiv  Detail & Related papers  (2026-06-09T02:11:01Z)
• From Confident Closing to Silent Failure: Characterizing False Success in LLM Agents [0.0]
  False success is common but varies by setting: 45--48% of failures in single-control tau2-bench domains, 3% in dual-control telecom, and 75.8% among AppWorld self-assessing coding-agent trajectories with explicit status claims.<n> TF-IDF detectors achieve task-disjoint AUROC 0.83 on tau2-bench and 0.95 on AppWorld, recovering 4--8x more false successes than the best judge at the same flag rate with 3,300x lower latency.
  arXiv  Detail & Related papers  (2026-06-01T00:13:23Z)
• Sequential Consensus for Multi-Agent LLM Debates: A Wald-SPRT compute governor with calibration-based failure detection [0.0]
  Multi-agent debate improves factuality and reasoning, but most recipes pick a fixed round count.<n>We adapt Wald's Sequential Probability Ratio Test (SPRT) as a plug-in compute governor for LLM debates.<n>On GSM8K the rule stops in 1.01 average rounds (4.06 LLM calls) at 97.0% accuracy vs 99.0% for fixed-5 debate at 15 calls: a 3.7x call reduction at -2pp accuracy.<n>On MMLU the calibrated KL collapses to about 0 and the rule caps on 99.5% of items at 2.1x cost.
  arXiv  Detail & Related papers  (2026-05-18T23:43:12Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• Evaluating Tool-Using Language Agents: Judge Reliability, Propagation Cascades, and Runtime Mitigation in AgentProp-Bench [0.0]
  We introduce AgentProp-Bench, a 2,000-task benchmark with 2,300 traces across four domains.<n>We quantify judge reliability, characterize error propagation, and evaluate a runtime mitigation.<n>All code, data, traces, and human labels are released at https://github.com/bhaskargurram-ai/agenthallu-bench.
  arXiv  Detail & Related papers  (2026-04-17T21:15:35Z)
• Impact of Labeling Inaccuracy and Image Noise on Tooth Segmentation in Panoramic Radiographs using Federated, Centralized and Local Learning [46.232038247686745]
  Federated learning (FL) may mitigate privacy constraints, heterogeneous data quality, and inconsistent labeling in dental diagnostic AI.<n>We compared FL with centralized (CL) and local learning (LL) for tooth segmentation in panoramic radiographs across multiple data corruption scenarios.
  arXiv  Detail & Related papers  (2025-09-08T11:07:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.