AgentCheck: A Reproduce-Intervene-Mitigate Workbench for LLM Agents over MCP

• URL: http://arxiv.org/abs/2607.11098v3
• Date: Wed, 15 Jul 2026 06:22:24 GMT
• Title: AgentCheck: A Reproduce-Intervene-Mitigate Workbench for LLM Agents over MCP
• Abstract summary: AgentCheck runs an agent against its real tools and records every tool response, then re-runs the agent with the response perturbed by a fault (12 types) injector.<n> Matching tool calls are replayed from cache, and later tool calls go live after the agent diverges.<n>This yields a reproduce-intervene-confirm loop: the developer toggles a mitigation, re-runs against the identical fault, and sees if the failure goes away.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Tool-using LLM agents are mostly evaluated assuming all tools work. When a tool times out, returns a week-stale value, or has its description poisoned in deployment, the developer needs a controlled way to reproduce the failure, test a fix, and confirm the fix worked before deployment. We present AgentCheck, an open-source web workbench that turns an MCP server into an intervention surface. AgentCheck runs an agent against its real tools and records every tool response, then re-runs the agent with the response perturbed by a fault (12 types) injector. Matching tool calls are replayed from cache, and later tool calls go live after the agent diverges. This yields a reproduce-intervene-confirm loop: the developer toggles a mitigation, re-runs against the identical fault, and sees if the failure goes away. Scoring has two parts: deterministic pass/fail rules, plus an LLM judge for interpretive labels, validated against human annotations. Across five agents, the best passes 105/120 scenarios and the weakest only 77. The failures are usually silent, confident use of incorrect tool outputs rather than crashes. On the weakest agent, a retry mitigation raises success on timeout error faults from as few as 30% of cases to 100%, whereas stale-data faults remain near 3-4 of 10 regardless of the mitigation. AgentCheck makes these failure modes reproducible, comparable, and verifiable before deployment.

Related papers

• Agents Trust Tools Too Much: Measuring Reliance on Unreliable Tools [16.535602696436424]
  We evaluate how agents respond to unreliable tool returns using three tools-web search, LLM sub-agent delegation, and code execution.<n>For each tool, we corrupt its returns and measure whether agents adopt the corrupted content in their final answers.<n>Agents exhibit high levels of overtrust across all three settings.
  arXiv  Detail & Related papers  (2026-09-04T17:02:32Z)
• AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agents [83.37969790806069]
  LLM agent failures are difficult to debug because the step where an error surfaces is often not the one that caused it.<n>We present Agent DebuggerX, an open-source framework that organizes debug as a closed loop of Detect, Attribute, Recover, and Rerun.<n>At its core, Deep Debugger performs multi-turn root-cause diagnosis through global trajectory understanding.
  arXiv  Detail & Related papers  (2026-07-21T06:21:13Z)
• AgentTether: Graph-Guided Diagnosis and Runtime Intervention for Reliable LLM Agent Operation [27.0859707906163]
  AgentTether is a run-time repair framework that automates post-run diagnosis and guided recovery without modifying the underlying agent or environment.<n>It localizes failure-critical subtrajectories by combining an offline normal-behavior model with a run-local graph detector.<n>It can be deployed as an offline diagnostic-and-guidance tool or as an online repair layer.
  arXiv  Detail & Related papers  (2026-07-07T13:40:31Z)
• AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation [0.18567307896771823]
  AgentLens is a production-assessed benchmark for interactive code agents.<n>It pairs formal verification with LLM-written trajectory reviews and side-by-side comparisons.<n>We use it to diagnose model behavior, compare successive versions of our own agent, and catch product regressions in a nightly evaluation pipeline.
  arXiv  Detail & Related papers  (2026-07-07T11:27:43Z)
• When Agents Do Not Stop: Uncovering Infinite Agentic Loops in LLM Agents [7.986500985812644]
  We call this problem Infinite Agentic Loops (IALs)<n>IALs arise from the interaction between agent logic, framework semantics, runtime observations, and termination mechanisms.<n>We propose IAL-Scan, a static analysis tool for detecting IAL failures in real-world LLM agent projects.
  arXiv  Detail & Related papers  (2026-07-02T03:14:27Z)
• Capable but Careless: Do Computer-Use Agents Follow Contextual Integrity? [54.701199583505144]
  Cross-application access creates a privacy risk that has been largely overlooked.<n>We introduce AgentCIBench, an evaluation harness that turns this risk into executable, deterministically scored scenarios.<n>We evaluate 15 frontier agents and find a surprisingly high failure rate.
  arXiv  Detail & Related papers  (2026-06-22T11:36:58Z)
• LACUNA: Safe Agents as Recursive Program Holes [3.2613419151327343]
  LLM agents increasingly act by writing code, yet a split persists between the runtime that drives the agent and the code the model writes.<n>We present LACUNA, a programming model for agents that closes this split while preserving safety.<n>Our primitive expresses ReAct loops, sub-agents, skills, parallel decomposition, and multi-model planning as ordinary control flow.
  arXiv  Detail & Related papers  (2026-05-27T15:27:25Z)
• DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems [48.971606069204825]
  DoVer is an intervention-driven debug framework for large language model (LLM)-based multi-agent systems.<n>It augments hypothesis generation with active verification through targeted interventions.<n>DoVer flips 18-28% of failed trials into successes, achieves up to 16% milestone progress, and validates or refutes 30-60% of failure hypotheses.
  arXiv  Detail & Related papers  (2025-12-07T09:23:48Z)
• Where LLM Agents Fail and How They can Learn From Failures [62.196870049524364]
  Large Language Model (LLM) agents have shown promise in solving complex, multi-step tasks.<n>They amplify vulnerability to cascading failures, where a single root-cause error propagates through subsequent decisions.<n>Current systems lack a framework that can comprehensively understand agent error in a modular and systemic way.<n>We introduce the AgentErrorTaxonomy, a modular classification of failure modes spanning memory, reflection, planning, action, and system-level operations.
  arXiv  Detail & Related papers  (2025-09-29T18:20:27Z)
• Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems [50.29939179830491]
  Failure attribution in LLM multi-agent systems remains underexplored and labor-intensive.<n>We develop and evaluate three automated failure attribution methods, summarizing their corresponding pros and cons.<n>The best method achieves 53.5% accuracy in identifying failure-responsible agents but only 14.2% in pinpointing failure steps.
  arXiv  Detail & Related papers  (2025-04-30T23:09:44Z)
• MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation [96.71370747681078]
  We introduce MLAgentBench, a suite of 13 tasks ranging from improving model performance on CIFAR-10 to recent research problems like BabyLM. For each task, an agent can perform actions like reading/writing files, executing code, and inspecting outputs. We benchmark agents based on Claude v1.0, Claude v2.1, Claude v3 Opus, GPT-4, GPT-4-turbo, Gemini-Pro, and Mixtral and find that a Claude v3 Opus agent is the best in terms of success rate.
  arXiv  Detail & Related papers  (2023-10-05T04:06:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.