A Taxonomy of Runtime Faults in Model Context Protocol Servers

• URL: http://arxiv.org/abs/2606.05339v1
• Date: Wed, 03 Jun 2026 18:28:20 GMT
• Title: A Taxonomy of Runtime Faults in Model Context Protocol Servers
• Abstract summary: We present the first empirical taxonomy of runtime faults in MCP servers.<n>We analyzed 837 MCP-specific runtime fault threads from 473 actively maintained MCP server GitHub repositories.<n>To assess the taxonomy's external validity, we surveyed 55 MCP server developers.
• Score: 3.282178252052344
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: MCP (Model Context Protocol) enables LLMs (Large Language Models) to interact with external tools and data sources via a standardized protocol. Its rapid adoption in tool-augmented Artificial Intelligence (AI) workflows has introduced new reliability challenges, such as configuration parameters that are accepted but not enforced at runtime, leading to unintended default behavior, whose runtime fault characteristics remain empirically unexamined. We present the first empirical taxonomy of runtime faults in MCP servers. We manually analyzed 837 MCP-specific runtime fault threads from 473 actively maintained MCP server GitHub repositories and derived a taxonomy using a bottom-up open coding procedure. The taxonomy comprises 11 top-level categories and 27 subcategories (73 leaf fault types), covering recurrent failures across protocol interactions, tool invocations, schema enforcement, state management, model-provider integration, security validation, and timeouts or explicit cancellations of in-progress operations. To assess the taxonomy's external validity, we surveyed 55 MCP server developers. Respondents reported experiencing an average of 20 of the 27 fault subcategories, and no category remained unobserved. These results indicate that the taxonomy reflects widely observed runtime failures in MCP-based systems and shall assist AI software maintenance and evolution in the future.

Related papers

• MTAC-IFBench: Benchmarking Instruction-Following in Multi-Turn Agentic Coding [83.0805548642958]
  MTAC-IFBench is a comprehensive benchmark for instruction-following in software development.<n>It features multi-turn progressive software development instructions with diverse constraints spanning 6 primary and 18 secondary categories.<n>MTAC-IFBench identifies significant deficiencies in existing code agents in multi-turn instruction-following, with their performance degrading rapidly as the interaction session grows longer.
  arXiv  Detail & Related papers  (2026-09-14T03:57:37Z)
• Description-Code Inconsistency in Real-world MCP Servers: Measurement, Detection, and Security Implications [15.815506064408074]
  We present a comprehensive study of Description-Code Inconsistency (DCI) in real-world MCP servers.<n>We propose a comprehensive taxonomy spanning functionality inconsistencies and undeclared side effects.<n>We apply this taxonomy to a large-scale dataset comprising 19,200 description-code pairs extracted from 2,214 real-world MCP servers.
  arXiv  Detail & Related papers  (2026-06-03T11:51:32Z)
• Unsafe by Flow: Uncovering Bidirectional Data-Flow Risks in MCP Ecosystem [8.214897650566494]
  Model Context Protocol (MCP) have quickly become the interface layer between LLM agents and external tools.<n> Vulnerabilities manifest in two directions: requester-controlled arguments may propagate to sensitive operations, while untrusted external or sensitive internal data may surface.<n>We present MCP-BiFlow, a static analysis framework built around MCP-aware entrypoint recovery, protocol-specific taint modeling, and interprocedural propagation analysis.
  arXiv  Detail & Related papers  (2026-05-08T15:03:51Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• From Docs to Descriptions: Smell-Aware Evaluation of MCP Server Descriptions [15.082788201596422]
  We present the first systematic study of description smells in MCP tool descriptions and their impact on usability.<n>We find that description smells are pervasive, reflecting a "code-first, description-last" pattern.<n>In competitive settings with functionally equivalent servers, standard-compliant descriptions reach 72% selection probability.
  arXiv  Detail & Related papers  (2026-02-21T17:45:07Z)
• Don't believe everything you read: Understanding and Measuring MCP Behavior under Misleading Tool Descriptions [20.772501129444993]
  We study how mismatches between externally presented tool descriptions and underlying implementations shape the mental models and decision-making behavior of intelligent agents.<n>Our results show that while most servers are highly consistent, approximately 13% exhibit substantial mismatches that can enable undocumented privileged operations, hidden state mutations, or unauthorized financial actions.<n>Our findings demonstrate that description-code inconsistency is a concrete and prevalent attack surface in MCP-based AI agents, and motivate the need for systematic auditing and stronger transparency guarantees in future agent ecosystems.
  arXiv  Detail & Related papers  (2026-02-03T14:31:52Z)
• MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers [5.463884405989425]
  We introduce MCP-Atlas, a large-scale benchmark for evaluating tool-use competency.<n>It includes 1,000 tasks designed to assess tool-use competency in realistic, multi-step orchestrate.<n>We score tasks using a claims-based rubric that awards partial credit based on the factual claims satisfied in the model's final answer.
  arXiv  Detail & Related papers  (2026-01-31T23:19:39Z)
• MCP-Flow: Facilitating LLM Agents to Master Real-World, Diverse and Scaling MCP Tools [58.5971352939562]
  Large Language Models increasingly rely on external tools to perform complex, realistic tasks.<n>Existing MCP research covers few servers, depends on costly manual curation, and lacks training support.<n>We introduce MCP-Flow, an automated web-agent-driven pipeline for large-scale server discovery, data synthesis, and model training.
  arXiv  Detail & Related papers  (2025-10-28T10:42:17Z)
• MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Servers [86.00932417210477]
  We introduce MCP-Universe, the first comprehensive benchmark specifically designed to evaluate LLMs in realistic and hard tasks through interaction with real-world MCP servers.<n>Our benchmark encompasses 6 core domains spanning 11 different MCP servers: Location Navigation, Repository Management, Financial Analysis, 3D Design, Browser Automation, and Web Searching.<n>We find that even SOTA models such as GPT-5 (43.72%), Grok-4 (33.33%) and Claude-4.0-Sonnet (29.44%) exhibit significant performance limitations.
  arXiv  Detail & Related papers  (2025-08-20T13:28:58Z)
• AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarios [51.46347732659174]
  Large Language Models (LLMs) have demonstrated advanced capabilities in real-world agentic applications.<n>AgentIF is the first benchmark for systematically evaluating LLM instruction following ability in agentic scenarios.
  arXiv  Detail & Related papers  (2025-05-22T17:31:10Z)
• SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints [59.645885492637845]
  SOPBench is an evaluation pipeline that transforms each service-specific SOP code program into a directed graph of executable functions.<n>Our approach transforms each service-specific SOP code program into a directed graph of executable functions and requires agents to call these functions based on natural language SOP descriptions.<n>We evaluate 18 leading models, and results show the task is challenging even for top-tier models.
  arXiv  Detail & Related papers  (2025-03-11T17:53:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.