Don't believe everything you read: Understanding and Measuring MCP Behavior under Misleading Tool Descriptions

• URL: http://arxiv.org/abs/2602.03580v1
• Date: Tue, 03 Feb 2026 14:31:52 GMT
• Title: Don't believe everything you read: Understanding and Measuring MCP Behavior under Misleading Tool Descriptions
• Abstract summary: We study how mismatches between externally presented tool descriptions and underlying implementations shape the mental models and decision-making behavior of intelligent agents.<n>Our results show that while most servers are highly consistent, approximately 13% exhibit substantial mismatches that can enable undocumented privileged operations, hidden state mutations, or unauthorized financial actions.<n>Our findings demonstrate that description-code inconsistency is a concrete and prevalent attack surface in MCP-based AI agents, and motivate the need for systematic auditing and stronger transparency guarantees in future agent ecosystems.
• Score: 20.772501129444993
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The Model Context Protocol (MCP) enables large language models to invoke external tools through natural-language descriptions, forming the foundation of many AI agent applications. However, MCP does not enforce consistency between documented tool behavior and actual code execution, even though MCP Servers often run with broad system privileges. This gap introduces a largely unexplored security risk. We study how mismatches between externally presented tool descriptions and underlying implementations systematically shape the mental models and decision-making behavior of intelligent agents. Specifically, we present the first large-scale study of description-code inconsistency in the MCP ecosystem. We design an automated static analysis framework and apply it to 10,240 real-world MCP Servers across 36 categories. Our results show that while most servers are highly consistent, approximately 13% exhibit substantial mismatches that can enable undocumented privileged operations, hidden state mutations, or unauthorized financial actions. We further observe systematic differences across application categories, popularity levels, and MCP marketplaces. Our findings demonstrate that description-code inconsistency is a concrete and prevalent attack surface in MCP-based AI agents, and motivate the need for systematic auditing and stronger transparency guarantees in future agent ecosystems.

Related papers

• A Taxonomy of Runtime Faults in Model Context Protocol Servers [3.282178252052344]
  We present the first empirical taxonomy of runtime faults in MCP servers.<n>We analyzed 837 MCP-specific runtime fault threads from 473 actively maintained MCP server GitHub repositories.<n>To assess the taxonomy's external validity, we surveyed 55 MCP server developers.
  arXiv  Detail & Related papers  (2026-06-03T18:28:20Z)
• Description-Code Inconsistency in Real-world MCP Servers: Measurement, Detection, and Security Implications [15.815506064408074]
  We present a comprehensive study of Description-Code Inconsistency (DCI) in real-world MCP servers.<n>We propose a comprehensive taxonomy spanning functionality inconsistencies and undeclared side effects.<n>We apply this taxonomy to a large-scale dataset comprising 19,200 description-code pairs extracted from 2,214 real-world MCP servers.
  arXiv  Detail & Related papers  (2026-06-03T11:51:32Z)
• AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions [78.49000936275773]
  We introduce AgentHijack, a benchmark designed to evaluate the robustness of computer-use agents under common corruptions.<n>We evaluate a variety of desktop tasks that utilize MLLM-based agents and discover that even minor instances of corruption can result in substantial performance degradation.<n>We propose AgentHijack-Agent, a framework that integrates an action generator with enhanced grounding capabilities and an onlooker responsible for behavior summarization and environment checking.
  arXiv  Detail & Related papers  (2026-05-25T11:09:22Z)
• From Docs to Descriptions: Smell-Aware Evaluation of MCP Server Descriptions [15.082788201596422]
  We present the first systematic study of description smells in MCP tool descriptions and their impact on usability.<n>We find that description smells are pervasive, reflecting a "code-first, description-last" pattern.<n>In competitive settings with functionally equivalent servers, standard-compliant descriptions reach 72% selection probability.
  arXiv  Detail & Related papers  (2026-02-21T17:45:07Z)
• From Tool Orchestration to Code Execution: A Study of MCP Design Choices [20.817336331051752]
  Model Context Protocols (MCPs) provide a unified platform for agent systems to discover, select, and orchestrate tools across heterogeneous execution environments.<n>Recent MCP designs have incorporated code execution as a first-class capability, an approach called Code Execution MCP (CEMCP)<n>This work formalizes the architectural distinction between context-coupled (traditional) and context-decoupled (CEMCP) models, analyzing their fundamental scalability trade-offs.
  arXiv  Detail & Related papers  (2026-02-17T19:03:08Z)
• MCPShield: A Security Cognition Layer for Adaptive Trust Calibration in Model Context Protocol Agents [39.267334469481916]
  We propose MCPShield as a plug-in security cognition layer that ensures agent security when invoking MCP-based tools.<n>Our work provides a practical and robust security safeguard for MCP-based tool invocation in open agent ecosystems.
  arXiv  Detail & Related papers  (2026-02-15T19:10:00Z)
• The Why Behind the Action: Unveiling Internal Drivers via Agentic Attribution [63.61358761489141]
  Large Language Model (LLM)-based agents are widely used in real-world applications such as customer service, web navigation, and software engineering.<n>We propose a novel framework for textbfgeneral agentic attribution, designed to identify the internal factors driving agent actions regardless of the task outcome.<n>We validate our framework across a diverse suite of agentic scenarios, including standard tool use and subtle reliability risks like memory-induced bias.
  arXiv  Detail & Related papers  (2026-01-21T15:22:21Z)
• MCPAgentBench: A Real-world Task Benchmark for Evaluating LLM Agent MCP Tool Use [12.220519951554133]
  MCPAgentBench is a benchmark based on real-world MCP definitions to evaluate the tool-use capabilities of agents.<n>The evaluation employs a dynamic sandbox environment that presents agents with candidate tool lists containing distractors.<n>Experiments conducted on various latest mainstream Large Language Models reveal significant performance differences in handling complex, multi-step tool invocations.
  arXiv  Detail & Related papers  (2025-12-31T02:09:48Z)
• MCP-SafetyBench: A Benchmark for Safety Evaluation of Large Language Models with Real-World MCP Servers [17.96465932881902]
  We present MCP-SafetyBench, a comprehensive benchmark built on real MCP servers.<n>It incorporates a unified taxonomy of 20 MCP attack types spanning server, host, and user sides.<n>Using MCP-SafetyBench, we systematically evaluate leading open- and closed-source LLMs.
  arXiv  Detail & Related papers  (2025-12-17T08:00:32Z)
• MCPGuard : Automatically Detecting Vulnerabilities in MCP Servers [16.620755774987774]
  The Model Context Protocol (MCP) has emerged as a standardized interface enabling seamless integration between Large Language Models (LLMs) and external data sources and tools.<n>This paper systematically analyzes the security landscape of MCP-based systems, identifying three principal threat categories.
  arXiv  Detail & Related papers  (2025-10-27T05:12:51Z)
• Mind Your Server: A Systematic Study of Parasitic Toolchain Attacks on the MCP Ecosystem [13.95558554298296]
  Large language models (LLMs) are increasingly integrated with external systems through the Model Context Protocol (MCP)<n>In this paper, we reveal a new class of attacks, Parasitic Toolchain Attacks, instantiated as MCP Unintended Privacy Disclosure (MCP-UPD)<n>The malicious logic infiltrates the toolchain and unfolds in three phases: Parasitic Ingestion, Privacy Collection, and Privacy Disclosure, culminating in stealthy exfiltration of private data.
  arXiv  Detail & Related papers  (2025-09-08T11:35:32Z)
• How Can Input Reformulation Improve Tool Usage Accuracy in a Complex Dynamic Environment? A Study on $τ$-bench [58.114899897566964]
  In a multi-turn conversational environment, large language models (LLMs) often struggle with consistent reasoning and adherence to domain-specific policies.<n>We propose the Input-Reformulation Multi-Agent (IRMA) framework, which automatically reformulates user queries augmented with relevant domain rules.<n>IRMA significantly outperforms ReAct, Function Calling, and Self-Reflection by 16.1%, 12.7%, and 19.1%, respectively.
  arXiv  Detail & Related papers  (2025-08-28T15:57:33Z)
• MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Servers [86.00932417210477]
  We introduce MCP-Universe, the first comprehensive benchmark specifically designed to evaluate LLMs in realistic and hard tasks through interaction with real-world MCP servers.<n>Our benchmark encompasses 6 core domains spanning 11 different MCP servers: Location Navigation, Repository Management, Financial Analysis, 3D Design, Browser Automation, and Web Searching.<n>We find that even SOTA models such as GPT-5 (43.72%), Grok-4 (33.33%) and Claude-4.0-Sonnet (29.44%) exhibit significant performance limitations.
  arXiv  Detail & Related papers  (2025-08-20T13:28:58Z)
• MCP-Orchestrated Multi-Agent System for Automated Disinformation Detection [84.75972919995398]
  This paper presents a multi-agent system that uses relation extraction to detect disinformation in news articles.<n>The proposed Agentic AI system combines four agents: (i) a machine learning agent (logistic regression), (ii) a Wikipedia knowledge check agent, and (iv) a web-scraped data analyzer.<n>Results demonstrate that the multi-agent ensemble achieves 95.3% accuracy with an F1 score of 0.964, significantly outperforming individual agents and traditional approaches.
  arXiv  Detail & Related papers  (2025-08-13T19:14:48Z)
• Visual Document Understanding and Question Answering: A Multi-Agent Collaboration Framework with Test-Time Scaling [83.78874399606379]
  We propose MACT, a Multi-Agent Collaboration framework with Test-Time scaling.<n>It comprises four distinct small-scale agents, with clearly defined roles and effective collaboration.<n>It shows superior performance with a smaller parameter scale without sacrificing the ability of general and mathematical tasks.
  arXiv  Detail & Related papers  (2025-08-05T12:52:09Z)
• We Urgently Need Privilege Management in MCP: A Measurement of API Usage in MCP Ecosystems [28.59170303701817]
  We conduct the first large-scale empirical analysis of Model Context Protocol security risks.<n>We examine 2,562 real-world MCP applications spanning 23 functional categories.<n>We propose a detailed taxonomy of MCP resource access, quantify security-relevant API usage, and identify open challenges for building safer MCP ecosystems.
  arXiv  Detail & Related papers  (2025-07-05T03:39:30Z)
• SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints [59.645885492637845]
  SOPBench is an evaluation pipeline that transforms each service-specific SOP code program into a directed graph of executable functions.<n>Our approach transforms each service-specific SOP code program into a directed graph of executable functions and requires agents to call these functions based on natural language SOP descriptions.<n>We evaluate 18 leading models, and results show the task is challenging even for top-tier models.
  arXiv  Detail & Related papers  (2025-03-11T17:53:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.