From Docs to Descriptions: Smell-Aware Evaluation of MCP Server Descriptions

• URL: http://arxiv.org/abs/2602.18914v1
• Date: Sat, 21 Feb 2026 17:45:07 GMT
• Title: From Docs to Descriptions: Smell-Aware Evaluation of MCP Server Descriptions
• Abstract summary: We present the first systematic study of description smells in MCP tool descriptions and their impact on usability.<n>We find that description smells are pervasive, reflecting a "code-first, description-last" pattern.<n>In competitive settings with functionally equivalent servers, standard-compliant descriptions reach 72% selection probability.
• Score: 15.082788201596422
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The Model Context Protocol (MCP) has rapidly become a de facto standard for connecting LLM-based agents with external tools via reusable MCP servers. In practice, however, server selection and onboarding rely heavily on free-text tool descriptions that are intentionally loosely constrained. Although this flexibility largely ensures the scalability of MCP servers, it also creates a reliability gap that descriptions often misrepresent or omit key semantics, increasing trial-and-error integration, degrading agent behavior, and potentially introducing security risks. To this end, we present the first systematic study of description smells in MCP tool descriptions and their impact on usability. Specifically, we synthesize software/API documentation practices and agentic tool-use requirements into a four-dimensional quality standard: accuracy, functionality, information completeness, and conciseness, covering 18 specific smell categories. Using this standard, we conducted a large-scale empirical study on a well-constructed dataset of 10,831 MCP servers. We find that description smells are pervasive (e.g., 73% repeated tool names, thousands with incorrect parameter semantics or missing return descriptions), reflecting a "code-first, description-last" pattern. Through a controlled mutation-based study, we show these smells significantly affect LLM tool selection, with functionality and accuracy having the largest effects (+11.6% and +8.8%, p < 0.001). In competitive settings with functionally equivalent servers, standard-compliant descriptions reach 72% selection probability (260% over a 20% baseline), demonstrating that smell-guided remediation yields substantial practical benefits. We release our labeled dataset and standards to support future work on reliable and secure MCP ecosystems.

Related papers

• LLM within MCP Matters: Measuring Inefficient Resource Utilization Driven by LLMs [6.657352531941857]
  The Model Context Protocol (MCP) standardizes how servers expose data to Large Language Models (LLMs)<n>A common server design embeds frequently used reference data, such as identifier lookup tables, directly in the server instructions.<n>We report a 54,000-trial study across 24 LLMs on a production legal-information MCP server.
  arXiv  Detail & Related papers  (2026-08-09T04:17:12Z)
• Description-Code Inconsistency in Real-world MCP Servers: Measurement, Detection, and Security Implications [15.815506064408074]
  We present a comprehensive study of Description-Code Inconsistency (DCI) in real-world MCP servers.<n>We propose a comprehensive taxonomy spanning functionality inconsistencies and undeclared side effects.<n>We apply this taxonomy to a large-scale dataset comprising 19,200 description-code pairs extracted from 2,214 real-world MCP servers.
  arXiv  Detail & Related papers  (2026-06-03T11:51:32Z)
• From Insight to Action: A Novel Framework for Interpretability-Guided Data Selection in Large Language Models [73.72877445629383]
  Interpretability-Guided Data Selection (IGDS) is a framework that first identifies causal task features through frequency recall and interventional filtering.<n>We validate IGDS on mathematical reasoning, summarization, and translation tasks within Gemma-2, LLaMA-3.1, and Qwen3 models.
  arXiv  Detail & Related papers  (2026-04-28T03:16:24Z)
• Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptions [12.206378714907075]
  We examine 856 tools spread across 103 MCP servers empirically, assess their description quality, and their impact on agent performance.<n>We identify six components of tool descriptions from the literature, develop a scoring rubric utilizing these components, and then formalize tool description smells based on this rubric.<n>We find that 97.1% of the analyzed tool descriptions contain at least one smell, with 56% failing to state their purpose clearly.
  arXiv  Detail & Related papers  (2026-02-16T16:10:11Z)
• Don't believe everything you read: Understanding and Measuring MCP Behavior under Misleading Tool Descriptions [20.772501129444993]
  We study how mismatches between externally presented tool descriptions and underlying implementations shape the mental models and decision-making behavior of intelligent agents.<n>Our results show that while most servers are highly consistent, approximately 13% exhibit substantial mismatches that can enable undocumented privileged operations, hidden state mutations, or unauthorized financial actions.<n>Our findings demonstrate that description-code inconsistency is a concrete and prevalent attack surface in MCP-based AI agents, and motivate the need for systematic auditing and stronger transparency guarantees in future agent ecosystems.
  arXiv  Detail & Related papers  (2026-02-03T14:31:52Z)
• MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers [5.463884405989425]
  We introduce MCP-Atlas, a large-scale benchmark for evaluating tool-use competency.<n>It includes 1,000 tasks designed to assess tool-use competency in realistic, multi-step orchestrate.<n>We score tasks using a claims-based rubric that awards partial credit based on the factual claims satisfied in the model's final answer.
  arXiv  Detail & Related papers  (2026-01-31T23:19:39Z)
• MCPAgentBench: A Real-world Task Benchmark for Evaluating LLM Agent MCP Tool Use [12.220519951554133]
  MCPAgentBench is a benchmark based on real-world MCP definitions to evaluate the tool-use capabilities of agents.<n>The evaluation employs a dynamic sandbox environment that presents agents with candidate tool lists containing distractors.<n>Experiments conducted on various latest mainstream Large Language Models reveal significant performance differences in handling complex, multi-step tool invocations.
  arXiv  Detail & Related papers  (2025-12-31T02:09:48Z)
• MCP-RiskCue: Can LLM Infer Risk Information From MCP Server System Logs? [3.4468299705073133]
  We present the first synthetic benchmark for evaluating large language models' ability to identify security risks from system logs.<n>We define nine categories of MCP server risks and generate 1,800 synthetic system logs using ten state-of-the-art LLMs.<n>Pilot experiments reveal that smaller models often fail to detect risky system logs, leading to high false positives.
  arXiv  Detail & Related papers  (2025-11-08T05:52:53Z)
• InfoMosaic-Bench: Evaluating Multi-Source Information Seeking in Tool-Augmented Agents [60.89180545430896]
  InfoMosaic-Bench is the first benchmark dedicated to multi-source information seeking in tool-augmented agents.<n>It requires agents to combine general-purpose search with domain-specific tools.<n>This design guarantees both reliability and non-triviality.
  arXiv  Detail & Related papers  (2025-10-02T17:48:03Z)
• MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Servers [86.00932417210477]
  We introduce MCP-Universe, the first comprehensive benchmark specifically designed to evaluate LLMs in realistic and hard tasks through interaction with real-world MCP servers.<n>Our benchmark encompasses 6 core domains spanning 11 different MCP servers: Location Navigation, Repository Management, Financial Analysis, 3D Design, Browser Automation, and Web Searching.<n>We find that even SOTA models such as GPT-5 (43.72%), Grok-4 (33.33%) and Claude-4.0-Sonnet (29.44%) exhibit significant performance limitations.
  arXiv  Detail & Related papers  (2025-08-20T13:28:58Z)
• LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools? [50.60770039016318]
  We present LiveMCPBench, the first comprehensive benchmark for benchmarking Model Context Protocol (MCP) agents.<n>LiveMCPBench consists of 95 real-world tasks grounded in the MCP ecosystem.<n>Our evaluation covers 10 leading models, with the best-performing model reaching a 78.95% success rate.
  arXiv  Detail & Related papers  (2025-08-03T14:36:42Z)
• SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints [59.645885492637845]
  SOPBench is an evaluation pipeline that transforms each service-specific SOP code program into a directed graph of executable functions.<n>Our approach transforms each service-specific SOP code program into a directed graph of executable functions and requires agents to call these functions based on natural language SOP descriptions.<n>We evaluate 18 leading models, and results show the task is challenging even for top-tier models.
  arXiv  Detail & Related papers  (2025-03-11T17:53:02Z)
• Towards Realistic Evaluation of Commit Message Generation by Matching Online and Offline Settings [77.20838441870151]
  We use an online metric - the number of edits users introduce before committing the generated messages to the VCS - to select metrics for offline experiments.<n>We collect a dataset with 57 pairs consisting of commit messages generated by GPT-4 and their counterparts edited by human experts.<n>Our results indicate that edit distance exhibits the highest correlation with the online metric, whereas commonly used similarity metrics such as BLEU and METEOR demonstrate low correlation.
  arXiv  Detail & Related papers  (2024-10-15T20:32:07Z)
• FamiCom: Further Demystifying Prompts for Language Models with Task-Agnostic Performance Estimation [73.454943870226]
  Language models have shown impressive in-context-learning capabilities. We propose a measure called FamiCom, providing a more comprehensive measure for task-agnostic performance estimation.
  arXiv  Detail & Related papers  (2024-06-17T06:14:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.