Related papers: MCP Pitfall Lab: Exposing Developer Pitfalls in MCP Tool Server Security under Multi-Vector Attacks

MCP Pitfall Lab: Exposing Developer Pitfalls in MCP Tool Server Security under Multi-Vector Attacks

URL: http://arxiv.org/abs/2604.21477v1
Date: Thu, 23 Apr 2026 09:39:15 GMT
Title: MCP Pitfall Lab: Exposing Developer Pitfalls in MCP Tool Server Security under Multi-Vector Attacks
Abstract summary: We present MCP Pitfall Lab, a protocol-aware security testing framework that operationalizes developer pitfalls as reproducible scenarios.<n>Pitfall Lab enables practical, end-to-end assessment and hardening of MCP tool servers under realistic multi-vector conditions.
Score: 0.7305019142196584
License: http://creativecommons.org/licenses/by/4.0/
Abstract: Model Context Protocol (MCP) is increasingly adopted for tool-integrated LLM agents, but its multi-layer design and third-party server ecosystem expand risks across tool metadata, untrusted outputs, cross-tool flows, multimodal inputs, and supply-chain vectors. Existing MCP benchmarks largely measure robustness to malicious inputs but offer limited remediation guidance. We present MCP Pitfall Lab, a protocol-aware security testing framework that operationalizes developer pitfalls as reproducible scenarios and validates outcomes with MCP traces and objective validators (rather than agent self-report). We instantiate three workflow challenges (email, document, crypto) with six server variants (baseline and hardened) and model three attack families: tool-metadata poisoning, puppet servers, and multimodal image-to-tool chains, in a unified, trace-grounded evaluation. In Tier-1 static analysis over six variants (36 binary labels), our analyzer achieves F1 = 1.0 on four statically checkable pitfall classes (P1, P2, P5, P6) and flags cross-tool forwarding and image-to-tool leakage (P3, P4) as trace/dataflow-dependent. Applying recommended hardening eliminates all Tier-1 findings (29 to 0) and reduces the framework risk score (10.0 to 0.0) at a mean cost of 27 lines of code (LOC). Finally, in a preliminary 19-run corpus from the email system challenge (tool poisoning and puppet attacks), agent narratives diverge from trace evidence in 63.2% of runs and 100% of sink-action runs, motivating trace-based auditing and regression testing. Overall, Pitfall Lab enables practical, end-to-end assessment and hardening of MCP tool servers under realistic multi-vector conditions.

Related papers

ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox [61.862814740220806]
$textbfComplexMCP$ is a benchmark designed to evaluate agents in rigorous conditions.<n>Built on the Model Context Protocol (MCP), $textbfComplexMCP$ provides over 300 meticulously tested tools.
arXiv Detail & Related papers (2026-05-11T16:20:51Z)
Unsafe by Flow: Uncovering Bidirectional Data-Flow Risks in MCP Ecosystem [8.214897650566494]
Model Context Protocol (MCP) have quickly become the interface layer between LLM agents and external tools.<n> Vulnerabilities manifest in two directions: requester-controlled arguments may propagate to sensitive operations, while untrusted external or sensitive internal data may surface.<n>We present MCP-BiFlow, a static analysis framework built around MCP-aware entrypoint recovery, protocol-specific taint modeling, and interprocedural propagation analysis.
arXiv Detail & Related papers (2026-05-08T15:03:51Z)
MCPHunt: An Evaluation Framework for Cross-Boundary Data Propagation in Multi-Server MCP Agents [10.895733091955547]
We present the first benchmark that isolates verbatim credential propagation across multi-server MCP trust boundaries.<n>Across 3,615 main-benchmark traces from 5 models, policy-violating propagation rates reach 11.5--41.3% across all models.<n>A prompt-mitigation study across 3 models reduces policy-violating propagation by up to 97% while preserving 80.5% utility.
arXiv Detail & Related papers (2026-04-30T13:01:03Z)
Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents [66.97968363332465]
We introduce Claw-Eval, an end-to-end evaluation suite addressing all three gaps in agent benchmarks.<n>Claw-Eval comprises 300 human-verified tasks spanning 9 categories across three groups.<n>Every agent action is recorded through three independent evidence channels.
arXiv Detail & Related papers (2026-04-07T17:43:18Z)
ShieldNet: Network-Level Guardrails against Emerging Supply-Chain Injections in Agentic Systems [56.613157564882925]
Malicious behaviors are embedded in seemingly benign tools, silently hijacking agent execution, leaking sensitive data, or triggering unauthorized actions.<n>Despite their growing impact, there is currently no comprehensive benchmark for evaluating such threats.<n>We propose ShieldNet, a network-level guardrail framework that detects supply-chain poisoning by observing real network interactions.
arXiv Detail & Related papers (2026-04-06T05:15:00Z)
AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents [50.481033105867205]
We introduce AgentProcessBench, the first benchmark dedicated to evaluating step-level effectiveness in realistic, tool-augmented trajectories.<n>The benchmark comprises 1,000 diverse trajectories and 8,509 human-labeled step annotations with 89.1% inter-annotator agreement.<n>It features a ternary labeling scheme to capture exploration and an error propagation rule to reduce labeling ambiguity.
arXiv Detail & Related papers (2026-03-15T16:13:58Z)
MCPTox: A Benchmark for Tool Poisoning Attack on Real-World MCP Servers [12.669529656631937]
We introduce MCPTox, the first benchmark to evaluate agent robustness against Tool Poisoning in realistic MCP settings.<n> MCPTox generates a comprehensive suite of 1312 malicious test cases by few-shot learning, covering 10 categories of potential risks.<n>Our evaluation reveals a widespread vulnerability to Tool Poisoning, with o1-mini, achieving an attack success rate of 72.8%.
arXiv Detail & Related papers (2025-08-19T10:12:35Z)
LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools? [50.60770039016318]
We present LiveMCPBench, the first comprehensive benchmark for benchmarking Model Context Protocol (MCP) agents.<n>LiveMCPBench consists of 95 real-world tasks grounded in the MCP ecosystem.<n>Our evaluation covers 10 leading models, with the best-performing model reaching a 78.95% success rate.
arXiv Detail & Related papers (2025-08-03T14:36:42Z)
Model Context Protocol (MCP) at First Glance: Studying the Security and Maintainability of MCP Servers [16.794115541448758]
Anthropic introduced the Model Context Protocol (MCP) to standardize this tool ecosystem in late 2024.<n>Despite its adoption, MCP's AI-driven, non-deterministic control flow introduces new risks to sustainability, security, and maintainability.<n>We evaluate 1,899 open-source MCP servers to assess their health, security, and maintainability.
arXiv Detail & Related papers (2025-06-16T14:26:37Z)
Backdoor Cleaning without External Guidance in MLLM Fine-tuning [76.82121084745785]
Believe Your Eyes (BYE) is a data filtering framework that leverages attention entropy patterns as self-supervised signals to identify and filter backdoor samples.<n>It achieves near-zero attack success rates while maintaining clean-task performance.
arXiv Detail & Related papers (2025-05-22T17:11:58Z)
SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints [59.645885492637845]
SOPBench is an evaluation pipeline that transforms each service-specific SOP code program into a directed graph of executable functions.<n>Our approach transforms each service-specific SOP code program into a directed graph of executable functions and requires agents to call these functions based on natural language SOP descriptions.<n>We evaluate 18 leading models, and results show the task is challenging even for top-tier models.
arXiv Detail & Related papers (2025-03-11T17:53:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.