Benchmarking MCP Vulnerability Detection from a Paired Evaluation Perspective
TL;DR
A dedicated benchmark for pairing MCP server vulnerability detection is presented, comprising 75 paired vulnerable and fixed MCP tools synthesized by Gemini-1.5-Pro and manually cross-validated, adopting the Matched Pair Accuracy (MPA) metric to evaluate detection methods.
Abstract
Model Context Protocol (MCP) servers have become essential for connecting large language models (LLMs) with external tools, data sources, and execution environments. However, their privileged access and bridging role introduce significant security risks. Despite many LLM-based detection methods, it remains unclear whether they can semantically differentiate paired vulnerable and fixed MCP servers rather than just identifying risky API calls. In this work, we present a dedicated benchmark for pairing MCP server vulnerability detection, comprising 75 paired vulnerable and fixed MCP tools synthesized by Gemini-1.5-Pro and manually cross-validated. We adopt the Matched Pair Accuracy (MPA) metric to evaluate detection methods. Claude Haiku-4.5 achieves the best MPA of 0.853, while GPT-5-mini attains 0.842 F1 but only 0.653 MPA, showing that conventional metrics can overestimate paired detection performance. Our benchmark provides a new perspective for systematic, reproducible, and MCP-specific security evaluation.