Per-server prompt engineering is therefore a workaround rather than a fix; it is argued that MCP host applications should provide an explicit mechanism that places server instructions ahead of tool selection in the client LLM's deliberation.
Abstract
The Model Context Protocol (MCP) standardizes how servers expose data and tools to Large Language Models (LLMs). A common server design embeds frequently used reference data, such as identifier lookup tables, directly in the server instructions: the system-prompt text a server hands to the host application. When a query concerns an entry of the embedded table, the model can act on it immediately instead of re-discovering the same information through a search tool. We test whether client LLMs actually consume such instruction-embedded data, reporting a 54,000-trial study across 24 LLMs (9 Claude, 6 Gemini, 9 GPT) on a production legal-information MCP server. A diagnostic condition that removes the competing search tool shows that failures are dominated by behavioral preference rather than missing capability. With search unavailable, 23 of 24 models read the embedded data reliably (hit ratio at least 98%); with a search tool merely present, 9 models drop below 15%. A 2^3 factorial analysis of three instruction-level interventions reveals strong interaction effects: combining all three restores at least 86% for 20 of 24 models, but individual interventions can backfire for specific model families. Per-server prompt engineering is therefore a workaround rather than a fix; we argue that MCP host applications should provide an explicit mechanism that places server instructions ahead of tool selection in the client LLM's deliberation.
The Model Context Protocol (MCP) standardizes how large language model applications communicate with external tools, but leaves the application side unspecified: unlike traditional dependencies resolved through package managers, developers integrating MCP servers face no conventions for configuration, communication, or human oversight. This ecosystem is also under-researched, with existing work focused on servers rather than the applications consuming them. We conduct a large-scale study of 1,723 MCPApps mined from GitHub. We first derive MCPAppTax from a representative sample, then use an LLM-assisted pipeline to apply it across the full dataset, characterizing server integration across configuration, SDK use, and human-in-the-loop mechanisms. Our results show that the ecosystem has converged on some practices but not others: most MCPApps configure servers using files (85.2%) and use an official SDK (81.1%) to communicate with servers, yet no naming convention has emerged for configuration files. Human oversight diverges most, logging (90.8%) and enable/disable controls (77.2%) are common, but only 37.2% gate tool execution behind a blocking approval step, leaving the LLM able to invoke any enabled tool unconditionally in most MCPApps.
Muhammad Hamza Arshad Majeed, May Mahmoud, Sarah Nadi· arXiv.org· 0 citations
Studies of the Model Context Protocol (MCP) server ecosystem draw their samples in ways that quietly select for servers that work: reference sets, popularity lists, hand-curated frames, or pipelines that repair a server until it starts. We report what an unrepaired probability sample actually contains. From a 24,135-server registry census we draw 400 npm/stdio servers with a published seed and probe each one over the wire. Only 48.8% complete an initialize handshake, against 66.7% for a hand-curated frame measured with the same instrument, and the dominant failure is not missing credentials (13.3%) but servers that never start at all (37.5%). Among the 195 that do run, hard conformance is total: zero fatal JSON Schema violations across 2,766 advertised tools. Optional safety annotations are the real variance, and the tool-level omission rate on a random draw is 58.8% against 41.5% on the curated frame, so curation flatters this figure too. We then compare the tool descriptions these servers advertise against two tool-use benchmark corpora using one method held constant. Real MCP tools show 2.8% near-duplication at cosine 0.70, and all of it lies within single servers: cross-author near-duplication is 0.0% at every threshold tested. BFCL v4 shows 16.7%, of which 16.4 points lie between independently presented tasks. UltraTool shows 0.3%, cleaner than real tools, so this is a property of BFCL and not of synthetic corpora as a class. Separately, 68.8% of raw BFCL rows and 85.6% of raw UltraTool rows are exact name-plus-description repeats, against 0.4% for real MCP, so any statistic computed over these releases without global deduplication measures repetition rather than tools. All figures regenerate from released scripts and a published seed.
Resource constrained single-board computers including Raspberry Pi, NVIDIA Jetson Nano, Arduino UNO Q, Orange Pi, and LattePanda motivate on-device small language model (SLM) agents that reduce cloud dependence, improve data locality, and tolerate intermittent connectivity. Model Context Protocol (MCP)-style tool invocation demands more than fluent generation: an agent must emit machine-readable JSON, select the correct tool, supply all required arguments, and avoid unintended actions. We establish a platform-agnostic CPU baseline by evaluating five open-weight models below two billion parameters Phi-1.5, Pythia-1.4B, TinyLlama-1.1B-Chat, Qwen2.5-0.5B, and Qwen2.5-1.5B on 100 prompts spanning weather retrieval, web search, calculation, email composition, and task creation, under greedy decoding and nucleus sampling. A recovery parser strips Markdown fences, extracts brace-delimited substrings, and scores parseability, tool-name correctness, argument completeness, and value agreement. Under this criterion, Qwen2.5-1.5B achieves 75% (greedy) and 79% (sampling); Qwen2.5-0.5B achieves 72% (greedy) but drops to 32% under sampling. Phi-1.5 scores 0%; Pythia and TinyLlama reach at most 7%. A strict post-hoc audit finds only 5 of 1,000 raw responses directly parseable as JSON, exposing near-total dependence on output recovery. A CPU resource probe shows Qwen2.5-1.5B requires 7,960 MiB and 30.782 s mean latency; Qwen2.5-0.5B uses 3,637 MiB and 10.627 s, revealing a reliability-resource trade-off for edge deployment. These results do not cover the named boards directly or a full MCP implementation. Safe deployment requires schema validation, constrained generation, least-privilege execution, and human escalation for consequential actions.
This single-site technical case study examined a middleware-based integration pattern for consuming IP-reputation intelligence on an Internet-facing MikroTik RouterOS router without requiring the router to process the provider API directly. A Python Flask service retrieved AbuseIPDB blacklist records using a minimum abuse confidence score of 90, retained valid IPv4 addresses in process memory, and exposed newline-delimited pages containing at most 100 entries. A scheduled RouterOS script retrieved the pages, repeated IPv4 validation, avoided duplicate insertion, and populated an address list used by input- and forward-chain drop rules. Operational observations were derived from archived aggregate dashboard plots covering 29 April-5 May and 8-14 May 2026. The raw event export and original dashboard aggregation configuration were unavailable; therefore no causal effect or time-normalized attack rate was estimated. In the baseline plot, API/admin combinations represented 71.1% of recorded activity, the highest displayed aggregation bin was approximately 16,000 events, and one source exceeded 100,000 cumulative interactions. In the post-deployment plot, the previously dominant API/admin pattern was absent, most displayed bins were approximately 200-450 events, the largest was about 1,400, and the most active source was about 1,200. These changes were temporally associated with deployment but may also reflect external variation in Internet attack activity. The engineering contribution is a RouterOS-oriented integration pattern that isolates the provider credential, transforms the reputation feed into an ingestible plaintext representation, and bounds router-side processing through pagination and defensive validation. The evidence supports technical feasibility at one site, not causal effectiveness, resource savings, or generalizability.
Database connectors for the Model Context Protocol (MCP) let large language model (LLM) agents run SQL against production data, and they return rows verbatim into the model's context window with no filtering. This paper contributes a reproducible benchmark and reference-gateway design, evaluated on synthetic data, for what controls at that boundary can and cannot do. Working over a synthetic-PII SQLite database seeded with canary records and a minimal, MCP-style JSON-RPC client/server, we build a six-rung defense ladder, from an unprotected passthrough to a gateway that combines engine read-only, a single-statement SQL guard, a server-authenticated disclosure policy driven by AST source-column lineage, deterministic reversible tokenization, and egress redaction, and measure each rung's marginal effect with a leak oracle that shares no code with the gateway. Two findings are the point of the paper. First, controls that look sufficient are not: an engine-enforced read-only connection blocks writes but leaves over-exposure unchanged, and a naive allow-list keyed on the output column name drops benign over-exposure to zero yet is defeated by nearly half of our 17-attack suite through trivial aliasing and encoding (SELECT hex(ssn) AS x, SELECT password_hash AS x, two bypasses raised in peer review of an earlier draft). Only when disclosure decisions are made on source columns recovered from the SQL AST (resolving aliases, expressions, aggregates, unions, and CTEs) does attacker success on this suite fall to zero while over-exposure stays at zero. Second, masking direct identifiers is not de-identification: on our data, tokenizing names and emails still leaves 99.2% of customers uniquely re-identifiable from three quasi-identifiers, until date-of-birth is generalized (to 12.5%) and then minimized away (to 0%). Alongside, we report an egress PII detector whose held-out F1 is 0.986 but whose recall on adversarially transformed PII is only 0.20, quantifying why value-level detection must be a backstop, not the primary control. We do not evaluate the current MCP specification/SDK, real MCP server implementations, or real LLMs; those, and a faithful reproduction of the disclosed PostgreSQL connector vulnerability, are stated as future work. All numbers are measured on synthetic data; no real personal data is used.
Ye.Ye. Piotrovskyi· American Impact Review· 0 citations
OO-Spec is fastest among all evaluated methods in all 21 target-benchmark cells, and outperforms every evaluated released learned drafter in each comparable cell, while the same sidecar improves on ToolSpec by 34.1% on average.
Zhi-Heng Zhang, Mu-Jie Xu, Fei Sun et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
