An agent picks a tool by reading its name, description, and parameter schema — metadata that is easy to get wrong and almost never tested. MCP Tuner sits between any host and any MCP server and scores that metadata, logs the wire traffic, and A/B tests the wording.
Descriptions [PASS] C tool-description-present every tool has a description [WARN] M tool-description-when-to-use 2 descriptions say what it is, not when to reach for it subjects: search_cloudflare_documentation, migrate_pages_to_workers_guide Input schema [FAIL] H param-description-present 1 parameter undescribed subjects: search_cloudflare_documentation.query fix: describe each param — missing param descriptions degrade call correctness Annotations [PASS] H tool-annotations-present tools declare annotations [SKIP] H tool-destructive-hint no tool is explicitly marked non-read-only 21 pass · 3 warn · 1 fail · 3 skip
Fleet scan · 2026-08-30
Every number below came from running mcptuner audit against the live endpoint. No server was modified, and none was told it was being measured.
| Server | Tools | Score | C/H fails | Warns | Tool-list tokens |
|---|---|---|---|---|---|
| Microsoft Learnlearn.microsoft.com/api/mcp | 3 | 97% | 0 | 2 | ~1215 |
| icanhazdomainmcp.icanhazdomain.com | 1 | 97% | 0 | 2 | ~962 |
| Cloudflare docsdocs.mcp.cloudflare.com | 2 | 93% | 1 | 3 | ~449 |
| everything (reference)@modelcontextprotocol/server-everything | 13 | 87% | 2 | 3 | ~1914 |
| ipfactsmcp.ipfacts.com | 0 | 69% | 1 | 1 | ~1 |
What the scan found
The server declares the tools capability and then returns an empty list. Every host that connects shows a server with nothing in it. Either an auth gate is rejecting the listing silently or tool registration is broken — and no client-side error surfaces either way.
The query parameter of search_cloudflare_documentation has no description. The model is left inferring what to put in the one field that determines the result — the most common cause of a correct tool being called with the wrong argument.
Descriptions state what a tool is without stating when to reach for it. The trigger clause is the part a model matches against when several tools could plausibly apply, and it is the single most common omission across the fleet.
Modes
Audit and observe need no API key and no model — they are deterministic and free to run in CI. Tune and compare need an agent driver, and are specified but not yet built.
auditshippedScores tool, prompt, and resource metadata against a 28-rule catalog over stdio or Streamable HTTP. Exits non-zero on Critical or High failures, so it gates a build.
observeshippedA transparent proxy between host and server. Logs every JSON-RPC frame to JSONL, scrubs secrets on the way to disk, and forwards bytes untouched.
tunespecifiedA/B tests one variable at a time — a name, a description, a schema — across repeated trials, and reports selection accuracy and token cost with a confidence interval.
comparepartialStatic scoreboard across servers works today. The live arena — two servers with overlapping tools, measuring which an agent actually prefers — waits on the harness.
The rubric
Each rule has an id, a severity, a machine-checkable condition, and a fix. Status is four-valued — PASS, FAIL, WARN, SKIP — and a rule is never silently dropped: every non-pass carries a reason, enforced by a unit test. Heuristics report WARN rather than FAIL, because a heuristic must not be presented as certainty.
| id | sev | check → pass | fix |
|---|---|---|---|
| tool-description-distinct | H | no two descriptions exceed 0.9 token-set similarity | Differentiate — near-identical descriptions are the top cause of wrong-tool selection. |
| param-description-present | H | every property has a non-empty description | Describe each param; omissions measurably degrade call correctness. |
| tool-readonly-hint | H | annotations.readOnlyHint set explicitly | An unmarked write is indistinguishable from a read on the wire. |
| tool-description-no-injection | H | no imperatives aimed at the calling model | Descriptions describe. They never instruct the caller. |
| schema-not-closed | M | additionalProperties is not false | Prefer a corrective error message — it is the model's entire self-correction channel. |
| toollist-token-budget | M | serialized tools/list under 10k tokens | Fixed cost, paid on every single request. Trim or split. |
Invariants
Frames pass through byte-faithfully. Round-tripping traffic through typed models drops unknown fields — _meta, experimental capabilities, methods from a newer protocol version — corrupting the very traffic being observed. A build gate sends an _meta canary through the proxy and fails if it does not survive.
No agent framework, ever, in the wire path. Framework middleware can intercept tool calls, but only for agents built in that framework. This works with Claude Code, Codex, Cursor, Copilot — anything that speaks MCP.
Every rewrite is undone before it reaches the server. Renaming a tool for an experiment means mapping the alias back on the call. An unmatched alias is a hard error, not a silent passthrough.
Batch trials refuse side-effecting tools by default. A/B runs call tools repeatedly and automatically; a server exposing delete, send, or pay needs a sandbox mode or explicit per-tool opt-in first.
Evidence with every finding, a reason with every skip. Token counts are labeled estimates. Heuristics are labeled heuristics. A single-shot A-beats-B comparison is never reported as a result.
Start