Explore MCP tool-routing evals

[zachdunn]

Context

Vitest already proves MCP tool handlers work (apps/mcp/test, packages/uploads/test/mcp.test.ts). It does not prove that a model, given the full catalog, reaches for the right sibling.

Tool copy just changed in #856 (whoami instead of health, shorter param descriptions, schema examples). That is exactly the kind of change MCPJam's evals are meant to measure: Evaluating MCP servers. We want a small, easily testable set of cases that confirm agents pick the right tool, not a research bench.

This issue tracks exploration, not a committed harness.

Goal

A checked-in dataset of confusion pairs: natural-language prompt (never names the tool) → expected tool → tools that must not fire. Score tool name first, arguments second with partial match. Run a few times on one cheap model. Gate on "expected fired" and "forbidden did not."

What we already looked at

Worth stealing the case shape from:

• MCPJam evals / @mcpjam/sdk: query + expected tools + optional args; accuracy / recall / false-positive per tool
• alpic-ai/mcp-eval: smallest YAML CLI (prompt → first tool + partial args)
• MediaWiki MCP evals/: tool_selection.json plus confusion_pairs.json with expected_tool and not_tools
• Apify MCP evals: query + expectedTools + a one-line reason
• GitHub MCP offline evals: confusion matrix, no LLM-as-judge for selection
• Anthropic, writing tools for agents: overlapping tools confuse agents; expected-tool is a diagnostic when copy changes

Skip for this: MCP-Universe, LiveMCPBench, MCPMark, MCP Apps widget evals, multi-turn gyms, token-budget "efficiency" scores, LLM judges for selection.

Proposed work

1. Author ~15 cases (JSON or YAML) covering the pairs below. The dataset is the product; pick a runner later.
2. Try them once in MCPJam Inspector (npx @mcpjam/inspector) against hosted and/or stdio, enough to see if copy regressions show up.
3. Only then decide whether to wire @mcpjam/sdk / alpic into CI. Do not block PRs on a flaky LLM gate until the cases are stable.

First confusion pairs

Prompt (never names the tool)	Expect	Forbid
Which workspace / who am I?	whoami	doctor, usage
Is local setup healthy?	doctor (stdio only)	whoami
Host this PNG on PR 12 in acme/web	put with {pr, repo}	promote, comment
Stage this shot for branch feat/x	put with {branch, repo}	promote
Copy staged files into PR 12	promote	put
Refresh the attachments comment after a delete	comment	put, promote
Find shots of /settings after	find_files	list
What's under prefix gh/acme/web/	list	find_files
What's staged for this branch? (hosted)	find_files or list	(no hosted staged tool)
Capital of France / no upload	none	put

Hosted vs stdio catalogs differ (doctor / staged / attach / screenshot are stdio-only). Cases should say which catalog they apply to.

Out of scope

• Replacing Vitest contract tests
• Dummy MCP Apps widgets to pass Inspector "UI Tools Present"
• Merging gallery or metadata tools
• Cutting a CLI release just to run evals