evallisted

# Evaluate an MCP server `$ARGUMENTS` — the MCP server to evaluate. Optional; detect and ask when absent. ## What this measures A deterministic client can prove a server responds. It cannot tell you whether an LLM **finds** the tool, **picks** it over its neighbours, **calls** it with sane arguments, **survives** the payload, and **grounds** its answer in what came back. That joint system — server plus LLM — is what actually ships, and it is the only thing scored here. Which is why every measured tool call happens inside a sub-agent and never in this session. Two reasons, both structural rather than stylistic. A 50k-token result that lands in a sub-agent dies there; the same result here would eat the context that grading needs. And discovery cannot be measured from a context that already knows the answer — once you have read the schemas, you can no longer tell whether they were findable. ## Invariants | Rule | Why | |------|-----| | Never call the target's tools yourself | You are the instrument, not the subject | | Stages are gates, in order, none skipped | Each one's output is the next one's input; skipping produces a confident eval of nothing | | Cases run one at a time | Telemetry is attributed by a single marker file — two live cases corrupt every metric silently | | Write-capable tools are skipped unless individually approved by name | These calls hit live business systems | | Nothing defaults to false | Ungradable is a real outcome with a reason; a fabricated FAI