eval-rerunlisted
Install: claude install-skill EllypsisAI/ellypsis-plugin-labs
# Re-run one eval case
`$ARGUMENTS` — the case id from `eval-workspace/plan.json`.
Re-running a single case answers two questions that a full eval cannot. **Is this result
stable?** An MCP integration that passes once and fails once is not a passing
integration, and the only way to see that is to run the same brief twice. **Did the fix
work?** After the user changes a tool description or a schema, one case re-run is the
cheap confirmation — a full eval is not.
Always operates on `eval-workspace/` in the session cwd. There is no workspace argument
on purpose: the hooks resolve that fixed path when deciding whether to record telemetry,
so a rerun pointed at an archived workspace would write its marker somewhere the hooks
never look, capture nothing, and hand back an ungradable case that looks like a server
failure. To re-run against an archived workspace, restore it to `eval-workspace/` first.
## Steps
1. Load `eval-workspace/plan.json` and find the case by id. Not found → list the
available ids and stop.
2. Read the existing `cases/<id>/result.json`, `cases/<id>/grading.json`, and that
case's `telemetry.jsonl` lines **before** anything is overwritten. They are the
comparison baseline and the two JSON files are about to be gone. (Telemetry appends
and payload files are keyed by `tool_use_id`, so both runs' raw evidence survives.)
3. If the case's tool is write-capable, get fresh approval naming that tool. A rerun is a
new live call — the earlier approval cov