eval-benchlisted
Install: claude install-skill glslang/windbg-mcp
# Benchmarking a model against this server
## The grid, and what bites while running it
`docs/local-model-eval.md` is the result; this is what bites while running it again. The grid is
three scripts — the ollama driver, the Claude Code driver, and the matrix runner that spawns either
one per cell and grades the log afterwards.
**Record what the runtime *served*, not what you asked for.** `num_ctx` on a request does not
shrink an instance ollama already holds: with a 32,768 instance loaded, cells asking for 8,192 are
served 32,768 and look perfectly healthy — a 17,300-token prompt "fitting" in 8k, which is the
result the context axis exists to find and would have been fiction. `/api/ps` is the only place
the truth appears. Every record carries `served_context`, the grader marks a cell where the two
disagree with `?`, and the runner evicts the model between windows. The first run of the grid
recorded five such cells; they were dropped and re-run.
**The grader's three matching rules each came from a real wrong verdict**, and all three are in
`present()`:
- A number matches only **between hex boundaries** — `0x22` is the device type `ioctl_decode` asks
for and `0x22200B` is the code in the question, so plain containment passed for any answer that
repeated the question. One model scored correct while saying `FILE_DEVICE_KEYBOARD`.
- Leading zeros are formatting — the tool prints `0x802` and a model writing `0x0802` agrees with
it. That one marked a *correct* control ans