← ClaudeAtlas

eval-benchlisted

Run and read the local-model benchmark (`tools/local_model_eval.py`) against this server - the grid's fences, what the grader's matching rules are for, `--verify-key`, `--compare`/`--series`, and how a moved aggregate gets misread as a controlled result. Use when running the eval, adding or re-grading a task, or writing up a benchmark result.
glslang/windbg-mcp · ★ 9 · AI & Automation · score 74
Install: claude install-skill glslang/windbg-mcp
# Benchmarking a model against this server ## The grid, and what bites while running it `docs/local-model-eval.md` is the result; this is what bites while running it again. The grid is three scripts — the ollama driver, the Claude Code driver, and the matrix runner that spawns either one per cell and grades the log afterwards. **Record what the runtime *served*, not what you asked for.** `num_ctx` on a request does not shrink an instance ollama already holds: with a 32,768 instance loaded, cells asking for 8,192 are served 32,768 and look perfectly healthy — a 17,300-token prompt "fitting" in 8k, which is the result the context axis exists to find and would have been fiction. `/api/ps` is the only place the truth appears. Every record carries `served_context`, the grader marks a cell where the two disagree with `?`, and the runner evicts the model between windows. The first run of the grid recorded five such cells; they were dropped and re-run. **The grader's three matching rules each came from a real wrong verdict**, and all three are in `present()`: - A number matches only **between hex boundaries** — `0x22` is the device type `ioctl_decode` asks for and `0x22200B` is the code in the question, so plain containment passed for any answer that repeated the question. One model scored correct while saying `FILE_DEVICE_KEYBOARD`. - Leading zeros are formatting — the tool prints `0x802` and a model writing `0x0802` agrees with it. That one marked a *correct* control ans