benchmark-methodologylisted
Install: claude install-skill jgravelle/jcodemunch-mcp
# Benchmark methodology
Authority: `docs/harness/ARCHAEOLOGY.md` (the benchmark rows),
`benchmarks/METHODOLOGY.md`, `benchmarks/REPRODUCING.md`, CLAUDE.md
Practice 4, `docs/harness/FINDINGS.md` F-10, F-13, F-17.
Rules, each a pointer, none restated:
1. **Never hand-type a number.** Every figure comes from a run in THIS
session or the CI-captured artifact `benchmarks/jcm_reference.json`
(Practice 4). The comparators read that file; there is no estimator.
2. **Per row, never per total** (F-13): the total hid one cause behind
another. Diff threshold ids one by one.
3. **Five mirrors move together** (Practice 4): `results.md`,
`METHODOLOGY.md`, README, `provenance/measured.json`, `REPRODUCING.md`;
`tests/test_provenance.py` is the gate and `--reference` rewrites them.
4. **The reference is captured where the gate runs** (F-13):
`benchmark.yml` dispatch with `reference=true`; a dev-box number differs
by CRLF, walk order and a HOME ledger (F-17).
5. **Deterministic configuration** is the bench tier itself: `--offline`,
pinned corpora (`benchmarks/tasks.json`), the no-network fixture. Add no
flags.
6. **A refusal is not a zero**: an absent value prints `n/a`.
7. **Floors live only in `harness/thresholds.json`**; read one with
`uv run python -m harness threshold <id>`.
8. `benchmarks/schema_baseline.json` is the only source for schema-token
figures (CLAUDE.md "Tier-switch pricing"); the harness counts a
different payload and the two never agree d