← ClaudeAtlas

eval-calibrationlisted

Use after editing any eval spec, or when the user says "our scores keep going up", "is the rubric still right", "calibrate the spec", "rubric drift", "the eval passes everything now", "recalibrate". Regrades the frozen anchor set, measures drift, and distinguishes a generator that improved from a ruler that stretched. Writes workspace/evals/calibration/<type>-<date>.md. Tests the ruler, not the work. To judge an artifact use `eval-loop`, and to write or rewrite the rubric itself use `eval-spec-authoring`.
guerrilla2799/ops-and-scale-os · ★ 1 · Testing & QA · score 74
Install: claude install-skill guerrilla2799/ops-and-scale-os
# Eval Calibration Regrade the anchors. A rising pass rate is ambiguous until you have done this, and softening rubrics are invisible from the inside. ## When to use - Any time a spec is edited, including a wording change - Pass rates have moved and nobody knows why - The banned-vocabulary or voice reference changed, which changes every spec importing it - Quarterly, as a standing check ## Inputs - Reads: `workspace/evals/specs/<type>.md`, the frozen calibration set, prior calibration runs - Needs from user: nothing, if the frozen set exists. If it does not, go back to `eval-spec-authoring` ## Workflow ### 1. Understand the failure this catches A spec is edited a few times over some months. Each edit is small and reasonable, usually softening language that felt too harsh. Pass rates climb from about half to nearly nine in ten. Everybody reads that as the generator improving. It is not necessarily. The ruler may have stretched, and the only way to tell is to regrade artifacts whose correct scores you already know. ### 2. Regrade the frozen set Each anchor was assigned an expected band when the spec was written. Re-judge every one with the current spec, in a fresh context, and compare. ``` example-01 expected 85-95 scored 91 ✓ in band example-02 expected 70-79 scored 84 ✗ +5 above band example-03 expected 40-55 scored 62 ✗ +7 above band ... Mean drift: +4.8 points Direction: softening ``` ### 3. Read the drift | Mean drift | Reading | Action | |--