prompt-and-grader-auditlisted

# The prompt and the grader are one artefact A real case. The prompt said "return empty rather than guess". The grader then demanded the field anyway and marked empty as wrong. The eval was punishing exactly the behaviour the deployment wanted, and scoring memorisation of a reference directory instead. Fixing the grader moved one model from 3/12 to 8/12. No model changed. No prompt changed. The score was measuring the wrong thing the entire time. ## The refusal **Do not report a score until prompt and grader have been read side by side.** If they disagree anywhere, the score is invalid, not merely low. ## What to check, line by line 1. **Abstention.** If the prompt permits "I don't know" or empty, does the grader credit it? An abstention graded as an error trains you to prefer confident wrongness. 2. **Format.** Does the grader reject an answer that is correct but shaped differently? Trailing whitespace, casing, date format, currency symbols, ordering. 3. **Under-specification.** Anything the prompt leaves open, the grader must accept every reasonable reading of. Otherwise you are scoring mind-reading. 4. **Rubric drift.** Where the grader applies a rubric, check whether the labels follow it. On priority classification I found the model applied my own stated rubric more consistently than my labels did. 5. **Schema strictness.** `additionalProperties: false` against a field the harness itself injects will score a whole workload at zero and look like a m