gauntletlisted

# Gauntlet A skill is a claim: "the model does better with me loaded." Nobody gets to assert that — they prove it. Reading a skill tells you what it intends; only a controlled run tells you what it does. Gauntlet found a polished skill scoring WORSE than no skill at all, and found the mechanism (its evidence pressure manufactured fabrications). That class of defect is invisible to review and fatal in use. ## The Law 1. **Control arm or it didn't happen.** Every experiment includes a no-skill arm on the same task. A skill is only good if it beats the model without it. 2. **Blind judging, shuffled labels.** The judge never knows which output used a skill, which version, or that skills are involved. Shuffle label order between rounds so position bias can't repeat. 3. **The skill under test never sees the rubric.** Fixtures and ground truth live outside anything the tested arm reads. **And the fixture must be SYNTHETIC — arms must not be able to read the real subject system.** `codex exec --sandbox read-only` restricts writes, not reads; `-C <dir>` sets cwd and jails nothing; `claude -p` with `bypassPermissions` reads the whole machine. So a fixture naming a real repo, service, standard, or ticket that exists on this disk is not a fixture: arms silently cross-check it against reality, and the judge then scores TRUE findings as fabrications because the rubric assumed they could not know. Verified 2026-08-12 — a fake PR about a real service pr