keel-pilot

# /keel-pilot — never run a big batch blind Use when a task will produce **many outputs whose quality cannot be eyeballed one by one** — mass labeling/classification, bulk generation, large migrations/transforms (heuristic: >100 items, or any unattended run). The failure this kills: pipelines verified "sometimes, by mood" — quality problems discovered after 5,000 items, or never. The sequence below is the published norm, not an invention (Anthropic best-practices fan-out, Pangakis et al. 2023, SRE canarying, Write-Audit-Publish); evidence trail with sources: `research/*/findings.md` (2026-07-16). Five gates, in order. **A gate must PASS before the next opens** — report each as pass/fail. ## Gate 0 — DECLARE (before anything runs) Write into the task (`TASKS.md done-when:` or a short pilot plan next to the runner): - **acceptance metrics** — e.g. per-class precision/recall vs a gold set; schema-validity %; - **halt thresholds** for the full run — e.g. >5% schema-invalid, or error rate >2× the pilot's, in any rolling 100-item window → HALT; - **sample plan** — smoke size (10–50) · gold set (250+ human-labeled items when label quality is load-bearing; smaller only with the user's explicit OK) · honeypot rate (5–10%) · acceptance sample by the rule of three (0 errors in 60 ⇒ <5% true error rate, in 300 ⇒ <1%, at 95%). Thresholds chosen AFTER seeing outputs are not thresholds. Defaults above are the literature's — adjust per project, with the user. ## Gate 1 — SMOKE (10–...