eval-vs-abtest-routerlisted

Iterate-stage skill: routes a product question to offline eval or A/B test — by first classifying it as output-quality (a property of the artifact, judgeable against criteria) or user-behavior (a property of humans, observable only in the field). Use when the measurement tool is in question — 'eval this or A/B test it?', 'how do we know the new prompt is better', 'is an A/B test the right tool here', 'how do we measure if it worked' — or when /pm routes such a request here. Do NOT use to build the eval (eval-engine), to design the experiment itself, for build/kill decisions (ai-feature-go-no-go), or for eval-vs-experiment definitions.
Abhillashjadhav/PM-agent-OS · ★ 1 · AI & Automation · score 60

Install: claude install-skill Abhillashjadhav/PM-agent-OS

# Eval vs A/B Test Router Two tools, two kinds of questions. Evals judge artifacts; experiments observe humans. Route by what the question is *about* — pointing the wrong tool at a question measures something, just not the answer. ## Verification gates (defined first; output is blocked until all pass) - **G1 — Classification before tool:** every routing states the question's kind first — `OUTPUT-QUALITY` (artifact property, judgeable offline against criteria) or `USER-BEHAVIOR` (human reaction, field-only) — then the tool. A tool verdict with no classification fails. Compound questions are split, each half classified and routed, with sequencing. - **G2 — The misuse statement:** every routing says what the rejected tool would actually measure if used here ("an A/B test on accuracy measures error tolerance, not correctness") — not generic cons. The mismatch is the router's evidence. - **G3 — Field-honest arithmetic:** sample/timeline claims use provided volume numbers with the math shown, or say "depends on traffic" — no invented MDEs, significance thresholds, or user counts. Where both tools apply, the eval-first rule is stated: variants that fail quality gates don't enter experiments. ## Steps 1. **Extract the actual question(s).** "Is it better?" hides several questions; write each one out. Compound asks ("punchier, and will users like it?") get split now. 2. **Classify each:** does answering require judging the artifact against a definition of good (output-quality), or