ab-test-readout

# A/B Test Readout Skill The hard part of an experiment is the readout: not "B won" but "is this real, is it big enough to matter, and should we ship?" This skill turns results into an honest decision — and flags the ways A/B results lie. ## Working from a brief Given results (even partial), **write the full readout anyway**. If significance isn't provided, reason about it from the numbers and flag what's needed to confirm. Mark assumed figures. Never declare a winner without addressing significance and sample. ## Required Inputs Ask for (if not already provided): - **The hypothesis** and the **primary metric** - **Results** — control vs variant: conversions/rate, sample size per arm, duration - **Guardrail metrics** (revenue, retention, latency, complaints) that mustn't regress - **Pre-registered decision rule** (what would count as a win) if one exists ## Output Format ### 1. Verdict (one line) *Ship / Don't ship / Inconclusive — keep running* — with the headline number. ### 2. The result | Metric | Control | Variant | Relative lift | Significant? | |---|---|---|---|---| | Primary | | | | p / CI | | Guardrail(s) | | | | | State **statistical** significance (p-value / confidence interval) *and* **practical** significance (is the lift big enough to matter given the cost?). ### 3. Did it really win? Address the ways A/B tests mislead: - **Sample / power** — was the test adequately powered, or under-sampled? - **Peeking** — was the call made early, inflating false po...