ab-test-resultslisted

Interpret A/B test results and run the statistics on real numbers. Use when the user pastes visitor and conversion counts per variant, or asks "is this significant", "interpret these results", "did my test win", "which variant won", "calculate statistical significance", "what is the p-value", "confidence interval", "how many visitors do I need", "what sample size do I need", "how long should I run this test", "minimum detectable effect", "is my traffic split off", "sample ratio mismatch", "SRM", "sonuçları yorumla", "test bitti ne çıktı", "anlamlı mı", "kaç ziyaretçi lazım", "örneklem hesapla". Runs a real two-proportion z-test, confidence interval, required sample size, revenue and margin check, and an SRM check through scripts/analyze_results.py — the math is computed, never estimated — then states the decision and what happens next. To check whether the test was set up correctly in the first place, see ab-test-audit.
ali-demirbas/ab-test-playbook · ★ 0 · Web & Frontend · score 73

Install: claude install-skill ali-demirbas/ab-test-playbook

# ab-test-results — Result Interpretation and Sample-Size Math > **Language:** Output always matches the language you write in (CLAUDE.md rule 7). `${CLAUDE_PLUGIN_ROOT}/CLAUDE.md` and `${CLAUDE_PLUGIN_ROOT}/knowledge/methodology.md` are binding. Calculations are done with `${CLAUDE_PLUGIN_ROOT}/scripts/analyze_results.py` — significance and the p-value are never computed by hand or estimated, the script is run. ## Two modes ### A) Interpreting results (test finished or still running) 1. Get the control and variant's visitor + conversion counts. Ask if missing; if a rate was given without visitor counts (e.g. "5% in control, 6% in variant"), ask for the absolute numbers too — a confidence interval can't be computed from a rate alone. 2. Run: ``` python3 ${CLAUDE_PLUGIN_ROOT}/scripts/analyze_results.py significance \ --control-visitors <n> --control-conversions <n> \ --variant-visitors <n> --variant-conversions <n> ``` 3. Don't show the raw JSON output; interpret it through the `methodology.md` lens: - If `normal_approx_valid: false` comes back, **don't interpret anything else**: the z-test doesn't apply to this test (a rare-event case), the p-value and confidence interval aren't reliable. Don't declare a winner/loser; say more data needs to be collected, or a method suited to rare events should be used. This holds even if the sample is large. - If `is_significant: false` comes back, **don't just say "lost" on its own**. Check for a `low_sample_wa