ab-test-analyzer

# A/B Test Analyzer ## Overview This skill helps design and analyze controlled online experiments (A/B and A/B/n tests) with statistical rigor and honest interpretation. It covers the full lifecycle: hypothesis framing, sample-size and duration planning, choosing the right test, computing p-values and confidence intervals, and avoiding the traps that produce false wins. Keywords: A/B test, split test, experiment, hypothesis, sample size, power, MDE, minimum detectable effect, statistical significance, p-value, confidence interval, conversion rate, lift, two-proportion z-test, t-test, chi-square, peeking, multiple comparisons, Simpson's paradox, SRM, novelty effect. Use this skill whenever someone wants to plan an experiment, decide if a result is real, or sanity-check an analysis someone else did. ## Core Mental Model - An A/B test estimates a **causal effect** by randomizing units (usually users) into control and treatment. - You are testing a **null hypothesis** (no difference) against an **alternative** (there is a difference). A p-value is `P(data this extreme or more | null is true)` — NOT the probability the null is true, and NOT the probability your variant is better. - Two error types: **Type I** (false positive, rate = alpha, typically 0.05) and **Type II** (false negative, rate = beta; **power** = 1 - beta, typically 0.80). - **You must fix sample size and test duration BEFORE you start.** Stopping when significant ("peeking") inflates the false-positive rate dr...