experiment-designlisted

# Experiment Design — playbook Experiments produce useful answers only when they are honestly designed. Underpowered tests produce noise; peeking produces false positives; guardrail-blind tests produce shipped regressions. ## 1. Hypothesis — one sentence with a mechanism `If we do <change>, <primary metric> will move by <effect size> because <mechanism>.` - The metric is real and pre-defined (it is in the metrics plan). - The effect size is the MDE — the minimum effect that would matter for the business. Smaller MDE = bigger sample required. - The mechanism is why this should work. If you cannot articulate the mechanism, the result will not generalise. ## 2. Test types — pick by what you can randomize - **A/B test** — randomize users into control vs treatment. Default for feature changes. - **Multivariate** — multiple variants at once (A/B/C/D). Beware: inflates multiple-comparison risk; correct alpha or use one decision metric. - **Holdout** — keep a slice of users in pure control long-term, even after launch. Useful for measuring cumulative effect. - **Switchback** — for marketplace / two-sided contexts where user-level randomization leaks. Alternate the treatment in time windows by region or segment. If randomization is impossible (a regulator change, an architecture change that touches everyone), do not pretend — declare it observational and label the result co-movement, not effect. ## 3. Sample size & statistical power A test that cannot detect the