ab-testinglisted

# A/B Testing ## Experiment Design ### Hypothesis Template ``` We believe that [change] will cause [metric] to [increase/decrease] because [reasoning based on data/insight]. We'll know this is true when we see [statistical significance at X% confidence]. ``` Example: > "We believe that adding customer logos above the signup CTA will increase trial signups by ≥10% because users cite trust as their #1 objection in exit surveys. We'll know this is true when we see 95% confidence with ≥500 conversions per variant." ### Variable Isolation Rule Test ONE change at a time per experiment. Multi-variable tests need multivariate setup (MVT) with much larger sample sizes. ### Control vs Treatment | | Control | Treatment | |--|---------|-----------| | Definition | Current experience | Modified experience | | Traffic split | 50% (typical) | 50% (typical) | | Changes | None | One specific change | ## Statistical Significance ### Key Concepts - **Significance level (α)**: Probability of false positive. Standard: α = 0.05 (5%) - **Confidence**: 1 - α = 95% confidence - **Power (1-β)**: Probability of detecting a real effect. Target: 80% - **p-value**: Probability that result is due to chance. Need p < 0.05 to call a winner - **MDE (Minimum Detectable Effect)**: Smallest improvement worth detecting ### Sample Size Calculator (Python) ```python import math def sample_size_per_variant(baseline_cr, mde_relative, alpha=0.05, power=0.80): """ baseline_cr: current conversion rate (e