ab-test-auditlisted

Audit an existing A/B test plan, running experiment or mockup pair for methodological flaws. Use when the user says "review my experiment", "is this test set up correctly", "what is wrong with this test", "check my A/B test", "is my test valid", "did I set this up right", "why did my test fail", "does this test have a confound", "test planımı denetle", "bu test doğru mu kurulmuş", "testimde sorun var mı", or shares variant designs, a test brief or a running experiment asking what is wrong. Checks confounds and multi-variable changes, missing or wrong primary metric, absent guardrails, p-hacking and peeking risk, sample ratio mismatch, selective attrition, novelty effect, unrealistic duration and overlapping concurrent tests. To interpret numbers from a finished test, see ab-test-results.
ali-demirbas/ab-test-playbook · ★ 0 · Testing & QA · score 73

Install: claude install-skill ali-demirbas/ab-test-playbook

# ab-test-audit — Test Plan Audit > **Language:** Output always matches the language you write in (CLAUDE.md rule 7). `${CLAUDE_PLUGIN_ROOT}/CLAUDE.md` and `${CLAUDE_PLUGIN_ROOT}/knowledge/methodology.md` are binding. ## Audit checklist Audit the shared plan/variants in this order; report every finding with its evidence: 1. **Variable isolation (most critical):** is there any difference between A and B OUTSIDE the tested element? Price, product, rating, badge, copy, ordering — any second difference is a confound. If variant visuals were shared, compare them element by element. 2. **Primary metric:** is it single and clear? If multiple metrics are being read with equal weight, flag it as p-hacking risk. 3. **Guardrail:** is a metric that could degrade while conversion rises (margin, returns, speed, support, abandonment) being watched? If not, suggest one fitting the scenario. 4. **Measurability:** can the metrics actually be measured with the tool in place? Flag unproxied "perception" metrics. Is the variant applied client-side (via JS after the page loads) or server-side? In a client-side implementation, the user can briefly see the control variant before it switches (flicker/FOUC) — this both breaks the experience and makes it ambiguous which variant that user should count toward. If unknown, flag it as an assumption that needs verifying. 5. **Sample size/duration:** is the test duration realistic given the traffic volume? Warn if the plan is shorter than two full weeks