How would you set up an A/B test for a new checkout flow?
Model answer
Model answerI'd start with the hypothesis: 'Simplifying checkout from 4 steps to 2 will increase conversion rate by X%.' Then define primary metric (conversion rate), secondary metrics (order value, time to checkout), and guardrail metrics (return rate, payment errors). I'd calculate required sample size, determine test duration accounting for day-of-week effects, split users randomly not sessions, and set a clear decision threshold before the test runs — not after.
Framework · A/B Test Design
- HHypothesis
Specific, falsifiable: 'X change will cause Y outcome measured by Z metric' - MMetrics
Primary (conversion), secondary (AOV), guardrails (errors, returns) - SSample size
Calculate before running — determines duration and confidence level - DDecision criteria
Set significance threshold (p<0.05) and minimum detectable effect before launch
Related questions
- Your product's DAU dropped 20% overnight. Walk me through how you'd investigate.
- How would you measure the success of a new AI customer support feature?
- How would you define the north star metric for a food delivery app?
- How do you handle conflicting signals between quantitative and qualitative data?
- Our activation rate is 40%. Is that good or bad, and what would you do?