How would you evaluate whether a product experiment was successful?
Model answer
Model answerSuccess evaluation starts before the experiment runs — not after. If you didn't define your primary metric and significance threshold upfront, you're vulnerable to p-hacking. Post-experiment, I check: did we hit statistical significance on the primary metric? What happened to guardrail metrics? Were there segment differences that the aggregate masked? I'd also check novelty effects — did engagement spike on day 1 and then drop? A good experiment answers the hypothesis, not just the metrics.
Framework · A/B Test Design
- HHypothesis check
Did the result confirm or reject the stated hypothesis? Be honest about both - MMetric integrity
Primary metric significance + guardrail metrics — both must pass - SSegment analysis
Did the effect concentrate in one cohort? That's the real insight - DDuration and novelty
Did engagement stabilize or was it a novelty spike? Check week-2 and week-3 data
Related questions
- Your product's DAU dropped 20% overnight. Walk me through how you'd investigate.
- How would you measure the success of a new AI customer support feature?
- How would you set up an A/B test for a new checkout flow?
- How would you define the north star metric for a food delivery app?
- How do you handle conflicting signals between quantitative and qualitative data?