ENTR

How do you measure the long-term impact of a product decision made two years ago?

Model answer

Model answerMeasuring retrospective impact is hard because you rarely have a clean control group. Best case: the decision was rolled out in phases, giving you a natural experiment. Worst case: it launched everywhere at once. In that case, I'd use pre/post trend analysis with external benchmarks as control — industry averages, competitor proxies. I'd be transparent about confidence levels: this is directional evidence, not causal proof.

Framework · Metrics from Scratch

  1. AAssumption mapping
    What did you believe would happen? Compare to what actually happened
  2. IInstrument retrospectively
    Find proxy metrics that existed before — user interviews, support volumes, NPS archives
  3. CFind a control
    Phased rollout? Different regions? Any natural control group helps
  4. RReport with confidence ranges
    Be explicit: 'We believe X caused Y, but we can't rule out Z'

Related questions

Metrics & Analysis

Case studies in this category

Practice this category