How would you measure the success of a new AI customer support feature?
Model answer
Model answerI'd separate user success metrics from business success metrics. For users: containment rate (did the AI resolve it without human escalation?), CSAT post-interaction, and repeat contact rate. For the business: cost per resolution, human agent deflection rate, and time to resolution. The tension: optimizing for containment can hurt CSAT if the AI is forcing resolution when users need a human. I'd set guardrails: CSAT cannot drop below X even if containment improves.
Framework · Dual Metrics Model
- UUser metrics
Containment rate, CSAT, repeat contact rate, resolution satisfaction - BBusiness metrics
Cost per resolution, deflection rate, agent time saved, SLA compliance - GGuardrails
Set minimum CSAT threshold — business gains can't come at user expense - TTime dimension
Short-term containment vs long-term trust — measure both
Related questions
- Your product's DAU dropped 20% overnight. Walk me through how you'd investigate.
- How would you set up an A/B test for a new checkout flow?
- How would you define the north star metric for a food delivery app?
- How do you handle conflicting signals between quantitative and qualitative data?
- Our activation rate is 40%. Is that good or bad, and what would you do?