KPIs for AI products: prompt success beats MAU
Marc GasserSoftware Entrepreneur · GTM & MarketingConnects AI with revenue operations and builds autonomous GTM systems for predictable growth.
TL;DR
- Classic SaaS metrics measure whether a feature is used – not whether it helps. For probabilistic features that is exactly the gap: an assistant can have high usage and bad answers at the same time, until trust tips over.
- Four layers carry an AI measurement system: quality (acceptance and correction rates, eval scores), engagement (return, depth), system (latency, cost per answer) and business (time saved, conversion, support deflection). Any single layer lies.
- The acceptance rate – the share of AI outputs adopted without rework – is the most honest single metric: it implicitly measures what thumbs buttons only collect with bias.
Key findings
- Explicit feedback (thumbs up/down) typically has single-digit response rates and strong bias – implicit signals such as accept, edit, retry and discard cover nearly every interaction.
- Google's HEART framework (Rodden, Hutchinson, Fu) remains the best starting point for UX metrics – extended for AI by the quality layer it didn't need back in 2010.
- Cost per accepted answer is the unit-economics metric of the AI era: it links inference cost to quality and exposes features that only look cheap because nobody uses their output.
Why MAU lies about AI features
An example that repeats across teams: the new AI assistant shows impressive usage curves – and six months later everything collapses. What happened: users tried it, got mediocre answers, quietly corrected them and eventually gave up. MAU reported success the whole time, because curiosity looks like value. For deterministic features, usage is a usable value proxy. For probabilistic ones it is merely the entry ticket.
The reason is in the nature of the thing: a search function that works, works the same for everyone. An LLM feature delivers a different quality to every user – depending on input, context and the model's form on the day. AI measurement therefore needs a layer classic product analytics never had: how good was the answer? And no off-the-shelf dashboard answers that – only a deliberately built feedback architecture does, a core task of the operate phase.
The four layers of an AI measurement system
Quality. Acceptance rate (adopted without edit), correction and retry rates, eval score on the fixed test set. This is the layer that decides the feature's life or death – and the only one that spots drift early.
Engagement. Return after 7 and 30 days, depth per session, share of workflows involving AI. HEART (happiness, engagement, adoption, retention, task success) helps as a grid here – for AI, retention is the most honest engagement signal.
System. Latency percentiles, error and timeout rates, cost per answer. Invisible to marketing, decisive for survival: a brilliant feature with eight-second latency doesn't get adopted; one with exploding token costs doesn't get renewed.
Business. Time saved per task, support deflection, conversion and upgrade impact. The layer C-level and customers want to see – credible only when the quality layer beneath it holds.
Health score: how healthy is your AI feature?
It works, but with headroom. Typical next step: tighten the feedback loop and deliberately lift the weakest of the three numbers.
Set the three sliders to your real values – the weighted score shows whether your feature delivers or merely runs.
Building feedback loops that don't lie
Thumbs buttons are better than nothing – but measure only them and you mostly hear the extremes: the delighted and the annoyed click, the middle stays silent. The load-bearing architecture relies on implicit signals that occur anyway: was the suggestion adopted? How heavily edited? Was there a retry? Was the result discarded? These signals cover practically every interaction and correlate directly with utility.
What matters is the connection to the operate phase: the same signals that feed the health score are your drift early-warning system and your raw material for better eval sets. A well-instrumented AI feature measures itself – the PM only has to set thresholds and respond to trends instead of anecdotes.
Frequently asked questions
Which KPIs do AI products need?
Four layers: quality (acceptance rate, correction and retry rates, eval scores), engagement (return after 7 and 30 days, depth of use), system (latency, cost per answer) and business (time saved, conversion, support deflection). Each layer alone tells a pleasant, wrong story.
Why isn't MAU enough for AI features?
MAU measures usage, not utility. For probabilistic features, MAU only sees a dying feature once trust has already tipped – on the curve, curiosity looks like value.
What is the acceptance rate?
The share of AI outputs adopted without rework. It is implicit, covers nearly every interaction and is the most honest single metric for an AI feature's quality.
Recommendations
- Make the acceptance rate your north-star quality metric. Adopted without rework – per feature, weekly, with a threshold. It is implicit, comprehensive and hard to embellish.
- Always report all four layers. One AI dashboard with quality, engagement, system and business side by side. Each layer alone tells a pleasant, wrong story.
- Instrument implicitly first. Accept, edit, retry, discard – before any thumbs button. Explicit feedback is the supplement, not the foundation.
- Calculate cost per accepted answer. Inference cost divided by accepted outputs. This one number links model quality, utility and margin – and belongs in every quarterly review.
Scope & caveats
- Acceptance rates are use-case-dependent: for creative tasks, editing is normal and not a quality defect. Calibrate thresholds per feature type instead of chasing one universal number.
- The health score in the widget is a thinking tool with illustrative weights, not a benchmark. The right weighting depends on your use case – a support bot weighs differently from a code assistant.
The takeaway
AI features die quietly – of corrections nobody measures. Run quality, engagement, system and business as four layers, make the acceptance rate your lead currency, and you see problems weeks before churn, steering on evidence instead of anecdotes.
Keep reading in the PM Lab
Related deep dives – from the same pillar and the adjacent phases.
Matching use cases from the library
From the article straight into practice: these use cases put the concepts to work with Teklens.
No new piece without you.
New articles, new interactive tools, new evidence – in your inbox first. And when you reply, we reply: you write directly with the authors, not with a no-reply.
No spam, no sharing, unsubscribe any time.