Teams arrive with a Kaplan–Meier plot that looks expensive. The x-axis is days since first value event. The y-axis is “still retained.” Someone has already drawn a vertical line at day seven. The argument in the room is about the height of that line, not about what the line is allowed to mean.
On the bench we start earlier. Who is in the risk set? A user who uninstalled but still receives email is not “at risk” in the same way as a user who opens the app and does nothing. If your event stream cannot see uninstalls, the curve is silently optimistic. Say that out loud before you compare treatment and holdout.
Censoring is not a footnote
Right-censoring — people who have not had enough calendar time to fail — is ordinary. What is not ordinary is pretending a user who lost a device, shared a login, or changed phone numbers is still under observation. In Bangkok wallets we have seen family device-sharing create phantom survival: the curve stays high because a different person is generating the “return” event.
The fix is not a fancier estimator. The fix is a leakage note on the memo: how many identifiers we believe are shared, and whether that rate differs between cells. If it does, the curve comparison is not a retention finding. It is a tracking finding.
Language that survives review
We ask students to write one sentence the legal team could live with. “Treatment users remained in the active set longer through day 28, under a user-level holdout with known 6% device leakage, excluding Songkran weeks.” That sentence is ugly. It is also harder to turn into a launch blog post, which is part of the point.
What we do not allow: “the feature improved retention by X%” when the only evidence is a non-holdout before/after, or a curve that starts at a different first event than the one in the tracking plan.
When a simple empirical curve is enough
Kaplan–Meier earns its keep when follow-up times differ. If everyone in the sample has a full four weeks of observation and you already excluded incomplete weeks, a plain proportion-still-active plot can be clearer for a product audience. The theatre is not the estimator. The theatre is the refusal to name the denominator.
Module 05 of Holdout Design for Product Retention spends an evening on this reading. Bring a curve from your own warehouse. We will mark the places it overclaims.