A growth lead once arrived with three creative variants, a CRM audience, and no event that distinguished “completed KYC” from “opened the KYC screen.” They wanted a holdout by Friday. We declined the Friday. We spent the week renaming events. The creatives waited. The experiment, when it finally ran, had a denominator everyone could defend.
That story is not glamorous, which is why it rarely appears in conference talks. It is the majority of our first calls.
What “good enough” instrumentation looks like
You do not need a perfect tracking bible. You need a frozen definition of the return event, a stable user key, a holdout flag that CRM tools cannot silently override, and a timestamp you trust. If any one of those is missing, randomisation is decoration.
Cohort Instrumentation Studio exists because the flagship course kept spending session one on tracking-plan archaeology. We split the work. If your events already behave, skip straight to Holdout Design. If they do not, do not buy a seat hoping the memo template will invent missing columns.
Holdout flags are product surface
A boolean in the warehouse is not enough if the messaging tool builds its own audience from “last 30-day openers” and ignores the flag. We have watched treatment users receive the “control” journey because a vendor integration lagged. Instrumentation includes the tools that can violate the cell, not only the events table.
Write the violation modes on the memo. That list is part of Retention Experiment Analytics. It is not “engineering detail.”
A small checklist before you book a bench
- Can you query the proposed return event for the last ninety days without a join that still has no owner?
- Does CRM honour a holdout flag today, not in a future sprint?
- Has the event been renamed in the last quarter, and is the old name still leaking into dashboards?
If two of those answers are no, start with instrumentation. See programmes or send the studio your tracking plan excerpt.