Subscription products are where sloppy A/B testing does the most financial damage, because the metric that matters — retained revenue — arrives months after the metric that's convenient, which is click-through. I spent years running experimentation on subscription funnels at CaaStle across a $30M–$50M ARR portfolio. The program produced $2.1M in ARR savings and 20% incremental revenue growth, and the biggest lesson was counterintuitive: the wins came from running fewer, better-designed tests, not more of them.
The rules that made the number real
- Measure cohorts through a billing cycle. A variant that lifts trial starts but degrades month-two retention is a loss wearing a win's clothing. Every subscription test we trusted tracked its cohort through at least one full renewal.
- Pre-register the decision, not just the metric. Before launch: primary metric, minimum effect worth acting on, run time, and what we'll do in each outcome. Post-hoc rationalization is how testing programs become fiction.
- No peeking promotions. Checking significance daily and stopping at the first p<0.05 is the most common way teams manufacture false wins. Fixed horizons or proper sequential methods — nothing in between.
- Test where the revenue is. Teams burn quarters testing button colors on the homepage while the cancellation flow — where a well-designed pause option can save a meaningful share of would-be churn — goes untested for years. The highest-ROI tests in subscription are in payment recovery, plan-change, and cancel flows.
What I do for subscription teams
Two engagement shapes. An experimentation audit (two weeks): I review your past twelve months of tests for validity — measurement windows, peeking, sample math — and it's common to find that a third of "wins" don't survive scrutiny; you leave with a corrected playbook and a ranked test backlog. Or fractional ownership of your growth experimentation for a quarter or two, running the program hands-on while training your team to keep the standards after I leave.