Most people by now have heard of the "Product key activation event". More generally, Facebook's 7 friends in the first 10 days, Twitter's 30 followers... get lots of mentions in the Product and Growth communities. Theses examples have helped cement the idea of statistically determining goals for the onboarding of new users. A few weeks ago, somebody from the Reforge network asked how to actually define this goal and I felt compelled to dive deeper into the matter.I love this topic and while there's already been some solid answers on Quora by the likes of Uber's Andrew Chen, AppCues' Ty Magnin and while I have already written about how this overarching concept a couple weeks ago (here) I wanted to address a few additional/tactical details.Below are the three steps to identify your product's "key activation event".
This is done by plotting the impact on conversion of performing and not performing an event in the first 30 days. This is the core of the content we addressed in our previous post.To simplify, I will call "conversion" the ultimate event you are trying to optimize for. Agreeing on this metric in the first place can be a challenge of itself...
For each event, you'll want to understand what is the required occurrence threshold (aka how many occurrences maximize my chances of success without hitting diminishing returns). This is NOT done with a typical logistic regression even though many people try and believe so. I'll share a concrete example to show why.Let's look at the typical impact on conversion of performing an event Y times (or not) within the first X days:
There are 2 learnings we can extract from this analysis:- the more the event is performed, the more likely to convert the users are (Eureka right?!)- the higher the threshold of number of occurrences to perform, the closer the conversion rate of people who didn't reach it is to the average conversion rate (this is the important part)We therefore need a better way to correlate occurrences and conversion. This is where the Phi coefficient comes into play to shine!Below is a quick set of Venn diagrams to illustrate what the Phi coefficient represents:
Using the Phi coefficient, we can find the number of occurrences that maximizes the difference in outcome thus maximizing the correlation strength:
Now that we have our ideal number of occurrences within a time frame for each event, we can rank events by their highest correlation strength. This will give us for each time frame considered, the "key activation event".
Because Data Science and Machine Learning are so sexy today, everyone wants to run regression modeling. Regression analyses are simple, interesting and fun. However they lead to suboptimal results as they maximize for likelihood of the outcome rather than correlation strength.Unfortunately, this is not necessarily a native capability with most analytics solutions but you can easily dump all of your data in redshift and run an analysis to mimic this approach. Alternatively, you can create funnels in Amplitude and feed the data into a spreadsheet to run the required cross-funnel calculations. Finally you can always reach out to us.Don't be dogmatic! The results of these analyses are guidelines and it is more important to pick one metric to move otherwise you might spiral down into an analysis-paralysis state.Analysis << ActionRemember, an analysis only exists to drive action. Ensure that the events you push through the analysis are actionable (don't run this with "email opened"-type of events). You should always spend at least 10x more time on setting up the execution part of this "key activation event" than on the analysis itself. As a reminder, here are a couple "campaigns" you can derive from your analysis:
Images:- MadKudu Grader (2015)- MadKudu “Happy Path” Analysis Demo Sample
Looking at the trial-to-paid funnels by customer segment is the best way to understand how your product and sales activities affect B2B SaaS conversionsRead Story
Your ideal customer profile can be determined through analysis, which is used to prioritize leads. Want to learn more? Read on here.Read Story