I recently read a great blog post about Cohort Analysis – measuring engagement over time from 52 weeks of UX and it got me thinking how to achieve this within Omniture SiteCatalyst.
As it turns out, with a bit of custom code, it’s pretty easy to do, and what an opportunity it opens up.
Cohort Analysis allows you to look at a group of people who start something at a specific time and monitor them over time to see whether their engagement increases or decreases. Then you can also compare them with people who start the same thing at a different point in time.
It also reminded me about Survival Analysis – which is a key type of analysis that any data-driven marketer would understand. Survival Analysis allows you to estimate the time a customer will remain a customer before they disengage. This information allows a company to introduce an appropriate loyalty offer, or incentivise the customer, just before they abandon. I’m not sure we can do a true Survival Analysis (that would be like a little holy grail – but I’m working on it) so I’ll save that for a future post.
Why you need to do this…
So back to Cohort Analysis. Say you run an awareness campaign and want to monitor engagement before, during and after, to see whether those that came to your site during the campaign were more or less engaged than those that came before or after the campaign, and what triggers disengagement.
To quote 52 weeks of UX “One reason why the cohort analysis is valuable is because it helps to separate growth metrics from engagement metrics.”
The example given in the UX blog is:
The people who joined our service in January make up the January cohort, the people who joined in February make up the February cohort, and so on. We then investigate how each cohort stays engaged over time, comparing the cohorts against each other to make sure that the people who joined in February are more engaged than those who joined in January, for example.
SiteCatalyst doesn’t natively allow you to do it – but that’s the beauty of being able to customise your implementation. You can make it do it.
So, in the following example, we’re grouping new visitors by the week (or month) that they became new visitors, and then watching them, compared to other groups, over time. We can now measure these different groups of people on things like page views, time on site, number of interactions, leads, etc, which allows us to better understand the impact of our marketing, or even the changes we make to our site, without being impacted by traffic growth.
So what do you see…
Once you’ve got it all set up and reporting, you can trend engagement over time. I think the trended engagement over time is the best use of these metrics rather than the ranking.
For example, the following shows (will show) the trended report for New Visitors Week traffic prop, with Average Time on Site as the measure.
Given that I was over-excited by this prospect, and couldn’t contain myself any more, there’s virtually no data in this (sorry, I’ll update this as we get more)…but…
The above shows average time spent on the site for visitors who started with us in week 44 was around 2 minutes. Week 45 starters spend about 7 minutes.
If we trend that over time, we’ll be able to see whether their engagement is increasing or decreasing over time. Next week, running the same report will again show the Week 45 group against the Week 44 group against the new group Week 46, as they’ll all have activity as we progress forward in time.
What you can do with this…
Look for the disengagement points or trends. After some time has passed, you’ll start to see trends in user disengagement. Those are the groups of people for whom engagement starts to slip. Then you can try to re-engage them with an offer or something.
You could also use this to monitor and react to conversion rates for different groups over time. You might see that conversion rates increase over time, as they become more familiar with your brand. You can use that in your next marketing campaign – you’ll know when to beginning serving content that will aid them to convert better, if for example, during the earlier visits they are “just browsing”.
Now it gets interesting…
Let’s assume you’ve noticed that there’s a significant change in engagement (or conversion) just before week 5 for many visitors – across all cohorts.
Use Test and Target to pick up the value of the month they started against the current month and give them a timely offer. Try to re-engage them. Test and Target, coupled with this SiteCatalyst data, can “see” the same profile information for each user and can use that to target them.
So many possibilities – just go out and explore them. Unleash the elephant!
So, now to the nitty gritty, techno mumbo jumbo of it all…
In our implementation, we’re using 2 s.props and 2 eVars. One to track the New Visitor Month value, and the other to track the New Visitor Week value. And now that we have lots of new s.props and eVars courtesy of Omniture we’re happy to use them.
We use Month as a long term indicator and obviously, Week as a bit of a shorter term indicator.
We assign every visitor a Month value and a Week value. If they don’t have a value (most likely because they are a new visitor), they get the current Month and Week value, which they keep with them in their cookiejar, presenting it back to us each time, so that we can then apply measures against those specific values.
We’ll treat everyone without a cookie as a new visitor. That way, everyone gets a cookie (ok, for the first little while, your new visitors may not technically be new visitors due to when you implement the code) but as we want to populate a cookie with the value of the current month (or week, depending how you want track it), that’s probably the best way to do it.
So, using the getAndPersistValue plugin you can set those two values into two s.props, and then set eVars from the s.props. In SiteCatalyst Admin just create your s.props and eVars like you normally would – we’ve named ours New Visitor Month and New Visitor Week. You’ll also want to turn on Pathing across the props and if you can, set Full Sub Relations on the eVars.
The getAndPersistValue plug-in obtains a value of your choosing and causes it to be populated into a SiteCatalyst variable for a determined period of time.
We’ve used the following code to figure out the current month and we’ve used s.prop34 and eVar44 for our New Visitor Month values:
//Set the NewVisitorMonth
s.prop34=s.getAndPersistValue(s.eVar44,'s_prop34',365);
if(!s.prop34&&!s.eVar44);
{
var thisdate=new Date();
var thismonth=new Array(12);
thismonth[0]="January";
thismonth[1]="February";
thismonth[2]="March";
thismonth[3]="April";
thismonth[4]="May";
thismonth[5]="June";
thismonth[6]="July";
thismonth[7]="August";
thismonth[8]="September";
thismonth[9]="October";
thismonth[10]="November";
thismonth[11]="December";
s.prop34 = thismonth[thisdate.getMonth()]
s.eVar44=s.prop34;
}
//Set persisting prop34 to value of the eVar44.
s.prop34=s.getAndPersistValue(s.eVar44,'s_prop34',365);
So what we get in the reports are the names of the months and any measures we can apply against them.
If, like us, you’re wanting to do it by week as well, you can use the following code – we’ve used s.prop35 and eVar45 for New Visitor Week. This code is a little trickier as there’s not a nice easy way to get the value of the current week…but some rummaging around via Google led me to this code:
//Set the NewVisitorWeek
s.prop35=s.getAndPersistValue(s.eVar45,'s_prop35',365);
if(!s.prop35&&!s.eVar45);
{
Date.prototype.getWeek = function() {
var onejan = new Date(this.getFullYear(),0,1);
return Math.ceil((((this - onejan) / 86400000) + onejan.getDay()+1)/7);
}
var thisdate = new Date();
s.prop35 = thisdate.getWeek();
s.eVar45 = s.prop35
}
//Set persisting prop35 to value of the eVar45.
s.prop35=s.getAndPersistValue(s.eVar45,'s_prop35',365);
So the code basically says if s.prop34 (or 35) doesn’t exist already and neither does eVar44 (or 45), then go figure out what month it is (or week) and set the s.prop to the correct value, then set the eVar to the value in the s.prop, then persist the s.prop for 365 days (yeah – good luck with that cookie sticking around). Still with me?
One of the problems of the above code (which could probably be better I know), is that it relies on the users computer to have the correct date set – and it’s amazing that some people don’t appear to have that !!!
Note that s.getAndPersistValue has three arguments:
- Currently populated variable or value to persist (s.eVar44 and eVar45 above).
- Cookie name; used to store the value (s_prop34 and s_prop35 above).
- Period of time for persistence, in days (“365” above would cause the value to be populated into the selected variable on every page view made by the given user for the next 365 days). If omitted, defaults to session which will totally negate what you’re trying to achieve.
The reason that you need to use both an s.prop and an eVar is that in SiteCatalyst, you can then see Time Spend, Page Views, Visits, Visitors and Calculated metrics against the traffic reports, and all your success events, leads, sales etc, participation measures and so forth, against your conversion reports. Just remember with your full sub relations, if you want to break down one against the other, you need full subs against both eVars you wish to see.
That’s about it really. Pretty insightful and a little bit fancy.
