Customer Lifetime Value techniques: why you need to be careful with cohort analysis

This is part 2 of a 3-part series discussing techniques to calculate Customer Lifetime Value (CLV) in a retail setting.  In the first part, we explained why ARPU-based approaches often lead to inaccurate CLV numbers.  In this piece, we dive into the pros and cons of Cohort-based CLV calculation.

Cohort analysis is a simple, yet powerful, form of analysis that tracks how a group of users behaves over time.  Cohort analysis provides many valuable insights – and one of those is that it can be used to predict customer lifetime value.

Predicting CLV based on cohort analysis is not only simple – it can also be fairly accurate.  In our own data sets, 1 year value predictions derived from cohort analysis have an average margin of error around ±15%.  While that number might seem high, it’s a huge improvement over the numbers derived with ARPU.

However, we need to take caution.  There are many situations in which cohort analysis produces inaccurate CLV numbers – and the margin of error in these cases can be higher than 150%.

How to predict customer lifetime value by using Cohort Analysis

To derive CLV numbers from cohort analysis, we first need to define our cohort.  One popular approach is to define the “average cohort.” To do so, we can run simple queries to determine what a customer spends, on average, in his first month of being a customer.  We can repeat the process to determine what a customer spends, on average, in his second month, and so on.

We can then chart how the average group of customers behaves as time passes after the first purchase.

The average customer spends roughly $45 in his first month.  As time passes, many customers stop making purchases.  As a result, the average spend per month declines over time.

It’s fairly straight-forward to use our cohort graph to derive lifetime value predictions.  If we want to predict the 12-month value of a new customer, we simply sum the average revenue per month of all 12 periods in the above graph.  If we want to predict what a 7-month old group will spend in the next 5 months, we simply sum up the last 5 points in this graph.

This illustrates the major advantage that cohort analysis has over ARPU when it comes to predicting lifetime value.  ARPU treats all customer-months the same.  Cohort analysis recognizes that early months are worth more, on average, than later months – and predictions improve as a result.

The numbers: margins of error as low as 5% and higher than 150%

To gain an understanding of how accurate cohort-based CLV numbers are, we looked across various clients in different retail verticals (see note at the bottom for an explanation of how we use holdout tests to derive these numbers).

  • On average, using cohort analysis to predict the 12-month-value of a new customer had a margin of error of 15%.  Huge improvement over ARPU!
  • On average, cohort-based 24-month-value predictions had margins of error of 22%.
  • On the other hand, in one case, a  24-month prediction turned out to have a margin of error over 150%.  In another example, a 12-month prediction had a margin of error of 40%.  In fact, in the majority of client data sets that we’ve seen, there are spot cohorts where predictions had margins of error over 40%.  There’s a lot of variance here!

The moral of the story, as always, is to test your Customer Lifetime Value numbers. Cohort analysis is undoubtedly a smarter way to predict CLV when compared with ARPU – and it can produce fairly reasonable numbers.  However, there are some common situations where cohort analysis really misses the mark.

What makes cohort-based CLV misfire? 

Cohort analysis is purely historical.  When things change, accuracy suffers.  We studied a handful of clients where cohort analysis offered particularly poor lifetime value predictions.  We found a few common themes:

  • When a company is growing fast: early adapters are often very different customers than those acquired at each level of growth.  Average cohorts can be weighed up or down based on these older customers.
  • When the business itself is changing: when companies expand into new verticals or expand their product offerings, ordering behavior often changes dramatically.
  • When market dynamics are changing: when markets become more competitive or industry regulations shake up.  In one case, newer cohorts of users are worth 60% less than older cohorts due to these changes.
  • When there isn’t enough data: if your retail store is one year old, and you’re trying to predict two-year value, a simple historical technique such as cohort analysis obviously falls short.

If your business falls into one of these cases, you should take extreme caution when using cohort analysis to predict lifetime value.

Taking things further

Any approach to predict lifetime value that simply expects the past to match the future will have the issues in the situations described above.  In our next piece in the series, we’ll dig into more advanced approaches to Customer Lifetime Value.  We’ll discuss the pros and cons of some techniques that try to make sense of historical data in a way that accurately predicts what will happen in the future.

 

*To run a holdout test, we looked at data sets that had many years of history.  We temporarily chopped off the last year or two to create our “holdout.”  For example, to test a two-year prediction, we chopped off data from Feb 2010 onward.  We then make a prediction as if we were standing in Feb 2010 – only using data up until 2010, just like we would in real-time.  After making this prediction, we can look at our holdout set to see what actually happened.