If you want to learn more about A/B testing, and are in New York. I’m teaching at General Assembly on June 12. Register with .

A/B testing is a useful tool to determine which page layout or copy works best to drive users to reach a given goal. Companies like 37 signals use A/B testing to improve conversion rates on the site, HubSpot uses it to increase email conversions, and Zynga uses A/B testing to increase engagement in its games [1,2,3].Whenever you run an A/B test you must decide when you have gathered enough data, and you can pick the winning idea and implement it for all users. People typically plug the conversion numbers into an online calculator*, and if the result is ‘significant’ they pick the winner.

However deciding when to stop a test using significance is wrong.

Significance testing is useful when the goal is inference. If we want to make falsifiable statements, and draw conclusions through experimentation, then we use statistical significance to measure certainty. However in the business setting, we want to make a decision with some other goal in mind: increasing conversions, improving ease of use, maximizing profit, or some other objective. In these cases there are better criteria to determine an experiment stopping point.

Screenshot from our Optimizely experiment.

Suppose we are running a test with two ideas, call them A and B, and one idea is better than the other. The longer that we run the test, the better we are able to quantify how much better A is than B. However, the longer we run the test the more users that we expose to the inferior idea.

In a classical testing environment we would decide that we want to be able to detect differences in conversions as 1 percentage point, we would pick a confidence level (say 95%), find the appropriate sample size and run the test. At the end of the test we could say either, A is better than B, B is better than A, or A and B are within 1 percentage point, and we would be certain with 95% confidence that the statement we made was correct.

P. Armitage led the frequentist school in developing methodologies for repeated significance tests.

However using a Bayesian approach, we first determine how many people will be exposed to the result of the test. We need to balance cost of the test, against the cost of making the wrong decision. The more users that will be exposed to the result, the higher the cost of making the wrong decision, so we can justify running a longer test.

This cost is formally called ‘regret’, and is measured as the difference between the realized actualized, as compared to the optimal revenue that could be actualized if we had perfect information.

These two approaches have been debated in the clinical trial literature. In clinical trials scientists must balance providing a potentially inferior treatment to patients, against the learnings that they gain to help future patients. The Bayesian approach was developed by Anscombe in the 60s, and is widely used in clinical trials today [5].

Anscombe provides a formula to determine the stopping point of an experiment. The experiment should be terminated when the following condition is true.

Where y is the difference between results of A and B, k is the expected number of future users who will be exposed to a result, and n is the number of users who are exposed to the test so far. And  Phi-inverse is the quantile function of the standard normal.

So what does this mean? How do using the results from the Anscombe paper affect the actual performance.

In the following example we simulate 100,000 visits to the site, with two ideas, idea A has a 21% conversion rate, and idea B has a 20% conversion rate. We evaluate how the two ideas perform using a significance test, compared to Anscombe’s stopping rule, and compared to picking a fixed sample size of 10K.

Then we simulate 10,000 different iterations and calculate the regret.

Below are two plots of a typical path of an experiment. We plot the advantage of A over B. While A, in the long run, is better than B, there are short spells where B performs better than A. We also see a visual representation of the two confidence intervals.  Zooming in on the first 2,000 visitors, we see the problem with the repeated significance testing. A short run of conversions on idea B results in B being declared the winner around 100 visitors into the test.

Using Anscombe’s stopping rule is much better than using significance testing.  With a 40% less regret than using repeated significance testing.  The traditional way of using repeated significance testing leads to higher regret, and produces the correct answer less often than Anscombe’s method or using a fixed sample size.

In conclusion, using Anscombe’s methods minimizes regret, at the cost of giving up the ability to make inference. If you aim to make inferences about which ideas work best, you should pick a sample size prior to the experiment and run the experiment until the sample size is reached. But if you want to maximize conversions, use Anscombe’s rule.

*There are a lot of A/B testing calculators out there. If you do decide to pick your sample size in advance. I personally like ABBA [6]. Since it uses Aggresti-Coull confidence intervals and performs mulitple test correction, which avoid two other pitfalls not covered in this article.

Interested in learning more about A/B testing? I’m teaching a tutorial on how to design and analyze online experiments. You can sign up here.

Two weeks into the month-long campaign the Pebble has become the highest grossing Kickstarter project of all time. One of us finally broke down and kicked-in, and we started wondering how many will they sell when the campaign finishes.

We decided to take a stab at predicting the final number of backers. Once we got started, we thought it would be even more fun to challenge anyone else out there to come up with better prediction. (Details at the bottom of post)

Here I discuss several different ways that we can predict new product launches, that should serve as a starting point for your models. We’ll see the best method on May 18th when the project is over. Also we encourage readers to contribute guesses and send us their solutions. We’ll do a follow up post in May to see who gets the closest.

The Pebble project launched on Kickstarter on April 11, initially it got a lot of traction, and then got a second wave of backers on April 17-19.

One simple approach is to assume constant velocity of sales. They have 44,643 backers in 15 days, so we can project that they will sell 110,119 over 37 days.

Clearly this is not the case, as the initial wave of sales has likely already passed.

So let’s investigate some additional models. We’ll evaluate one spline based regression model which handles the problem encountered with the polynomial regression, and then look at three probability models.

We can fit a natural spline to the cumulative number of orders. The natural spline works by fitting local polynomials internally and then enforcing the constraint that the function is linear outside of the bounds. This is equivalent to extrapoaliting based on the last portion of the sales.

You can do it using the following R code

lm(cumulative~ns(day,df=3),data=pebble)

However extrapolation is a very difficult problem. We are trying to use the data that we have observed to project out into the future. Regression based approaches are often very good for interpolation, i.e. predicting a response from predictors that are within an observed range. However, regression is of limited utility in extrapolating much beyond the range of the data.

As an alternative to regression, we can use probability models. Probability models make some relatively strong, yet robust assumptions about customer level behavior, and then are able to extrapolate based on that.

Here we’ll evaluate three succesively more complicated probability models and display the results.

Exponential Gamma

In this model we assume that individual customers have an exponentially distributed interpuchase time and that these interpurchase times are distributed across the population with a gamma distribution. Breaking this down a little more. Exponential interpurchase time essentially means that a customer who has not yet bought a Pebble has a constant probability of purchasing. Now there is heterogenity across the population. Some people are more likely to make a purchase, while the opposite is true for others. This is accounted for by the gamma distribution which characterizes the spread of purchase frequencies across the customer base.

peg <- function(x,r,a){
  return(1-((a)/(a+x))^r)
}

Weibull Gamma

This is is similar to the previous distribution, but we replace the exponential distribution with a weibull distribution. A weibull distribution is similar to the exponential distribution, but it has an additional term that introduces duration dependence. Duration dependence specifies that a customer who has not made a purchase after a certain time is either more likely to make a purchase or less likely to make a purchase. Again, we assume the scale parameter of the Weibull distribution is distributed across the population with the a gamma distribution.

pwg <- function(x,r,a,c){
  1-(a/(a+x^c))^r
}

Weibull Latent Class

Here we again use the weibull distribution as our underlying probability distribution, but in this case rather than characterizing the customer base with a continous distribution we assume that there are two classes of customers. These are latent (unobserved) classes, and we infer them from the underlying distribution. Using this distribution we can start to capture the second peak in the data.

weibull.latentclass <- function(x,r,a,p, trans=F){
  if(!(length(r) == length(a) && length(p) == length(r)-1))
     stop(paste("r and a must be same length, p must be one",
                "less than number of classes"))
  classes <- 1:length(r)
  p.c <- c(p,0)
  weights <- exp(p.c)/sum(exp(p.c))
  unweighted <- sapply(classes, function(y)
                       weibull.trunc(x,r[y],a[y],trans))
  weighted <- unweighted %*% weights
  return(weighted)
}

Projections

We can visualize the daily and cumulative projections of the different models:

Of the four reasonable models, each of the models comes up with slightly different predictions for the number of backers:

Averaging these together, and we predict that the Pebble will have 55,583 backers by the time the funding closes on May 18th. Assuming that the average backing size stays constant at $147 per backer, this will yield a total amount raised of $8.19MM.

Just for fun, here are the guesses of the rest of the team:

Competition

These are some pretty basic models, and none of them fit the data exceptionally well. We could try to improve the models by aggregating data from other Kickstarter projects, or layering covariates such as media coverage of the Pebble’s success, or it’s promotion as a Kickstarter featured project. You could also come up with a more sophisticated model for how average contribution size changes over time.

Interested in making a prediction? Send your predictions to pebble@custora.com by 11:59 EST Monday April 30. The winner gets some serious street cred, a beer on us if they’re in town, and we’ll throw in a free black Pebble. We’ll also invite you to write a guest post on our blog describing your solution. To get you started, here is some code you can use to pull data from Kickstarter. Winner is the person who comes the closest to predicting the total amount raised, and who provides the methodology for how they came up with the solution (excel, matlab, R or whatever tools you want to use are fine).

#!/usr/bin/perl
use LWP::Simple;
use Date::Format;
use DateTime::Format::Strptime;
 
my $maxpage = 901
my $strp = new DateTime::Format::Strptime(pattern => "%b %d %Y",on_error=>'croak');
my $strf = new DateTime::Format::Strptime(pattern => "%Y-%m-%d",on_error=>'croak');
 
 
for (my $i = 1; $i <=$maxpage; $i++){
    my $url = "http://www.kickstarter.com/projects/597507018/pebble-e-paper-watch-for-iphone-and-android/backers?page=$i";
    my $content = get $url;
    die "Couldn't get $url" unless defined $content;
 
    while($content =~ m/<div class="date">([^<]*)/g){
        $date=$1;
        if($date =~ m/^\d/){
            my @dt = localtime(time);
 
            print strftime("%Y-%m-%d\n",@dt);
        }
        else{
            my $formatted =  $strp->parse_datetime("$date 2012");
            print $strf->format_datetime($formatted)."\n";
        }
    }
}

Methodologies: Data was pulled from the Kickstarter to find the number of backers at a given day. We fit the probability models using truncated models and using maximum likelihood estimation for the difference of CDFs.

Further Reading

Morrison, Donald G. and David C. Schmittlein (1980), “Jobs, Strikes, and Wars: Probability Models for Duration,” Organizational Behavior and Human Performance, 25 (April), 224–251.

It’s no secret that mobile commerce is exploding.  As a result, many of our customers have been using Custora to understand the shopping behavior of their mobile customers.  Looking beyond aggregate metrics, we’ve been surprised to see just how much “the mobile shopper” varies across clients and verticals.  We put together the following infographic to highlight some of the stories we’ve seen (click to view in full size):

If you’re curious to understand the customer lifetime value and ordering behavior of your mobile shoppers, let us know!

The “layer cake” cohort visualization is one of our favorite forms of historical customer analysis.  Instead of just focusing on top-line revenue numbers, we color-code monthly revenue based on the “acquisition cohort” it belongs to.  Each color represents the revenue earned from a group, or “cohort,” of customers who joined in a given month (hat tip to Roberto Medri from Etsy who suggested this visualization to us).

The layer-cake cohort graph provides three valuable pieces of information:

1. What % of revenue comes from newer and older users.
2. How quickly a group of new customers fades away.
3. Aggregate revenue trends.

This is a layer-cake diagram of Firm A, a retailer who does not have strong customer retention.  Notice how quickly each cohort fades away.  In an average month, roughly 90% of revenue earned comes from new customers.

Firm B is a retailer with much better customer retention.  Notice how each cohort band remains strong over time.  In this graph, roughly 70% of monthly revenue comes from new customers with 30% coming from repeat business.  The cohort bands shrink down quite a bit after their initial month because customers don’t make orders every month – but the customers clearly stick around for a while.

Firm C is a retailer with great customer retention and customers who purchase with high frequency.  Nearly 80% of monthly revenue comes from repeat business.

Taking things further

The layer-cake graph is an informative way to kick off retention analysis, but it doesn’t paint the whole picture.

If we define retention as how long customers stick around, Firm B and Firm C actually have very similar retention.  However, Firm C’s customers purchase with much greater frequency.  Drawing out retention for users who order at different frequencies can provide further insights.  For example, we might discover that retention revenue numbers are all derived from a small pocket of high-value, sticky customers.  Or, we might discover that many customers stick around, each ordering once per quarter.  If it turns out we have a pocket of all-star customers, we’d most certainly want to zoom in on those customers to learn how we can find more of them!

Finally, and perhaps most obviously, the layer-cake graphs don’t provide answers on what to do next – they don’t shed insight as to how we might improve customer retention.  To change the shape of your layer-cake graph, we need to drill beyond aggregate cohort metrics and hone in on individual-level insights.  The more we know about specific customers, the better chance we have to keep our customers engaged and happy.

We’ll write more soon on some of the most effective forms of individual-level insights we’ve seen with retention marketing – from churn detection to behavioral segmentation to customer lifetime value prediction.

If you’re interested in automating these graphs – and drilling from aggregate cohort analysis all the way down to individual-level customer insights, get started with Custora today!

At Custora we allow all the numbers on the screen to be lazy-loaded. We precache all of the main page views, but there are too many possible ways that a user can slice the data to pre-compute all of the values.

If you hit one of these pages, you see a loading bar that looks like this:

Writing code to deal with all the lazy loading can be a bit of a challenge. Basically, it means that whenever we get a number from our internal statistics api, we have to accept the possibility that it is not rendered correctly. And we use the statistic term loosely, it can be a float, an array, or some other ruby class. Anything that is computed from the raw transaction data goes through this API.

So to render the pages we make two passes. First to run through and figure out what statistics need to be calculated. We then send these all to delayed job, display a pending page that perodically polls to see if the jobs are done, and when the jobs are completed we display the rendered page.

We have an interesting technical challenge, how do we handle these uncomputed statistics. When we first started we ran into all kinds of errors. We would have an expression like:

number_with_precision(client.new_customer_value)

When client.new_customer_value was hit, the job would be enqueued, but then the page would fail to load.

To solve this we tried checking if a statistic was nil, but this made convoluted code

(client.new_customer_value ? number_with_precision(client.new_customer_value) : nil)

So to solve this we made a do nothing class with the goal to fail silently as much as possible. So we have a class that is designed to fail gracefully no matter what you do with it. It should fail silently if you call

stat * 100

or

stat[1]

or

1 * stat

or even

stat[1][0].first.to_s

We could have done something like:

class NilClass
 def method_missing(*)
     return nil
 end
end

But this would have changed how nil behaves all over our application. Instead we decided to make a new class for unevaluated model statistics.

Another solution would have been to use the .try method on all statistics. But this is cumbersome, and doesn’t work when the unevaluated statistic is passed to another function.

So we came up with the unevaluated model statistic class.

Code:

class UnevaluatedModelStatistic
  def ==(_other)
    false
  end
 
  def method_missing(_method,*_args)
    self
  end
 
  def to_f
    0.0
  end
 
  def to_str
    ""
  end
 
  #for cohorts statistics
  def number_of_display_columns(_arg)
    1
  end
 
  def *(_arg)
    0
  end
 
  def +(_arg)
    0
  end
 
  def -(_arg)
    0
  end
 
  def coerce(_other)
    [self,_other]
  end
 
  def /(_arg)
    1
  end
end

On the second pass we end up with the completely rendered page:

This is the way we are attacking the problem. How would you approach it?

Jason Goldberg from Fab.com just wrote up a nice piece covering some customer lifetime value analysis we ran with the Fab team zooming in on iPad customers.

As Jason notes, the expected two-year revenue from iPad customers is over twice that of the average customer!

iPad customers also “convert” from members into paying customers much faster than the average customer.