Fun with funnel analysis: ACB testing

I’ve recently been working on a project  helping a games company define objectives and implement  methods for analysing player behaviour patterns in a  social game.    As part of this work, I got stuck into designing some funnels.   [If you’re not familiar with funnels,  MixPanel’s explanation is not a bad place to start. ]

Ever-so-briefly:  funnels enable you to define event sequences that occur in the world you are trying to understand,  and measure what proportion of users make it through the successive stages of the sequence.       Funnels are often used to look at drop-out rates for important ‘straight line’ processes such as check-out, where a user’s failure to complete all the steps of the process is most probably  your fault (and is probably at least partially fixable). Nice.  But you can use funnels in other ways, too.

I’ve gotten into using funnels to compare paths in a user journey.

You take the high road and I’ll take the low, and I’ll be in Scotland afore ye.

Which is the most successful route to Scotland?  To assess this you set up more than one funnel, and you compare them.  I call this ACB testing.

What is ACB testing?

ACB testing uses funnel comparisons to look at how the  route a user takes through an app affects the likelihood of reaching a particular goal.

I’m sure you can think of lots of cases when this is an interesting question, but here are some examples to get you started:

  • Does levelling up affect a player’s likelihood of returning to a game?
  • Does interacting with other users via chat affect the likelihood of buying virtual goods?
  • Does looking at a review for a product affect the likelihood of purchase?

ACB testing can help you get a grip on these questions.   What you do, is compare performance on a simple AB funnel with performance on a funnel that includes an intermediate step, let’s call it C.   This can give you insights into whether engaging in activity C promotes reaching B, or inhibits it.

But (watch closely now) there’s a trick to it.  You can’t directly compare results from the AB funnel to results from the ACB funnel, and sit back with a nice cup of coffee.   This is because, as shown in the illustration below,  the AB funnel includes all the user data from the ACB funnel, and if you did the obvious thing, you’d end up double counting people who go to step C en route to B.

Comparing paths using funnels

Let’s say that C is engaging in online chat, during a visit to a web site.   You want to know if chatting influences the likelihood of buying, B.  What you want is to compare how likely chatters and non-chatters are to buy.  You want to compare ACB with A~CB (the ~ means ‘not’, so ~C means ‘not C’).

But you can’t directly measure something which doesn’t happen, using most funnel implementations.   They can’t tell you when the dog didn’t bark.  So how do you do it?  It’s easy.   You just need to compute the “~C” results by looking at the difference between the AB funnel, and the ACB funnel, as shown in the table below.   (Before you get excited and start implementing chat functions left right and centre, I do need to point out that this is not real data – I’m just using it to explain how ACB testing works.)

completed Funnel Funnel by subtraction
A 100 100 100
C (n.a.) 20 80
B 50 20 30
Percent buying 50 100 30

In our made-up example, when you compare the traffic on the path taken by people who chatted (which you get from the ACB funnel), with the traffic on the path taken by people who didn’t chat (which you get by looking at the difference between the AB funnel and the ACB funnel), you see that people who chat are way more likely to buy.

How exactly did I compute the A~CB path?

The different funnels, AB and ACB,  are just different ways of looking at the same information.   So you know that if 100 people start the process with Step A, and of these 100, 20 people chatted, you can confidently infer that 80 people didn‘t chat.   Similarly,  if you know that of the 50 people who performed Step A and completed Step B, 20 of them chatted en route, you can confidently infer that 30 of them didn‘t chat en route.

This enables to to assess whether the activity of chatting can predict buying.  In our made-up example, it can.  And you can use funnels to try to trap this effect so you can look at it.  Which is pretty cool.

How is this different from AB testing?

With AB testing, you put people into two different treatment groups and see what happens. You observe the consequence of these different experiences for your downstream measure of interest. If you see a significant difference between groups, and you have assigned people to groups randomly, you can be pretty sure that it’s because of what that group’s experience was.

ACB testing is different.  With ACB testing, people put themselves into groups by their behaviour. If you see a difference it outcome, it could be due to some other factor which is responsible for people’s self-segregating into behavioural groups.   It’s different – but it’s still interesting.

Are funnels the best way to do ACB testing?

Funnels are not actually the most direct way of assessing the differences in paths taken.   The best way to do that is to parse the event stream yourself, and tally it up.  However, since funnels are commonly used in Web Analytics, and are easily available on a lot of fab (and free) analytics platforms, I thought that others might be interested in this way of using them that’s a bit different from their original purpose. It’s a new way of using them to funnel off value, if you like.   Try it in your sandbox and see what you find out.

5 thoughts on “Fun with funnel analysis: ACB testing

  1. I believe there is a slight error in your math in your ACB example. The percentage of people purchasing, but not chatting, should be 30/80 or 37.5% not the 30% you stated. I think you used 30/100 the total population in error.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s