Talking at Strata London Oct 1st on discovery-driven design in social apps

Hands up if you spot a theme in my recent posts.   (Hint:  they’re all about the fact I’m giving a talk)   Think I need to add a new category.  And take hold of my document model and its views, so this stuff is accessible but not mixed up with my think pieces.  Some day, over the rainbow.  If the rain ever stops.

Anyhow, idle threats aside, I’m talking about discovery-driven design in social apps in at Strata EU.

If there’s stuff you think is interesting – send it my way.  I always attribute, name-check, etc.

Fun with funnel analysis: ACB testing

I’ve recently been working on a project  helping a games company define objectives and implement  methods for analysing player behaviour patterns in a  social game.    As part of this work, I got stuck into designing some funnels.   [If you're not familiar with funnels,  MixPanel's explanation is not a bad place to start. ]

Ever-so-briefly:  funnels enable you to define event sequences that occur in the world you are trying to understand,  and measure what proportion of users make it through the successive stages of the sequence.       Funnels are often used to look at drop-out rates for important ‘straight line’ processes such as check-out, where a user’s failure to complete all the steps of the process is most probably  your fault (and is probably at least partially fixable). Nice.  But you can use funnels in other ways, too.

I’ve gotten into using funnels to compare paths in a user journey.

You take the high road and I’ll take the low, and I’ll be in Scotland afore ye.

Which is the most successful route to Scotland?  To assess this you set up more than one funnel, and you compare them.  I call this ACB testing.

What is ACB testing?

ACB testing uses funnel comparisons to look at how the  route a user takes through an app affects the likelihood of reaching a particular goal.

I’m sure you can think of lots of cases when this is an interesting question, but here are some examples to get you started:

  • Does levelling up affect a player’s likelihood of returning to a game?
  • Does interacting with other users via chat affect the likelihood of buying virtual goods?
  • Does looking at a review for a product affect the likelihood of purchase?

ACB testing can help you get a grip on these questions.   What you do, is compare performance on a simple AB funnel with performance on a funnel that includes an intermediate step, let’s call it C.   This can give you insights into whether engaging in activity C promotes reaching B, or inhibits it.

But (watch closely now) there’s a trick to it.  You can’t directly compare results from the AB funnel to results from the ACB funnel, and sit back with a nice cup of coffee.   This is because, as shown in the illustration below,  the AB funnel includes all the user data from the ACB funnel, and if you did the obvious thing, you’d end up double counting people who go to step C en route to B.

Comparing paths using funnels

Let’s say that C is engaging in online chat, during a visit to a web site.   You want to know if chatting influences the likelihood of buying, B.  What you want is to compare how likely chatters and non-chatters are to buy.  You want to compare ACB with A~CB (the ~ means ‘not’, so ~C means ‘not C’).

But you can’t directly measure something which doesn’t happen, using most funnel implementations.   They can’t tell you when the dog didn’t bark.  So how do you do it?  It’s easy.   You just need to compute the “~C” results by looking at the difference between the AB funnel, and the ACB funnel, as shown in the table below.   (Before you get excited and start implementing chat functions left right and centre, I do need to point out that this is not real data – I’m just using it to explain how ACB testing works.)

Step AB ACB A~CB
completed Funnel Funnel by subtraction
A 100 100 100
C (n.a.) 20 80
B 50 20 30
Percent buying 50 100 30

In our made-up example, when you compare the traffic on the path taken by people who chatted (which you get from the ACB funnel), with the traffic on the path taken by people who didn’t chat (which you get by looking at the difference between the AB funnel and the ACB funnel), you see that people who chat are way more likely to buy.

How exactly did I compute the A~CB path?

The different funnels, AB and ACB,  are just different ways of looking at the same information.   So you know that if 100 people start the process with Step A, and of these 100, 20 people chatted, you can confidently infer that 80 people didn‘t chat.   Similarly,  if you know that of the 50 people who performed Step A and completed Step B, 20 of them chatted en route, you can confidently infer that 30 of them didn‘t chat en route.

This enables to to assess whether the activity of chatting can predict buying.  In our made-up example, it can.  And you can use funnels to try to trap this effect so you can look at it.  Which is pretty cool.

How is this different from AB testing?

With AB testing, you put people into two different treatment groups and see what happens. You observe the consequence of these different experiences for your downstream measure of interest. If you see a significant difference between groups, and you have assigned people to groups randomly, you can be pretty sure that it’s because of what that group’s experience was.

ACB testing is different.  With ACB testing, people put themselves into groups by their behaviour. If you see a difference it outcome, it could be due to some other factor which is responsible for people’s self-segregating into behavioural groups.   It’s different – but it’s still interesting.

Are funnels the best way to do ACB testing?

Funnels are not actually the most direct way of assessing the differences in paths taken.   The best way to do that is to parse the event stream yourself, and tally it up.  However, since funnels are commonly used in Web Analytics, and are easily available on a lot of fab (and free) analytics platforms, I thought that others might be interested in this way of using them that’s a bit different from their original purpose. It’s a new way of using them to funnel off value, if you like.   Try it in your sandbox and see what you find out.

Measurement and analytics for social apps (and elephants)

State fruit - Tomato

Image via Wikipedia

Had a blast at last week’s London Facebook Developer Garage, which I  helped to organise.  Listened to lots of thought-provoking speakers,  including case studies from European social game studios Sharkius, and Weka Entertainment.   Kontagent presented from Seattle, via Skype –  and the connection didn’t flake out once.

I gave a talk, too, test-piloting a framework I’ve been thinking about for understanding measurement and analytics for social applications.   Invited the audience to throw tomatoes at me but fortunately none of them had come prepared.

Loads of fun (oooh, how I love giving talks!!), with only one big WTF moment –  my .ppt slides, so lovingly prepared in Office 2002, went all shy and invisible when viewed in a modern version on the Official Laptop Which Ran All Presentations.    (Perhaps the audience were too busy wondering what I was on about to throw projectiles…!)

Anyhow, here’s the good news.    The slides from my talk, Measurement and Analytics for Social Apps – Understanding your Elephant are now up on slideshare.

Fun fact: Slideshare is better at converting .ppt than Microsoft is,  so even if you are keeping up with the upgrade treadmill, you should be able to see them.

Enjoy.   Best viewed with feta, red onion, and olive oil.