Digging deeper into the ups and downs of comparative data

 

Analytics on phone

According to Wikipedia, ‘Simpson’s paradox, or the Yule–Simpson effect, is a paradox in probability and statistics, in which a trend that appears in different groups of data disappears when these groups are combined.’

A clear example can be found in an example from a website’s Google Analtyics reports, where we compare the success rates of two products. Consider these stats:

Product A Product B
Product page views 500 500
Purchase confirmation page views 190 212
Conversion rate 38% 42%

This simple table shows that Product B is the more successful product with 212 sales for every 500 people viewing product B on a web page. If you read this summary, as a manager or director, you may be inclined to push product B further in order to increase overall revenue. However, if you drill down into the stats a bit further, they can often lead to a completely different conclusion.

The table below shows the success rates of the same two products, this time revealing they are actually both displayed on two different pages.

Product A Product B
Product page views (page A) 127 316
Purchase confirmation page views (page A) 68 161
54% 51%
Product page views (page B) 373 184
Purchase confirmation page views (page B) 122 51
33% 28%

And combining the two:

Product A Product B
Product page views (combined) 500 500
Purchase confirmation page views (combined) 190 212
Conversion rate 38% 42%

As you can see, the results are far different, bringing a paradoxical conclusion. Product A has a better conversion rate when seen individually on both page A and page B, yet product B has a better conversion rate when you consider both pages together.

Such deeper results may make you reconsider which product is the better one. It can also open up whole new strategy options in terms of trying different products on different pages and monitoring the results. In this example, it’s not that easy to decide which is the better product but it definitely shows that you shouldn’t take important web page analysis at face value. To dig down further into the data and develop a better and more detailed understanding is crucial to getting the best out of analytics. There can be some interesting facts and figures lurking beneath, sometimes good, sometimes bad. Being skeptical is important, dig as deep as possible into the data, making sure the data is trustworthy and understanding how the results are linked to your website, will allow you to make the right changes in the long run.

Andy Jones
Andy is co-founder and lead developer at Arcimedia Web Agency. He is based in Lichfield, UK.

Leave a Reply

After you have typed in some text, hit ENTER to start searching...