More lies and bad analyses by Shareaholic are misleading the public

This is the original version of the article that appeared on Co.Design.

Earlier this week, Shareaholic published a report claiming that Reddit referral traffic is rapidly declining. Shortly thereafter, the report appeared on several respectable news outlets, including The Atlantic and Business Insider, chiming in and recommending that SEO experts start looking at web sites other than Reddit for referral traffic.

Normally I'd ignore a report like this because I don't care much about SEO, but the visualizations and analysis in Shareaholic's report were so bad -- and so misleading -- that I had to write a response to it.

It takes three points to make a trend

First off, Shareaholic's claim rests on their comparison of two cherry-picked data points: Reddit referral traffic at Dec '12 and Dec '13. They compare referral traffic between those two points and note that it's gone down by 33%, then wildly speculate that Reddit's referral traffic is crashing.

One of the things we learn in high school math class is that it takes at least 3 points to make a trend, not 2. In fact, if Shareaholic had posted this report in June '13, they could've told the opposite story: Reddit's referral traffic is skyrocketing (up 66%)! That's a sign right there that the methods they're using are flawed.

[caption id="attachment_2543" align="aligncenter" width="960"] Reddit referral traffic trend, Dec '12 to Jun '13[/caption]

Their visualization doesn't even show us if there's a trend

If we want to talk about trends, we need to consider all of the data. Shareaholic shows us a simple line plot connecting the dots, but doesn't even try to show us what trend is actually going on. Can you easily tell if there's really a significant downward trend going on there? That's the sign of a poor data visualization.

[caption id="attachment_2607" align="aligncenter" width="649"] Reddit referral traffic trend, Dec '12 to Dec '13. Image c/o Shareaholic[/caption]

The trend isn't even significant

When we take a closer look at the referral traffic data Shareaholic presented, we can draw a trend line through those points and notice that there isn't much of a trend going on at all: The line is almost flat, going down only 0.007% every month.

Of course, this trend line predicts that Reddit referral traffic will stay in constant decline, which we know isn't the case. Just look at what happened in June '13. There's a lot more to Reddit's referral traffic trend than meets the eye, and Shareaholic's oversimplified analysis doesn't even come close to explaining it.

[caption id="attachment_2545" align="aligncenter" width="960"] Reddit referral traffic trend, Dec '12 to Dec '13[/caption]

Even worse, if we drop the Dec '13 point entirely and plot the trend again, the line becomes even flatter (-0.004% per month). If a trend relies so heavily on one data point, then it isn't much of a trend at all. There simply isn't enough data here for Shareaholic to make any kind of claim about Reddit's traffic referral trends -- especially the audacious claims they made in the news.

[caption id="attachment_2544" align="aligncenter" width="960"] Reddit referral traffic trend, Dec '12 to Nov '13[/caption]

When it comes to sorting out the signal from the noise, this is pure noise that Shareaholic is trying to make look like a signal.

Fractions can be misleading

It's even strange that Shareaholic presented this data in terms of fraction of traffic referred from Reddit. For example, they say: "During December 2012, sites saw 0.33% of their overall traffic come from Reddit." Why not report the raw traffic numbers instead of a fraction?

[caption id="attachment_2593" align="aligncenter" width="890"] Fractions can mislead when total numbers aren't reported[/caption]

It could very well be that even though the fraction of traffic from Reddit went down, the total amount of traffic from Reddit went up.

To drive this point home: Even though 50% seems less than 75%, if we're comparing 75% of 200 visitors (= 150 visitors) vs. 50% of 1,000 visitors (= 500 visitors), clearly the 50% represents more traffic overall. In other words, even if there was declining fraction of traffic from Reddit, it could simply be because the web sites Shareaholic tracks are receiving more traffic overall.

With that in mind, we can't even be confident that the Shareaholic referral traffic data actually represents a decline in overall referral traffic from Reddit, as they so boldly claimed in the news.

Shareaholic doesn't care if their analysis is correct

Before publishing this article, I wrote to the Shareaholic team to explain the faults in their analysis. Their response?

Ultimately, our goal is to provide data [that] marketers can act upon, not data that they'd sit around in a room pondering about for hours on end.

It seems that Shareaholic doesn't even care if their analysis is correct, as long as it offers "data that marketers can act upon." Are you sure you want your marketers acting upon data that's as unreliable as this?

If Shareaholic can't even get basic data analysis right, I don't know how much confidence that should give you in the rest of their services.