Published on January 14, 2014 by Dr. Randal S. Olson
5 min READ
This is the original version of the article that appeared on Co.Design.
Earlier this week, Shareaholic published a report claiming that Reddit referral traffic is rapidly declining. Shortly thereafter, the report appeared on several respectable news outlets, including The Atlantic and Business Insider, chiming in and recommending that SEO experts start looking at web sites other than Reddit for referral traffic.
Normally I'd ignore a report like this because I don't care much about SEO, but the visualizations and analysis in Shareaholic's report were so bad -- and so misleading -- that I had to write a response to it.
First off, Shareaholic's claim rests on their comparison of two cherry-picked data points: Reddit referral traffic at Dec '12 and Dec '13. They compare referral traffic between those two points and note that it's gone down by 33%, then wildly speculate that Reddit's referral traffic is crashing.
One of the things we learn in high school math class is that it takes at least 3 points to make a trend, not 2. In fact, if Shareaholic had posted this report in June '13, they could've told the opposite story: Reddit's referral traffic is skyrocketing (up 66%)! That's a sign right there that the methods they're using are flawed.
If we want to talk about trends, we need to consider all of the data. Shareaholic shows us a simple line plot connecting the dots, but doesn't even try to show us what trend is actually going on. Can you easily tell if there's really a significant downward trend going on there? That's the sign of a poor data visualization.
When we take a closer look at the referral traffic data Shareaholic presented, we can draw a trend line through those points and notice that there isn't much of a trend going on at all: The line is almost flat, going down only 0.007% every month.
Of course, this trend line predicts that Reddit referral traffic will stay in constant decline, which we know isn't the case. Just look at what happened in June '13. There's a lot more to Reddit's referral traffic trend than meets the eye, and Shareaholic's oversimplified analysis doesn't even come close to explaining it.
Even worse, if we drop the Dec '13 point entirely and plot the trend again, the line becomes even flatter (-0.004% per month). If a trend relies so heavily on one data point, then it isn't much of a trend at all. There simply isn't enough data here for Shareaholic to make any kind of claim about Reddit's traffic referral trends -- especially the audacious claims they made in the news.
When it comes to sorting out the signal from the noise, this is pure noise that Shareaholic is trying to make look like a signal.
It's even strange that Shareaholic presented this data in terms of fraction of traffic referred from Reddit. For example, they say: "During December 2012, sites saw 0.33% of their overall traffic come from Reddit." Why not report the raw traffic numbers instead of a fraction?
It could very well be that even though the fraction of traffic from Reddit went down, the total amount of traffic from Reddit went up.
To drive this point home: Even though 50% seems less than 75%, if we're comparing 75% of 200 visitors (= 150 visitors) vs. 50% of 1,000 visitors (= 500 visitors), clearly the 50% represents more traffic overall. In other words, even if there was declining fraction of traffic from Reddit, it could simply be because the web sites Shareaholic tracks are receiving more traffic overall.
With that in mind, we can't even be confident that the Shareaholic referral traffic data actually represents a decline in overall referral traffic from Reddit, as they so boldly claimed in the news.
Before publishing this article, I wrote to the Shareaholic team to explain the faults in their analysis. Their response?
Ultimately, our goal is to provide data [that] marketers can act upon, not data that they'd sit around in a room pondering about for hours on end.
It seems that Shareaholic doesn't even care if their analysis is correct, as long as it offers "data that marketers can act upon." Are you sure you want your marketers acting upon data that's as unreliable as this?
If Shareaholic can't even get basic data analysis right, I don't know how much confidence that should give you in the rest of their services.