Some “Research” Reports Leave Me Scratching My Head

If you read this space regularly, you know that I’m a bit of a data nerd. I love statistics, charts, and all sorts of quantitative analysis. I probably trust my gut feel more than numbers in the end, but I like to absorb the numbers to fuel my thinking.

So it was with great interest that I read about a report from BrightCove and TubeMogul titled “Online Video & the Media Industry Quarterly Research Report for the fourth quarter of 2010.” I was immediately sucked in by a chart that shows massive growth in newspapers’ deployment of video content in the final quarter of last year — along with a measurable drop in video content from broadcasters.

This is just the sort of revelation that I hunger for because it counters prevailing wisdom and fuels new thinking. So I read more. And as I looked closer, I kept scratching my head trying to figure out what would account for such a dramatic shift in such a short period of time.

I went digging for the report’s methodology. That’s when I read this on the Brightcove blog:

The data used for the analysis included in this report was taken from an anonymous, cross-section sample of Brightcove customers representing media industry segments and brands. While the sample aggregates a sizable data set, it is not intended to be statistically representative of the online video industry as a whole, or of Brightcove’s entire customer base.

Huh? It’s based on Brightcove’s own customer data. That’s not unexpected since it’s the data they have ready access to. And while it would certainly introduce some uncertainty into the results, Brightcove is a large enough provider that the data could be interesting anyway.

However, it appears that they did not even make even a half-hearted effort to weight, correlate, or otherwise connect this data to the industry as a whole. Moreover, it isn’t representative of their own customer base. Is it just aimlessly random then?

Instead of telling us anything about the newspaper and broadcast industries, it’s just as likely that this data reflects the relative abilities of the Brightcove salesforce. Perhaps they brought on more newspaper customers in the latter part of the year and lost some broadcasters.

Further, it’s possible that many broadcast outlets may roll their own video solutions rather than turning to Brightcove since that’s a core function for a video producer. Conversely, newspapers are less likely to have video experts on staff, so they might be more likely to turn to a third party provider like Brightcove.

The purpose of this post is not to condemn Brightcove in general. I have been a happy customer of their services in the past and will continue to use them in the future.

Instead, my point is to draw attention to the fact that it is important to evaluate the source of all data and the methodology of research reports before relying too heavily on the information contained within them. The Brightcove/TubeMogul report might be entirely valid, but the methodology certainly raises some legitimate questions.