MeaningfulNoise.com

I just finished an exhaustive audit with a certain panel-based measurement company (that I will herein refer to as “the masked bandits”) and it is more apparent to me than ever that online measurement across industries is lacking standards and commonality of definition. This generally means we’re all using different measuring sticks. And whereas I greatly appreciate the power and subtly of statistical modeling (the secret sauce of “the masked bandits”), data sets are becoming more and more disparate and therefore the data is not qualified. The modeling may be right in method, but not in fit. I often hear “which system is right?” My answer to this question is the same as it is to which measure should I use - depends on who’s asking and why.

The masked bandit I audited is the only source for competitive detail at the DMA level so it’s really the only source I have. Problem is, they are reporting metrics that are often contrary to actuals (hence the audit). Bottomline is that they have a fairly limited methodology that isn’t consistent with generally accepted business practices of most online companies today. The masked bandits also use a direct-measure panel so the gap increase since they don’t have to deal with cookie deletion (nor do they even attempt to estimate with the intent of standardization).

Outcome - sometimes it’s right, sometimes not, and you never really know on which end of the spectrum you are going to get. I used to just say it’s not about the numbers but the changes month-to-month and year-over-year. Then I started seeing more and more contrary motion. Grrr…

So what’s an analyst to do? Get to know the systems, understand why the numbers are the way the are, and deal with it from an education perspective. Do not engage in a battle of sources with vendors and methodology. It’s not worth your time. That’s not to say you shouldn’t engage them to run diagnostics on their data collection because sites change all the time and they miss stuff. But when the system checks out, get to studying. The impact of a sales person sitting across from an advertiser explaining the dynamics of the numbers has proven itself over and over to be more effective than pitching a fit and yelling “it’s not fair.”

Do you know the difference between panel measurement, direct measurement, and recall measurement… and do you know why they don’t marry up? Do you know what category of measurement your vendors fall into? And most importantly, have you educated your cross-functional groups on this matter so they can be informed instead of incited? I’ve come to realize this is a huge problem everytime I sit across from a client that is sweating about ranking in System A versus System B which isn’t even close to actuals.

~Wayne, Without Margin