Dave Naffziger's Blog

Michael Arrington stirred up a few posts with his claim that ‘Comscore is notoriously flaky’:

One point of controversy was around Digg’s claim of 20 million unique monthly visitors and steep monthly growth, whereas the Comscore’s most recent September report shows only 1.3 million monthly unique visitors and flat growth since April (see chart below). Comscore is notoriously flaky, and these numbers are for U.S. households only. Comscore is almost certainly significantly under-reporting Digg traffic.

Fred Wilson, an early investor in Comscore, ‘takes exception to the word flaky’:

And I’d like to encourage everyone out there to sit down and understand third party measurement services before calling them “flaky”.

But, by Fred Wilson’s own estimates, Comscore is underreporting Digg’s traffic by 40%.

I’m sorry but you cannot build a ‘measurement’ service, based on a self-selected sample, particularly one that overrepresents users that sign up for proxied services to ‘protect their email from viruses’, and you certainly cannot make strategic decisions based on the data returned by the panel.

Fred goes on to make several good points about pitfalls associated with internal analytics (users that use multiple machines, different browsers or delete their cookies). A few commenters make incorrect points about crawlers (easily detected, and unlikely to significantly alter unique user numbers).

Time for an audit

The truth of about Digg’s users lies somewhere between Comscore’s 3M users and Digg’s 20M users. I generally dislike accountants and lawyers as I rarely feel that they share my interests, but when this much money is on the line it is certainly important to get your numbers straight. PWC has a very strong new media business assurance group that is likely the answer for both Digg and News Corp (albeit an expensive answer). We used their services at Quova to audit the accuracy and coverage metrics of the IP geolocation data that we provided to our users. I found them to be very intelligent and incredibly thorough - we learned a lot and improved a lot from their audit. They work extensively with the Internet Advertising Bureau and have largely driven the creation of their ad campaign measurement and audit guidelines. An expensive process, yes, but given the money on the line it will easily pay off for both parties so that they each know what they’re selling/buying.