Where do VG Chartz North American Numbers Come From?
by Brett Walton, posted on 10 November 2007 / 5,838 Views
Some common misconceptions I want to put to rest.
1. VG Chartz gets no data from retailers.
Wrong. How does anybody think we would ever be approaching the level of accuracy and putting out the quantity of data we do each week if we didn't get data from retailers. The "leap of faith" many struggle with is why would retailers provide us with data? Which retailers? How much data? Why can't we say? What are we hiding?
Well unfortunately if we were to release detailed information at this stage on exactly where we get our data from then everyone would jump on the bandwagon and competitors would likely get their cheque books out and try to block anyone from providing us with any more data. Suffice to say (and this is where I do have to simply ask people for trust) that we do get data from a number of small retailers. Anyone with a background in statistical analysis will know that you don't need a large sample size to get good results, merely a good sample. For the small amount of data we collect, we get good results because we get a good mixture of data from different retailers - each of which represent various portions of the market - typically split into specialist gaming stores, electronics stores, general retailers, toy stores and online retailers. We have representitive data from all of those "areas" of the market and scale data up to represent the marketshare that each "area" represents of the total. This happens across both America and Canada.
Why would retailers provide us with data? Data exchange, many retailers are keen to know where they fit into the market - this is NPDs business model, they don't "pay" retailers for data they exchange data from one retailer for breakdowns of the entire market. This is far more valuable to a retailer than any monetary payoff and is exactly the reason why Walmart don't work with any independent tracking services (or do they?). This is also the same method that Neilson are hoping to use if they ever enter the market.
We can't go after the larger nationwide retailers (as yet), but we certainly have some interesting partnerships with smaller regional and independent stores, startups and so on. I hope people understand why at this stage they have to remain under wraps - expect official announcements in coming months though. We are also implementing premium advertising schemes for new partners - a community of over 10,000 avid games and 4 million impressions per month is a big draw for small retailers. Especially since you can now target advertising to gamers based upon games they already own...
This is where site expansion (in terms of number of visitors) will only make us stronger.
2. VG Chartz data is "based on" NPD data - they are stealing and benefiting from the work of others.
VG Chartz releases weekly sales figures on the Tuesday following the Saturday of the close of the previous week. In 3 days we collect an enormous amount of data from different sources, analyse it, extrapolte it, make comparisons and checks and put the data up online. This is a huge effort from a very small team of amateur (in that they have other jobs) but very able (in that they are highly educated and experienced) people who are doing it for very little payoff. We have been doing this for 18 months now, at the same time as going back and adding older data - some of our partners have been supplying us with their historical data as well, and have been refining our methods week on week.
Do we adjust our data? Not as such. Do we adjust our methods then? Yes - which will of course alter some data. On what basis? If we believe that a particular data set differs significantly from other sources of data (data released into the public domain by tracking firms, manufacturers, analysts) then we do re-check our data and make adjustments to the methods / scaling factors used. This happens on a fairly infrequent basis - less often than we adjust due to internal data changes - and is something that every tracking firm and analyst does. I personally have no issues with "benchmarking" our data from time to time against other sources of data - as long as it has been made public. Note, however, that any decisions by NPD to stop releasing public information will not affect our figures or service at all in my opinion - I am more confident every week in our own data and relationships with publishers that our data will only go from strength to strength. It is certainly no loss to vgchartz!
3. VG Chartz data is inaccurate and worthless.
Well this obviously depends on personal opinion. But for me, a site that gives sales figures to such a depth and accuracy as we do here, at no cost to any user is anything but worthless - especially with all the powerful analysis and charting tools available.
If we get figures to within, say, 15% accuracy in a given week (understand that statistically weekly inaccuracies are likely to cancel out over time, not accumulate) then I am happy. We pegged Mario Galaxy in Japan based of one retailer at 285k, it came in at around 255k according to Famitsu / MC - that's close enough for me. It tells me how well the title performed. It didn't do 900,000 like NSMB, it didn't do 100,000 and bomb completely. It sold a similar amount to Mario Sunshine and was quite disappointing - although should hold well over the holidays. I can make all these conclusions whether it sold 255 or 285k. Same as when we put the latest Tony Hawk figures out. Very poor first week. How do we know that if we're not getting "real" data? If we'd have predicted based on last years trend we'd have it a lot higher. I'm sure we'll be 10-15% different to NPD but so what? You know the game performed poorly. We had Bioshock sales way above expectation - good guess was it? We had Metroid Prime below Bioshock - again spot on. Too much is made of 10-15% being inaccurate - it is enough to give a good idea of how a game performs, and that is what we are all about.
If we put Mass Effect figures at 75,000 when it comes out people will be in uproar. They expect 300k+. If figures are confirmed a few days later at ~100,000 are people going to accuse us of being "inaccurate" because we were 33% off? Surely to have even put our figure at less than 100k is an achievement and shows that we have far more insight into sales than anybody else. Expecially when NPD might only, for example, have it at 85,000 themselves. I think a bit of perspective is required here on what is accurate and what is not - especially where software is concerned. To consistently get within 10-20% is damn good. Again, I ask for trust here, but from spending over 10 years looking at videogame sales data then to be getting this kind of accuracy week on week is far beyond where the professional tracking firms themselves were only 10 years ago (you'd be lucky to get 30-40% accuracy).
I hope this clears a few things up. Anybody with half a brain can see how exciting what we are doing here is and the enormous potential.
Any questions then please ask. If anybody is in a position to potentially help us out further then please get in contact.
39 Comments