Response to Internet Comments about VGChartz

Tweet

Simon Carless yesterday wrote an article criticising VGChartz:

http://www.gamesetwatch.com/2008/06/analysis_what_vgchartz_does_and.php

A few comments:

• Up to "Iron Man and retrofitting" the article is factual, neutral and reasonable. Carless explains the background and reports our methodology but then dives into his assault with "But did you realize that VGChartz estimates can retroactively change by 100% or more based on 'official' chart results?"
• Carless then goes to cherry pick one example of a title where our estimates were off - Iron Man on PS2. Dismissing the other 4000 titles that we tracked for the month, many of which fell within a 5-10% margin of NPD data, Simon choses to focus on the one title that he can find with the biggest difference possible and make a point of it - totally missing the very nature of sampling and statistics. Would it shock anybody that NPD themselves have been a FACTOR OF TEN off on sales of a game in the past? We spoke to a developer who had recorded sales of over 500,000 units in the USA and NPD were reporting less than 50,000 sold. A factor of 2 out suddenly doesn't look so bad.
• Carless then goes on to labour the fact that VGChartz will adjust data - what other option would anybody suggest? Chart Track adjust data, NPD adjusts data, can VGChartz not adjust data if a better source shows we are wrong?
• Ben 10 - carless clearly shows he has no idea here by comparing VGChartz Americas data to D3 Publisher Worldwide data and also comparing sell-through to sell-in, hence the discrepancies.
• He then vaguely choses two titles (which he won't list) and makes some argument that they are incorrect - great reporting.
• He then goes on to criticise the MGS 4 day one estimate - European sales were indeed extrapolated FOR DAY ONE from Americas sales since there is a clear pattern with all PS3 games where Europe tends to be about 10-20% higher than the Americas. Nothing wrong with this and good to give a ballpark figure for day one. Why Carless has a beef with this and then goes on to suggest that all European figures are estimates in that case is beyond me. Again, totally failing to get the point.

So yes, there are literally hundreds of examples where our data for a game won't line up perfectly with another source, even some examples where we can be a double or half the sales reported elsewhere (although this is a rare occurance). What Carless fails to do is present a fair and balanced argument - for every title that we undertrack or overtrack by a significant margin, there will be 50 that fall within 10%. As an overview to the industry - which platform is performing well, which titles are performing well, which genres are performing well, our data is more than good enough. For an exercise in taking our dataset and a dataset from another company like NPD and cherry-picking examples of cases where the data disagrees then sure you can find many examples and why wouldn't you expect to do so? The two main tracking agencies in Japan - Famitsu and Media Create can sometimes disagree on the sales of a particular title by 100% (rare but it does happen) and both have 60% market coverage and a lot of overlap. It only takes one store to have a promotion or heavy sell on an item and it will skew all estimates.

For what VGChartz offers, the data is solid, free, readily available, timely and presented in a way that is easy to manipulate and analyse. It does annoy me how people hang on minor differences, mistakes or errors and use them to discredit a service without realising and applauding the hard work that goes in and the generally high quality of the data produced.

More Articles