Reviewing the Reviewer - Article
by Lee Mehr , posted 11 hours ago / 642 Views
"Listening to critics is like letting Muhammad Ali decide which astronaut goes to the moon." – Robert Duvall (Actor)
If we're being honest, "games journalism" has been more of a punchline than a legitimate concept within the community for ages. That sustained attitude hinges on one specific category: reviews. The pool of controversy is vast, from validated conspiracies like Jeff Gertsmann's forced exit from Gamespot to… shall we say suspicions of critics' socio-political biases tainting their assessment of certain works; though not all-encompassing, examples like some of the Days Gone discourse or a ragebait 1/10 score for Hogwarts Legacy couldn't help but turn a few heads. For the average gamer on a limited budget, it's understandable to see so much background radiation and wonder who can be trusted.
Enter Thomas Mahler: Co-Founder/CEO of Moon Studios, Writer/Director of the Ori duology, and Director of the upcoming No Rest for the Wicked. On September 2nd, he formally announced a website called VideoGamesCritic (VGCrit for short here) that – among other things – tabulates how gaming sites and major content creators align with the community writ large. Think gaming equivalent of "who watches the watchmen?" But that aforementioned word "trust" calls this point into question: is adherence to consensus the correct gauge of a critic's reliability?
Then & Now
It's funny that the further we look back at the 'art' of critiquing, such a question seems rather misguided. Just as paintings, literature, sculpture, and more date back millennia, so too does criticism; moreover, given the work done by Aristotle and others around his epoch, its utility was geared towards philosophy and theory. That mindset echoed throughout later generations when criticism eventually grew into a profession. During the Early Modern period, prominent voices like Samuel Johnson and Alexander Pope could use what they were evaluating as a launch pad for broader discussions. Prestigious schools on artistic and/or philosophical thought would have similar goals in mind, like looking at someone's work through a Christian or Humanist lens, as an example.
There was less expectation as to whether or not something was worth one's time. That's not to say such thinking was wholly absent, mind you, but the overriding assumption was the upper crust would partake regardless. Think about it: you're in the club and have disposable income, so why not see the new play with friends and discuss it afterwards? Hell, those with the deepest insights likely enjoyed an elevated social status as well. Once the hoi polloi become better educated and earned more than a pittance for a day's work, consuming culture grew into a regular habit.
Naturally, this growing mass of potential customers resulted in our modern climate: consumer criticism effectively acting as reportage. Assuming you grew up within (or near) the Millennial Generation, go back to gaming magazines of your youth or film reviews in newspapers of the same era for proof. How was the structure of x movie or y game? How does the gameplay compare, or the editing, or the storytelling, or whatever else? You get the point. Even if unstated, the goal of most reviews was considering the general audience, rather than indulging in antiquated ideals from centuries past. There was greater emphasis on viewing technical aspects of media, before select outlets would incorporate a 'culturally-conscious' vantage point. Considered alongside our collective history of criticism, it's understandable to abhor a measuring system slanted towards audience conformity – especially given VGCrit's front page tagline being "the one source of truth for gamers."
Consistency or Compliance?
Past its pomposity, though, the site proper succeeds at its intended goals. There's an impressively transparent system of assessing player sentiment across various outlets, Steam's in particular also weighs user influence based on total playtime. For nitpicky jackasses like myself, Mahler & co. premeditated many angles ahead of time:
- More weight is given towards those who purchased it instead of given a free code.
- Evolving player sentiment after patches or major updates is tracked, so critics' opinions are considered according to those timelines.
- There's consideration for discounting unjust review-bombing efforts altogether.
- Total review weight halves every two years, so no poor-performing site nor creator is permanently stuck with a lower trust score.
- It accounts for differences between more lenient and harsher sites on the backend.
Even with quibbles like Nintendo titles circumnavigating around stricter scrutiny from Steam's community, its methodology is quite thorough. The core contention, though, is about philosophy rather than policy.
After watching his appearance on the Sidescrollers Podcast and continued discussions on X/Twitter, it's interesting to see a creative's viewpoint on quality criticism – especially from someone who's only experienced near-universal acclaim. There's this peculiar expectation of a quasi-objective rubric naturally conforming to an average audience, while most critics (that I've worked with) have a different goal in mind: succinctly & cogently voicing their opinion to whoever's willing to listen. It's that simple! There will be challenges and pushback, but that's the nature of subjectivity. Looking towards a hypothetical audience on whether to praise or denigrate something would make you a less helpful critic in the long run.
Hell, never mind less helpful. Taken to its logical conclusion, you're compelled to embody an appeal to popularity, rather than earnestly evaluate something, in order to strive for a higher trust score. Going down that route leads to a potential quagmire of contradictions: what if a large chunk of the audience isn't "trustworthy" over a particular game? What if both critics and community alike have skewed sentiments over something because of ideology rather than actual content? What if the audience naturally sours on a title over time? Does anyone ahead of the curve garner extra credit on their trust score? Tracking which way the wind blows befits a weatherman, not a critic.
However…
Given my cantankerous attitude towards much of pop culture, it's practically a reflex to align with Mahler's mainstream detractors here in spirit; then again, given my previous track record, it's easy to be annoyed by some of their uncritical outbursts in kind. Show of hands: who's surprised to see Kotaku desperately point and sputter over this site announcement? Of course, it's important to admit the favorable rankings of my two current outlets (VGChartz & Cubed3), which could positively color my viewpoint; speaking anecdotally, it's rather surreal how close its early trust scores harmonize with my personal outlook with each place I've contributed to. Even with acknowledging these potential biases, though, the crux of my contention comes down to nuance.
The most important distinction is between what reviews are ideally and practically. Even if most writers echo my aforementioned arguments, I still find the data quite damning in other respects. Is anyone genuinely surprised that, of the over 150 outlets currently compiled on this leaderboard, several console-branded sites (Xbox Achievements, Completexbox, etc.) have some of the lowest trust scores? Is anyone shocked that Game Informer, a print magazine owned by Gamestop for over two decades, is also currently near the bottom? Would anyone be stunned to discover that, of the critic scores most misaligned with audiences, some of those hypothetical examples could have been heavily advertised on that particular site? Conversely, is anyone dumbfounded that most of the current Top 25 are less-popular independent joints? You can opine on Mahler's misguided outlook all you bloody well like, but it's foolish to ignore why this grading system feels so sensible to certain audiences and creatives.
There's no dismissing high-profile mismatches between critics and players, the potential reasons why, and continually lackluster self-policing. Here are just a few that come to mind:
- Older Call of Duty events, wherein writers would enjoy goodie bags and paid-for trips. Even if every critic vouched they only focused on said game, the positive experience within that environment played some kind of role.
- Initial glowing reviews for The Master Chief Collection, thanks to critics only experiencing multiplayer between each other on steady servers, rather than what the average consumer experienced for years.
- Metal Gear Solid V's review event, where the press were allotted a limited time window, only experienced it with unique in-game modifiers, and were never unsupervised whilst playing it.
- Recent rumors suggest Rockstar hosting Grand Theft Auto VI enclosed review events with similar conditions. Given the anxiety around its release and its close proximity to The Game Awards submission deadline, those invited critics are undoubtedly dealing with unique pressures.
For those who'll proclaim VGCrit anathema to evaluating games as art, it's rather convenient to disregard these rigged systems many sites go along with. This kind of symbiotic relationship treats games more as commercial products hoping to be sold at full price than an exploration of a title's design elegance, storytelling, and so on. You can't have such a prolonged, cozy network and then admonish someone else trying to judge your own accuracy – erroneous as that mentality may be.
However, However…
You see, this is why it's so aggravating to navigate a balance here. This isn't intended as shit-slinging, but it's hard to ignore such egregious elements. Because it's not like a system of early viewers, embargoes, etc. is wrong per se; that's existed in entertainment media for ages. The tension lies between enthusiast press yearning for access and big publishers' annoying hoops when giving it. Having seen and ultimately declined the excessive NDA demands for The Last of Us Part II, I understand when artificial rules can disrupt what you'd want to write about while not anticipating how your hands will be tied. Beyond that, most – if not all – big publishers can dangle said access or use other tricks to garner high praise for their Accolades Trailer. After all, money is on the line.
Publishers aren't inclined to stop courting positive bias and it's hard to find a genuine solution for outlets that enjoy the status quo; at the very least, perhaps there are some long-held defenses that could be reevaluated. Among the most popular used for and against most outlets is the variety of potential critics – especially on big sites. I've tried to correct people when they'd angrily say "can't believe IGN thinks…" instead of "can't believe this IGN critic thinks…" for years now. Though it's important to delineate between the two, does the audience also have a point? Perhaps some frustration stems from how randomized said outlet's writers come across, as though they're not following the same methodology, whether it's inherently harsher or easier with scores.
Perhaps other considerations, like critics delaying reviews for online-focused titles until playing with the general audience on launch day, could earn more trust than opinion alignment. It speaks to a willingness to be patient instead of indulging in this fucking rat race to reach Google's top searches. Not to blow my own trumpet (again), but perhaps a more concerted pushback against potentially overreaching NDAs could help as well; ensuring a publisher's PR doesn't make demands that could artificially limit how a critic appreciates or dislikes a work reflects a standard that's reader-oriented first and foremost. Whether or not these ideas are substantial enough, at least it's something that can still accommodate games media's current setup.
Conclusion
For the sake of escaping this wall of waffle: VGCrit's didactic language in fusing "trustworthiness" with audience alignment is a myopic position, though understandably influenced by past audience disparities that might've been commercially-motivated. Despite that mistaken outlook, there's still genuine value in the data itself; concrete numbers and graphs reveal uncomfortable trendlines for several bigger fish and console-branded sites, which then leads one to inquire if some overly-positive pieces were influenced by purchased ad space. If put off by such sweeping generalizations against an outlet, ReviewerLikeMe works by calculating which writers best fit you after ranking several games on their 5-star system. It's rather crude, what with it not currently enabling half-stars, but it's useful in putting a name to those most-aligned with your tastes – rather than an organization.
Then again, maybe neither avenue best answers that issue of trust. It can be a helpful barometer, sure, but not a guarantor of reliability in itself. Friction from disagreement can be more illustrative of what elevates or diminishes a work in someone's eyes, be it a design quirk, socio-political outlook, dollar-per-hour value, or whatever else, for a greater understanding of them going forward. That push 'n pull can also elucidate their familiarity (or lack thereof) with various genres, which in turn can be compared to your own experience; and whether more or less knowledgeable, the best examples of either feel more like explorations of a game's intended goals than something gauged as "the most objective review". In the end, any critic who maintains a keen insight and a consistent voice will be more reliable than an abstracted agreement meter could ever achieve.
Contractor by trade and writer by hobby, Lee's obnoxious criticisms have found a way to be featured across several gaming sites: N4G, VGChartz, Gaming Nexus, DarkStation, TechRaptor, and Cubed3! He started gaming in the mid-90s and has had the privilege in playing many games across a plethora of platforms. Reader warning: each click given to his articles only helps to inflate his Texas-sized ego. Proceed with caution.
More Articles
"...speaking anecdotally, it's rather surreal how close its early trust scores harmonize with my personal outlook with each place I've contributed to."
For the sake of further clarification: VGChartz & Cubed3 were ping-ponging between #15-#20 spots at the time. This anecdotal comment wasn't meant to throw shade at some of my previous spots, or at least not the contributors personally. It's more about what I'll call the underlying system for their reviews methodology. Even being overly negative to a few bad games gave me a reputation with one site.
