But there's been lots of chatter recently regarding the use of the star system. The quantity and variety of reivews that use stars is immense. Stars are used to rate everything from the caliber of your hotel in Madrid to the safety of your car. The level of subjectivity of star systems depends on how they're used. For hotels if rooms have 600-count sheets, marble floors, on-call butlers they'll get more stars. If crash test dummies can walk out of an accident to tell bad jokes, that car gets more stars. Dining reviews however, well, they're one of the most subjective reviews to offer a star system. Does the uptown, top floor bastion of fine dining with tired, mediocre dishes deserve more stars than a downtown taco shack with the best tacos this side of Baja?
Progressive periodicals are dispensing with the star system for restaurant reviews (I'm looking at you LA Times). On the other hand, certain French travel guides would cease to exist if they dropped their star system. Which brings me to our local periodical, which basks in the star system as if it were an innovation on par with movable type. Recently, the intrepid restaurant critics of The Miami Herald have gone on a tear of perplexing starred dining reviews that demonstrate they're complete lack of a methodology on how to dish them out. Not only does there not seem to be a fixed methodology across the critics, individuals seem to not have their own consistent credo.
So what does an MBA do when trying to quantify the perplexing star system? Well, he looks at the data, and given the Herald's propensity to dumb things down, they made it pretty easy. A peek of each review appears in the Herald-owned miami.com (and gets aggregated with other local online reviews by Eater Miami) prior to the full review running in the Herald. These peeks take each review and compact it into bullet points (which I'm a huge fan of) titled "What worked" and "What didn't". So here we have a great set of data points to use. Each observation of the restaurant fits into one of these categories. Steak was done to perfection = What worked. Lousy service with long waits = What didn't work. Easy as pie.
Then come the stars. Each reviewed restaurant is given a rating between one and four stars with one being, well, I don't know what it means because there's no legend to go by. I have to assume one is bad and four is exceptional. But a quick glance at a few reviews shows that a 1.5 stars denote subpar while a 3.5 star review = exceptional. I must then assume that 2.0 stars = par and that 4.0 stars = exceptional + 0.5.
All's good so far. We've got a list of "What worked" and "What didn't" at each reviewed restaurant as well as its stars. But how do these two data sets relate? Well, many times they don't. Sifting through at the data I looked at all observations under the "what worked" and "what didn't" to get a sum of total observations. I then took the number of "What worked" observations and divided by the total to get a "% of What Worked". Overlaying this percentage with the stars assigned in each review gets you a picture of how in or out of whack the two are. Examples of some recent reviews appear in the chart below with the blue bars denoting the star rating and the orange squares denoting the "worked %" of "worked observations" divided by "total observations".
There's few instances where the both data sets come out to similar results, but in most cases there's some pretty egregious differences. Let's look at the extremes. Victoria Pesce Elliot waxed poetic about Norman van Aken's Tuyo at Miami Dade College, giving the restaurant a study-high rating of 3.5 stars. At the other end of the spectrum is Blue Collar, also reviewed by VPE, which serves up unabashed American comfort food with sophisticated touches (and I mean Miami American where tostones and latkes peacefully coexist). Blue Collar received 1.5 stars. Now looking at the "What worked %", Tuyo came in at a so-so 73.7% (14 worked out of 19 total observations) which was below all but one of the 3 star establishments in this study. Blue Collar, on the other hand, had a 50% "What worked %" which, when compared to restaurants at the 2 star level, surpassed one and tied the other.
The discrepancies become more egregious when looking at the subjective observations of these reviews. Again, Tuyo, at 3.5 stars, should be considered one of the city's best; however, here's some of the dings from VPE. Would you think a 3.5 star establishment had "uninsipring, overworked pompano" or a "waiter who lacked a grasp of both English and the menu" and "a kitchen that can delay dishes"? These observations absolutely scream 3.5, or near perfection. Then let's take Kopas, a 2 star establishment that had a "What worked %" of 36.4%, lower than the 1.5 starred Blue Collar. Comments on Kopas' shortcomings included "It was a solitary pursuit. On both 8pm visits, we were the lone diners in a large, lacquered room with an empty outdoor patio and a sole waiter humming Pitbull tunes behind a shale-rock bar; chunks of fried meat and fish in soggy batter; an unidentified fish in classic tiradito; mushy shrimp ceviche." So an empty restaurant with soggy battered meat and unidentified fish gets a passing grade whereas Blue Collar, an establishment which (i) is consistently packed by young (and old, despite VPE review stating that the clientele was of the young variety), (ii) where you can identify what's on your plate, (iii) there's no Pitbull to be heard and lastly (iv) has a higher "What worked %", got rated as subpar.
It's well known that newspapers across the country have had to cut back, our own Miami Herald included. Staff reductions and mandatory furloughs have hit the morale of traditional journalists working at local papers. I believe eventually the market will lead these papers to the right business models that will ensure their longevity. But one characteristic that will survive is good, consistent reporting and content. The methodology being employed by the Herald for dining reviews is epicly backfiring. Perhaps a more factual style similar to the weekly "Fork in the Road" by Linda Bladholm (the only writer I tend to respect in the Herald's food and dining section) where there are no stars employed and no hyperbolic language to be found could bring some respect and regard to these reviews.