The Yelp Factor: Are Consumer Reviews Good for Business?

In a new study, Assistant Professor Michael Luca shows just how much restaurant reviews on Yelp affect companies' bottom lines. The more difficult question: Are these ratings reliable as a measure of product quality? Key concepts include:

• Each ratings star added on a Yelp restaurant review translated to anywhere from a 5 percent to 9 percent effect on revenues.
• Local, independent restaurants were the most affected by reviews, probably because diners have little information about them before the reviews were posted.
• Chain restaurants were largely unaffected by the ratings—presumably diners already had formed impressions thanks to the chains' ability to advertise. However, demand is beginning to shift from chain restaurants to local restaurants.
• The more difficult question to answer is whether the restaurant ratings are reliable as a measure of their actual quality.
• Online marketplaces must be concerned with not just reviews but all the factors that create trust with their users.

 

by Michael Blanding

In recent years, consumer review sites including Yelp, Citysearch, and TripAdvisor have become the first stop for recommendations on everything from dinner to dentists. Along the way, they've earned a loyal following from fans, but also the ire of businesses that find themselves hurt by dyspeptic reviews.

“Restaurants are a classic example in economics where the consumer has to make a decision based on very little information”

Most would agree these sites do influence consumers' decisions. In the paper Reviews, Reputation, and Revenue: The Case of Yelp.com, Harvard Business School Assistant Professor Michael Luca set to find out exactly by how much, and identify winners and losers in the process. "I have always been interested in how companies form their reputations, not only restaurants and hotels but also schools and doctors," says Luca.
How reputation is earned in the digital age is a fertile area of his research. Luca is looking at rankings, expert evaluations, online consumer reviews, and quality disclosure laws to see how they work in market settings, and which are most important for consumers.
To get a handle on the phenomenon, Luca focused on one of the most common uses of online ratings: restaurants. "Restaurants are a classic example in economics where the consumer has to make a decision based on very little information," he says.
In theory, ratings sites fill in the gaps by providing a tremendous amount of information on which to base decisions. In Seattle alone, Yelp, which launched in 2005, had accumulated 60,000 reviews by 2009, rating 70 percent of the city's restaurants. By contrast, The Seattle Times had reviewed roughly 5 percent. Luca chose to investigate Yelp not only because it was the first restaurant site to emphasize user reviews over professional critics, but also because of its historical database that tracks every review.
Luca compared the ratings over time with revenue data from the state of Washington to gauge how reviews impacted restaurants' bottom lines. In order to establish causality between reviews and a restaurant's success, Luca took advantage of the fact that Yelp rounds its ratings to the nearest half-star-so, for example, a 3.26 rating would be rounded up to 3.5 stars, and a 3.24 rating would be rounded down to 3 stars. Hence restaurants with similar ratings can have very different rounded ratings displayed to consumers. By applying a regression discontinuity analysis, Luca could study how revenues jumped when star thresholds were crossed and compare these with the more steady changes over time. The result: Luca could directly trace the effect to the ratings on Yelp.

Ratings impact revenue

When the dust settled, Luca determined that each ratings star added on a Yelp review translated to anywhere from a 5 percent to 9 percent effect on revenues (depending on the control variables and means of estimation)—more than he had expected.
Even more interestingly, within that number not all restaurants were created equal. Chain restaurants, in particular, were largely unaffected by the ratings, while the greatest effect was shown for independent restaurants. That makes sense according to economic theory, says Luca, since diners presumably already have some knowledge about chain restaurants, but can benefit from more information about their neighborhood spots.
"Yelp is somewhat of a substitute for traditional forms of reputation," says Luca. "People are not using Yelp to find out about McDonald's."
But Luca says the big chains should not be comforted by the findings. Indeed, he believes the data suggests that local eateries are starting to siphon off customers from the "Big Boys." Why? The Applebee's and the T.G.I. Friday's of the world have been safe bets for diners because their fare is of an expected quality and their menu well-known—thanks in part to big-budget advertising behind the chains. But review sites are leveling the playing field by allowing consumers to learn as much about independent restaurants as they know about the chains. "This is one reason why consumer demand is shifting from chain to independent restaurants in the period following the introduction of Yelp," Luca writes.

Where's the beef?

The more difficult question to answer is whether the public's reviews are reliable measures of a restaurant's actual quality. Restaurant owners routinely grouse about being held hostage to their Yelp ratings, accusing other restaurants of gaming the system by stacking reviews by friends and family, or simply falling prey to a popularity contest by uninformed palates.
"Given the evidence, I don't think many people would dispute Yelp's influence," says Luca. "The question is, is this a good thing or a bad thing? If the ratings are capturing real quality, then that's a force for good. To the extent that there is gaming or nonrepresentative views, that's a problem. The onus is on the review industry to change that."

“If the ratings are capturing real quality, then that’s a force for good”

Luca has been following up his initial research to determine how review sites can do just that. He notes that consistency of reviews were weakest when there were a low number—supporting the criticism by those restaurants that blame Yelp for killing business with a couple of bad reviews. "Some of the complaints I hear are from restaurant owners in small towns who get one or two reviews—it has an outsized influence on your business. It can be hard for them to wait for more reviews. Once you start getting 20 or 30 reviews, the variance goes down," says Luca. The flip side, of course, is that most small restaurants in smaller communities wouldn't get reviews to begin with if it weren't for the review sites, and for every eatery that suffers under the weight of a bad review there are presumably others boosted by positive reviews.
Luca applauds features on Yelp that try to improve content quality, such as requiring reviewers to create publicly available profiles, and identifying some as "elite reviewers" based on their number of reviews, so users can give them extra weight. Holding users accountable with social networking tools makes Yelp stronger than similar sites like TripAdvisor, which doesn't have such accountability. On the other hand, Luca approves of the more extensive options that TripAdvisor provides for sorting and categorizing reviews of hotels—for example, business travelers can find reviews written by other road warriors, who are likely to have similar requirements.
"The ability to quickly find reviews from people with similar tastes is one potential area of improvement for review systems," says Luca. "At this point, there is no single site that has mastered it all."

Beyond reviews

But even as consumer reviews influence more and more purchase decisions, websites must be concerned with the larger picture of all the factors that create trust between them and their users. "Reviews are one piece of the reputation puzzle, but a comprehensive reputation system must do more to facilitate trust by incorporating other types of information as well. For example, Yelp could include other publicly available information about restaurants."
Expanding on this research, Luca is working with HBS Assistant Professor Benjamin Edelman to put a framework around reputation-building in online marketplaces. They are currently finishing work on a case study about Airbnb, where anyone can offer a room in their house or an apartment for short-term rentals.
"Compared to restaurant choice, people are taking a big risk when deciding to rent their apartment to a total stranger or to stay in a stranger's house," Luca says. Because of this, Airbnb offers not only a review system but also insurance to pay for incidents resulting from unreliable guests, authentication to make sure that guests and hosts are who they say they are, and a screening process to give market participants the ability to decide when and when not to do business with each other.
"The right mix of these tools is very context-specific, and this case helps to highlight some of the challenges to building a robust reputation system," Luca says.
As it becomes more de rigueur for consumers to use website reviews and other information to make important decisions—from where we are going to vacation to who will treat us when we're sick—it's all the more critical that businesses provide consumers with enough quantitative and qualitative information to ease their decision making.
And that has Luca thinking. "How can we extend this to other industries, and how can we better design these sites?" says Luca. "It really comes down to what types of truths we want to find in each site and what we can do to ensure reviews are of a high quality."



Source: hbswk.com