Month: November 2018

Can You Trust Bad Online Ratings?

Source: Yale Insights On:

Read On

Collage of online ratings

By Roberta Kwok

Consumers rely on online ratings to guide everything from restaurant choices to furniture purchases. One review might not be trustworthy, but when websites average their ratings together, the thinking goes, its score should reflect the “wisdom of the crowd” and accurately capture its quality.

But research by Balázs Kovács, an assistant professor of organizational behavior at Yale SOM, and his collaborators suggests that this system contains a flaw. According to their analyses, products or businesses that initially receive poor reviews are likely to get fewer ratings in the future. Since the rest of the “crowd” doesn’t weigh in, the item may be stuck with an unfairly low score. It “might never be corrected,” Kovács says.

“If there are few reviews, the score is not only inaccurate but is biased in a systematic way,” adds Gaël Le Mens, a professor of behavioral science at the Universitat Pompeu Fabra, who collaborated with Kovács, along with Judith Avrahami and Yaakov Kareev of the Federmann Center for the Study of Rationality at the Hebrew University of Jerusalem. “There is a systematic tendency for such scores to be  underestimations.”

Read the study: “How Endogenous Crowd Formation Undermines the Wisdom of the Crowd in Online Ratings”

To understand the team’s reasoning, consider the following scenario. Two new restaurants of equal quality open. An accurate rating for both would be, say, three stars. The average of the first few reviews will likely not be exactly three stars since the sample size is small. By chance, restaurant A might get lucky and average four stars after the first week, while restaurant B gets unlucky and averages two stars.

What happens next? Diners who read online ratings will likely choose restaurant A instead of B. Restaurant A accumulates more reviews, and the rating eventually converges to an accurate average of three stars. But since few people try restaurant B, it gets stuck at two stars—even though it’s as good as A.

The rating isn’t the only problem, Kovács says. Review sites often order options from highest to lowest scores. If your business is poorly reviewed, “you go to the bottom,” he says. “People are not going to choose and rate you.”

To find evidence for this pattern, the researchers analyzed 78 million Amazon ratings and 2.2 million Yelp ratings. Highly rated products and businesses accumulated reviews more quickly than poorly rated ones did, they found. If an item’s score increased by one star, it received reviews an average of 16% faster on Amazon and 14% faster on Yelp.

To demonstrate that this leads items with few reviews to be underestimated, the team created a computer model. The model simulated the process of users selecting items to rate, based on their current scores, and submitting more ratings. As expected, an item that started out poorly was likely to remain stuck with a score that was lower than its true quality.

There’s no need for businesses to plead for high ratings. “If you just tell them, ‘Write an honest review,’ it will help.”

Next, the researchers conducted an experiment in which participants rated pictures presented on their computers. First, the participants were shown 50 buttons, each corresponding to a picture, and had to choose 10 to view and rate. For some participants, the buttons were ordered so that those corresponding to pictures with the highest ratings were at the top of the page, while lower-rated ones were near the bottom. Previous research suggests that in this situation, people are more likely to pick options near the top of the page, and thus pictures with the highest ratings would receive the largest quantity of additional ratings. This is what happened in the experiment. And just as the model predicted, pictures with few ratings tended to earn average scores that were lower than their “actual” quality, as determined by other participants rating pictures presented at random. 

The team also demonstrated they could reverse the bias by displaying the buttons in the opposite order. For some participants, the buttons were ordered from lowest- to highest-rated. In this case, pictures with few ratings got average scores that exceeded their actual quality. 

Finally, the team studied ratings of iPad cases on the French and German Amazon sites. The products were exactly the same, but ratings differed depending on the site. By comparing the data, the researchers could isolate the effect of the number of reviews. They found that having 10 times more reviews was linked to an average score 0.35 stars higher.

The study suggests that customers shouldn’t dismiss low-rated items with few reviews. “You’re likely to be positively surprised,” Le Mens says.

The results also suggest that businesses that start off with bad reviews can counteract the effect by asking more customers to review their product or service. There’s no need to plead for high ratings, Le Mens says: “Even if you just tell them, ‘Write an honest review,’ it will help.”

Online review sites also could take steps to counteract this bias, the researchers say. For instance, the sites could avoid displaying average scores for products with only a few reviews. And instead of ordering options based strictly on ratings, they could randomize the order a bit. Or the sites could identify items with few reviews and encourage users to rate them. “That’s one of the ways to fight this,” Kovács says.

Three Questions: Prof. James Baron on Amazon’s New Minimum Wage

Source: Yale Insights On:

Read On

An employee at an Amazon fulfillment center in Robbinsville, New Jersey. Photo: Bess Adler/Bloomberg via Getty Images.

Is a $15 minimum wage enough for Amazon employees to be financially secure?

No. For a full-time employee, $15 an hour works out to $31,200 annually, which definitely does not provide financial security, particularly in the urban labor markets in which Amazon tends to operate.

What is the likely effect of this change on Amazon and its workforce?

Amazon is likely to have an easier time recruiting and retaining entry-level people, but it is not clear what will happen to those who are a bit above the new minimum wage, who have seen their relative standing eroded somewhat by the bottom being raised. In research my colleague Amy Wrzesniewski and I have done on a firm that did something similar, they saw increased turnover among employees who were exempt from the pay change, who were presumably disgruntled about their loss of relative status within the firm.

Are other employers likely to follow Amazon’s lead?

Not all firms have been subjected to the bad press about labor issues that Amazon has, and so may not be so inclined to jump on this bandwagon. On the other hand, Amazon is presumably making this change in part because they are finding it difficult to recruit and retain people in a tight labor market, and their competitors (a number of whom have already raised minimum pay) may be forced by the labor market to comply. And having now decided voluntarily to go to $15 an hour, Amazon now has a strong interest in using its lobbying power to try and get the minimum wage raised so as to impose this burden on their competitors.

This website uses cookies to ensure you get the best experience on our website.