Scoring Bias: Why Jordyn Wieber and Early Competitors Miss Medals

Answers to your questions about the news.
July 30 2012 6:00 PM

Did Gymnast Jordyn Wieber Perform Too Soon?

In Olympic scoring, the last shall be first.

US gymnast Jordyn Wieber performs on the beam during the women's qualification of the artistic gymnastics event of the London Olympic Games on July 29, 2012.
Would Jordyn Wieber have made it past preliminaries if she had gone last?

Photo by Ben Stansall/AFP/GettyImages.

Yesterday, Jordyn Wieber, the reigning world champion in the women's all-around gymnastics competition, failed to qualify for the event in the Olympics. Legendary coach Bela Karolyi blasted Team USA's decision to have Wieber compete first, adding that later performers are usually given better scores. "This shouldn't have happened," said Karolyi, the husband of national team coordinator Marta Karolyi. "This is a definite line-up mistake."

Is there really an advantage to performing toward the end of a competition?

Yes. The "last is best" effect occurs regardless of the scoring process: whether scores are given at the end of the competition (end-of-sequence judging) or after each performance (step-by-step judging). The power of “serial position effects” to influence a competition’s outcome has been observed in natural settings including Olympic figure skating, Olympic gymnastics, the Queen Elisabeth Music Contest, the World Synchronized Swimming Meet, and a Nebraska state high school gymnastic meet. 

Advertisement

Carnegie Mellon University researcher Wändi Bruine de Bruin analyzed 47 years of the Eurovision Song Contest. The data set is particularly revealing because the competition switched from end-of-sequence voting to step-by-step scoring in 1975. The benefit of performing late persisted.

Research offers several possible explanations for this effect. End-of-sequence judging, which we see in contests such as American Idol or the Academy Awards, may reward later performances because of the way memory and emotion work. Over time, our evaluations of disgusting meals and powerhouse singers become less extreme as the intensity of our emotional response fades. Our impression of the last performers, meanwhile, is fresher and suffers from less interference from competing memories. When the phone lines finally open, our memory of swooning for contestant No. 1 has long faded. We misinterpret the meaning of our current, vivid emotions, and insist that the last performer was the real deal.

Step-by-step evaluations by highly trained judges also suffer from a myriad of biases. In Olympic gymnastics, evaluators are given direct instructions by the International Gymnastics Federation to base the participants' scores on an ideal version of a performance with the same elements. The first performances are thus typically judged against a mythic, Platonic idea of perfection. Early in competitions, judges also tend to dole out moderate scores in the event that later routines will be even more deserving of high marks. 

Later performances are scored according to the judges' revised standard of performance: namely, that established by the first performances. In addition to inadvertently lowering their standards, judges tend to focus on the unique, positive traits of the later performances—something that's impossible for them to do for the first performers. One of Bruine de Bruin's studies, which analyzed figure-skating results from 1994 to 2004, found that the last to perform had a 14 percent chance of winning, compared to a mere 3 percent for the first participants.

Performers also suffer or benefit from social comparison, an effect that's been verified by researchers Lysann Damisch and Thomas Mussweiler from the University of Cologne. Their analysis of the 2004 Olympic gymnastics competition concluded that a major influence on a gymnast's score was the performance of the previous gymnast, which can alter a score by up to two-tenths of a point. As long as your performance isn't marred by errors, going after a skilled gymnast actually increases your score.

Karolyi's comment that Wieber deserved to go last because she is the best ignores the fact that having the best performers go last is, mathematically speaking, the least fair possibility. Since gymnastics competitions are decided by razor-thin margins—Wieber won the world all-around title in Tokyo by 0.033 points—she may have benefitted from the effects of sequential judging in the past. 

Unfortunately, competitions routinely reinforce the serial-position-effect bias. In figure skating, the order of the second-round performances is decided by score, with those in the lead going last. A more reasonable alternative would be to randomize the order in the first round of performances, and then reverse that order in the second round.

Despite overwhelming data pointing to the unfairness of having top performers go last, event organizers aren't likely to let science get in the way of performance order anytime soon, for one simple reason: ratings. 

The notion of saving the best for last surrounds us in many domains. Movie studios purposefully delay releasing their Oscar-bait films until late in the year; of the past 25 best-picture winners, six were released in the first half of the year, while 16 were released in November and December. American Idol producers know to place their star performers at the end, which helps build drama throughout the show and ensures that viewers stick around. But the science shows that we don’t always save the best for last: We save what’s last for last. Whether or not they’re the best is another story.

Got a question about today’s news? Ask the Explainer.

Explainer thanks Wändi Bruine de Bruin from Carnegie Mellon University.

Video Explainer: Why Don’t Firewalkers Burn Their Feet?

This video was produced from an original Explainer by L.V. Anderson.

Want more questions answered? You can now watch video Explainers at Slate's News Channel on YouTube.

Karla Starr is working on a book about the science of luck and chance. She can be found on Twitter at @karlastarr.

  Slate Plus
Slate Picks
Nov. 25 2014 3:21 PM Listen to Our November Music Roundup Hot tracks for our fall playlist, exclusively for Slate Plus members.