The great Carmelo debate: Do basketball advanced-box-score stats really work?

The great Carmelo debate: Do basketball advanced-box-score stats really work?

The stadium scene.
March 7 2011 6:53 PM

The Great Carmelo Debate

Why can't basketball stats nerds separate the superstars from the ball hogs?

Carmelo Anthony. Click image to expand.
Carmelo Anthony

In his first few games with the New York Knicks, Carmelo Anthony has been everything everyone expected him to be. To admirers, he's putting up the numbers of a legitimate superstar: about 25 points and seven rebounds a game. Detractors see a different Melo: a ball hog who's shooting a meager 42 percent from the field. "I think what Carmelo does is, the more players you have guarding him, the more he wants to shoot. Which is the opposite of what you're supposed to do," says economist Dave Berri, author of several books on sports statistics and player evaluation.

In the run-up to the NBA trading deadline, Berri told the Wall Street Journal that if the Knicks sold the farm to bring in Carmelo, they would win "roughly 29 games over a full year." After seeing which players were actually involved in the deal, he said the team might win 50. * A lot of people thought that sounded off. New York Times stats guru Nate Silver argued that, on the contrary, Melo is "the ultimate team player" because his offensive game draws defenders and allows his teammates to get more wide-open shots, boosting their field goal percentage. Carmelo's value, Silver and other analysts said, was spilling over his own stat line and into the box scores of his teammates.

To number-crunching nerds raised on baseball, in which pitcher and batter compete in statistically delicious one-on-one duels, such messy "interaction effects" are a lumpy blemish in the box score. In January, seasoned baseball sabermetrician (and occasional Slate contributor) Phil Birnbaum argued in his blog that basketball's so-called "advanced" box-score stats are so gnarled with this problem that they can't be trusted. (He went so far as to call them "the RBIs of basketball"—a brutal insult to lovers of well-validated statistics.) Is Birnbaum right? Is this why people can't agree on the value of Carmelo?


The statisticization of basketball is a relatively new enterprise. Its modern incarnation is usually traced to Dean Oliver, a Ph.D.-trained statistician who started scrutinizing the game on a per-possession basis in the 1990s and derived the so-called "Four Factors" that make for hoops success: shooting, turnovers, rebounds, and free throws. After the publication of Moneyball, Michael Lewis' best-selling 2003 book on the numerical analysis of baseball, basketball stat-heads redoubled their efforts. Their hunt for a definitive, all-in-one performance metric—something analogous to baseball's wins above replacement, perhaps—yielded an array of competing stats, including: Efficiency, a simple tally that adds up a player's successes (like points) and subtracts his failures (like turnovers); Player Efficiency Rating, a more complex figure that assigns weights to these individual measures; and Wins Produced, a stat that assesses how various performance parameters correlate with team wins and then divvies up credit to players. These well-known measures are only the beginning: Today, the Internet is awash with stats sites seeking to describe basketball player talent in a more accurate, mathematical way.

But these metrics are limited by the available data. Not only does the box score fail to capture defense adequately—it tells us about blocks and steals and nothing else—but its tallies are polluted with hidden interaction effects. Take rebounds: If a point guard plays tight defense and forces his counterpart into a lousy shot, that act is probably credited to the center or forward who grabs the board. Rebounds also suffer from so-called "diminishing returns"—the idea that players on the same team effectively compete with one another for boards. Often a particular player—say, Minnesota Timberwolves center Kevin Love—serves as his team's designated glass-cleaner, and he scoops up balls that his teammates might well have grabbed anyway. Sometimes players work together as rebounding partners: During Jason Kidd's tenure with the New Jersey Nets, he grabbed a lot of boards thanks to his teammate Jason Collins, who would box-out the opposing team's best rebounder. "Year after year [Collins] had very low rebound numbers … but his team did really well rebounding when he was on the court," one stats expert who works for an NBA team told me. (A similar problem afflicts those who want to assess a baseball player's "range" in the field: The shortstop and third baseman compete for balls in the hole—if one guy gets to it, his teammate doesn't.)

This suggests that statistics that give players a lot of credit for rebounding prowess, like Berri's Wins Produced, may be weighing the numbers wrong. In past years, Berri's system has declared that rebounding studs like Ben Wallace have been the most valuable players in the NBA, ranking at the very top in Wins Produced. (Berri argues that reducing the importance of rebounds in his formula doesn't change its results.) Other box-score stats have different biases: John Hollinger's Player Efficiency Rating, for example, is often accused of overvaluing high "usage" players—ball hogs who take a ton of shots and hence score more points. But is a high usage rate a symptom of having a weak supporting cast or of a propensity to launch crappy shots? The stats don't say. Dan Rosenbaum, an economist for the Office of Management and Budget who also works as a consultant for the Cleveland Cavaliers, says the problem with many of these supposedly advanced stats is that they are "not empirical"—rather than rank players by value, they merely reflect their creators' sense of the importance of different facets of the game.

A statistic that appears less biased on its face is adjusted plus-minus. This is a measure, popularized by hockey, that reflects the difference between the total number of points a team scores and allows with a particular player on the floor. Analysts then adjust the figure to account for the fact that some players must toil on lousy teams. The greatest virtue of adjusted plus-minus is that it captures defense—you can see, for example, that Carmelo appears to be an elite offensive player but a subpar defender. The technique can also examine the impact of top rebounders: Kevin Love consistently rebounds in double digits, but his contribution to his team's total boards is only about two to three per game, according to one analysis.The chief analytic consultant for one NBA team told me that the impact of Love's rebounding might net a team only about three points per game. "The adjusted rebounding numbers seem to suggest that rebounding is vastly overrated," he said. Other "good" rebounders, such as the Celtics' Troy Murphy, who has averaged 8.5 boards a game over his career, appear to exert a negative overall effect on team rebounding. It might be that he's getting in the way of his teammates or failing to box out when he's not in on the play himself.

Or it could be that these stats just aren't any good. The biggest problem with adjusted plus-minus is that you need several years' worth of data to get a reliable signal. Once again, interaction effects are the issue: Coaches tend to use the same combinations of players over and over again, so certain guys will play together more often than others. A player who is in a lineup with LeBron James will benefit from that association and have a higher plus-minus as a result. And teams that don't recognize these flaws in the metric can get bitten. The Dallas Mavericks—with their science-loving owner, Mark Cuban—were early adopters of adjusted plus-minus. But they made some crucial player personnel errors, especially in devising playoff lineups, because they didn't fully appreciate the noisiness of the data, according to one NBA stats analyst who requested anonymity so that he could speak candidly about another team. Still, many NBA stats experts do use versions of adjusted plus-minus as part of their player evaluation system. Some front offices also look at Player Efficiency Rating to get a basic sense of offensive talent. Several NBA statisticians told me that they did not know of any franchise that relies on Wins Produced.

The world's top sports statisticians are working hard to devise better metrics. Last week, they gathered in Boston for the MIT Sloan Sports Analytics Conference. On Friday, one of the presentations touched on interaction effects in the NBA and could perhaps shed light on the alleged "Carmelo effect" (in which a top offensive player boosts his teammates' scoring). Economists Matt Goldman and Justin Rao looked at whether teams do a good job of distributing shots among players and whether individual players shoot too often or not often enough. Their model considers each basketball possession as a complex economic problem in which a team seeks to optimize its chances of scoring via two interlocking strategies: first, players must decide whether to keep the ball to themselves or pass to a teammate; then they decide, on a second-by-second basis, whether to take a shot at the basket or to hold the ball for a better opportunity.

Their analysis suggests that very few players in the league overshoot. Most elite offensive players such as Carmelo, they say, should be taking evenmoreshots. Shooting is always better than a turnover, after all, and leads to offensive rebounds in about 30 percent of misses. (According to the study, only a handful of players are legitimate overshooters, including the notorious chucker Monta Ellis.) What's more, an offensive superstar allows his teammates to shoot more selectively, because they can always pass him the ball late in the shot clock, with the confidence that he can use his athleticism to get off a decent attempt. Of course, this theory may be leaving out important elements of the game, such as player fatigue and the need to stay healthy.

So it's doubtful these data will resolve the Carmelo debate. Still, nearly all of the NBA stats experts I interviewed saw Melo in about the same way: an elite offensive threat who underperforms on defense, especially in zone coverage. One joked that it was surprising Carmelo had gotten so much attention, given that he was not even the best player traded before the deadline. That player would be Deron Williams, by a significant margin. (Shane Battier, the famous "No Stats All-Star" who was traded from the Rockets to the Grizzlies, also continues to rate well among the stats set.)

Today about half of NBA teams work with statisticians, and they tend to outperform those that don't. But the heyday of the all-in-one advanced-box-score stat may actually be behind us; coaches now chart players' strengths and weaknesses using services that slice up piles of game film into digestible pieces. That lets them scrutinize the quality of pick-and-rolls and investigate whether their power forwards are better attacking the basket from the left or right post. There are statistics involved, but in the end a flesh-and-blood human must sit there and fix his eyeballs on the tape.

Dean Oliver, the founding father of hoops math, who now runs the numbers for all of ESPN, says the main benefit of statistical analysis is simply that it lets coaches keeps tabs on more games. "The numbers do not see any individual game as well as a person. But they see all of the games," he said. In the next breath, though, Oliver stated with confidence that Dwayne Wade is the "most important guy to take away" on the Miami Heat—not LeBron James. "Not everybody knows that," he said—including many opposing coaches who appear to be keying on LeBron. So how did he know that? His computerized mathematical game-analysis tool, called Roboscout, told him. "It's not an obvious thing when you watch the game," he said. "But when you do the analysis, that comes out."

Correction, March 8, 2011: The original version was misleading about Dave Berri's assessment of the Carmelo Anthony trade. His prediction that the Knicks would win just 29 games with Carmelo was based on an earlier version of the deal that never came to pass. (Return to the corrected sentence.)

David Merritt Johns is a doctoral student with the Center for the History and Ethics of Public Health at Columbia University’s Mailman School of Public Health.

  News & Politics
March 31 2015 5:00 AM How the Founder of the Fugees Became a Big-Time Political Donor Without Anyone Knowing The musical artist chose to fund a super PAC through opaque, legal, and increasingly popular means.
  Slate Plus
Lexicon Valley
March 31 2015 9:17 AM The Redline of March Overheard on email: Slates copy desk rounds up the month’s style and grammar rulings.