Over the past four years, Aaron Schatz and his staff at Football Outsiders have applied the same principles of critical thinking to professional football that Bill James used to revolutionize baseball analysis two decades ago. Pro Football Prospectus 2007, their third annual NFL guide, analyzes every team and is loaded with research on everything from player development to on-field strategy. In today’s excerpt, the last in a three-part series, Schatz explains why it’s so difficult to analyze NFL games and how the league itself may be standing in the way of a deeper understanding of the pro game. On Tuesday, he argued that no NFL team will ever emulate the 2006 Indianapolis Colts. And Wednesday, Schatz used Football Outsiders’ advanced stats to identify some surprise playoff contenders.
When I first started Football Outsiders, we derived all of our statistics from the NFL’s official play-by-play sheets. Our formulas were different from what had come before—we looked at individual plays rather than total yardage, and we adjusted for context and strength of schedule—but every bit of data that we studied was in the public domain, available to anyone who cared to analyze it.
We gradually realized that the official play-by-play had serious limitations. Statistics like yards after catch and quarterback hurries would show up on NFL telecasts but were nowhere to be found in the league’s stat packages. There were also very few statistics for defensive players and none for offensive linemen.
Since the NFL wouldn’t provide the details we needed, we started providing them ourselves. Thus began the Football Outsiders Game Charting Project: Over the past two seasons, more than 50 volunteers have taped and analyzed every game broadcast, tracking everything from pre-snap formations and blitz patterns to yards after catch. To put our project into perspective, the NFL provided more than 54,000 lines of play-by-play information last season. Our goal is to add several layers of detail to nearly all of those 54,000 lines.
The game charting project has been a huge success, creating plenty of new statistics for us to use on our Web site and in Pro Football Prospectus. What’s frustrating, however, is that we could be doing so much more. Who’s standing in the way of advanced analysis of NFL games? Strangely enough, it’s the NFL itself, which continues to withhold from the public the best footage of pro football games.
The game charts that we generate from standard TV broadcasts are loaded with data. Our charters start with formation, marking down the number of running backs, wide receivers, and tight ends on each play. Thanks to this research, we know that Seattle lined up without a tight end more often than any other team last season. We also know that reports that this year’s Steelers will open up their offense with more four-wide receiver sets are a little silly; the 2006 Steelers used four-wide formations more often than any other team.
Thanks to game charting, we can identify strengths and weaknesses that don’t show up in conventional stats. We know, for example, that the Indianapolis defense faced 50 percent more draw plays than any other defense, an example of offenses taking advantage of the Colts’ overaggressive pass rush. But when it came to harassing quarterbacks, that overaggressive pass rush did its job. Conventional wisdom said Colts defensive end Dwight Freeney had a poor 2006 season—after registering at least 11 sacks in each of his first four seasons, he had only 5.5 last year. Our game charters knew differently. While sacks are a well-known stat, there is no official tally of hurries—the times a rusher forces the quarterback to run or throw before he’s ready. Football Outsiders’ charters filled that void and figured out that the league leader in quarterback hurries last year was … Dwight Freeney.
Another large swath of the game that’s missing from the official play-by-play is pass coverage. Tackling numbers are useless when it comes to identifying good defensive backs; when a DB makes a tackle it’s usually because he just failed to prevent a reception. Interceptions are too rare to serve as a good evaluation tool—it would be like judging a baseball player solely on triples. There is an unofficial statistic called “passes defensed,” but these are notoriously inconsistent from one official scorer to another. (The scorer in Philadelphia often gives cornerbacks credit for “defending” a pass that went 10 feet over the receiver’s head.)
To get better data, game charters determine which defender was primarily responsible for covering the receiver at the time of the throw. If the defense was playing zone coverage, we try to figure out who was responsible for the location where the ball was thrown, regardless of whether the pass was complete or not. The coverage stats we’ve generated from game charting have significantly advanced our ability to rate defensive backs. In Pro Football Prospectus 2006, we raved about a little-known safety in Arizona named Adrian Wilson. The rest of the world learned who Wilson was in 2006, and he made his first Pro Bowl. Before last year’s postseason, we wrote about how poorly New Orleans cornerback Fred Thomas performed in our coverage stats. Over the next two weeks, he got burned by both Philadelphia and Chicago for long touchdowns.
The information we’ve collected on pass coverage has been vital to understanding the reasons some defenses succeed and others fail. That’s why it’s incredibly frustrating that this data is so difficult to collect. Broadcast camera angles often don’t show the secondary, making it impossible to identify whether the defense is using man or zone coverage. On passes longer than a few yards, the camera won’t show the receiver until the pass is in the air. And there’s usually no way to tell if a safety is responsible for deep help on a long pass reception unless the network shows a replay from a different angle.
Pass coverage isn’t the only thing that’s tough to discern. The sideline angle that’s used in most television footage makes it hard to track offensive formations. Most teams put a player’s number on his shoulders, not his sleeves, and they don’t often show players from the back or front before a play unless it’s a close-up on the quarterback’s face. If the announcers don’t say something, it can be hard to tell if the guy at the top of the screen is a receiver or a running back. Cutaways to the sidelines or to the studio for news updates make tracking pre-snap motion completely impossible. It’s also impossible to analyze run-blocking and defensive formations using the camera angles on standard television broadcasts.
There is a way that we could track everything we want, including defensive coverage and the battle at the line of scrimmage, with far more accuracy. Every single play in every NFL game is filmed from both the sideline and the end zone perspectives. This “coaches’ film” is then distributed throughout the league to help teams prepare for future opponents. Only NFL teams and NFL Films, the league’s in-house production arm, are given access to the film. The only place it’s ever shown to the public is on the NFL Network and the ESPN show NFL Matchup.
Anyone who has watched Matchup knowsthe benefits of watching coaches’ film. Since the sideline camera is placed much higher than standard television cameras, you can see all 22 players at all times. Each player’s number is visible, so it’s elementary to identify each formation. It’s also much easier to identify defensive coverage and the cause-and-effect of actions on the field is clearer. On television, run blocking looks like a big pile of players smashing into each other; the end-zone perspective illuminates the interplay of the offensive and defensive lines, making it easier to identify traps or pulls by individual linemen. With coaches’ film, we could do a better job assessing individual offensive linemen and track how well defenses play in both nickel and dime formations.
The NFL has surpassed every other sports league when it comes to turning a filmable event into profitable entertainment. Yet for some reason, the league is sitting on hours of footage that dedicated fans—and dedicated statisticians—would kill to see. Broadcasts that incorporated the official coaches’ film would be hugely popular with hard-core fans, even if the league showed these enhanced telecasts only during the offseason. Why not package the coaches’ film for each team’s 16-game season as a DVD set? The league would make a huge profit, devoted fans would have a great souvenir, and football analysis would take a huge step forward.
In baseball, statistical analysis started outside the game and worked its way in. Fans like Bill James and Pete Palmer made up their own stats and created communities of interested fans. Slowly, the national media took note of their work and then it infiltrated front offices thanks to general managers like Billy Beane and Theo Epstein. In the NFL, advanced analysis works in the opposite direction. Teams have been using game film for years to learn new things about how the game is played, and many teams have statistical analysts on staff. Fans can’t do similar work because they just don’t have the data. Our goal at Football Outsiders is to turn that around. I’m confident that we can make it happen, with just a little help from the NFL.
Interested in joining the Football Outsiders charting project? Send an e-mail to Bill Moore at charting@footballoutsiders.com.
