The draft comes along, and you have pick number 16. How much benefit do you gain from all that intelligence gathering? Suppose, if you like, that your scouts are absolutely correct, that they have the players ranked perfectly.
Well, if Andrew is available when your turn comes along, you snap him up for a gain of seven spots. But that's not guaranteed, because, after all, you're not the only team doing scouting! If any one of the teams drafting from ninth to 15th came to the same conclusion, they've already grabbed him. In that case, your benefit from all that scouting winds up being ... zero.
What if Bob is available when your turn comes along? Well, you're going to pass on him, because you know he's not that good. But, if you hadn't done the scouting, you would have taken him with your No. 16 pick. You would have had a loss of 14 spots.
In the case of Bob, your intelligence did help you. It helped you a lot. And, it doesn't matter if other teams scouted him. Even if every other team reached the same conclusion you did, you've still saved yourself a big mistake by scouting him too. If you hadn't scouted him, you would have made a big mistake.
The moral: You gain more by not being stupid than you do by being smart. Smart gets neutralized by other smart people. Stupid does not.
If you're still not convinced, try this. I gather 10 people, and show them a jar that contains $1, $5, $20, and $100 bills in equal proportions. I pull one out, at random, so nobody can see, and I auction it off. The bidding will probably top out at around $31.50, which is the value of the average bill.
I do it again, but, this time, I'm not that careful, and you get a glimpse of the bill. So does Susan, the stranger sitting next to you.
Well, if it's a $100 bill, you and Susan bid up the price to $99.99. Neither of you really benefit.
But, if it's a $1 bill ... neither you nor Susan bids. Each of you would have had a 1-in-10 chance of paying $31.50 for the bill and suffering a loss of $30.50. On an expected value basis, each of you gained $3.05 from your secret knowledge.
As I said at the Sloan Conference—well, I don't remember saying it, but someone else said I did—"one of the things that analytics can do really well is filter out the really stupid decisions."
What I was probably thinking, was something like this: If the 1980 Expos had had a sabermetrics department, they could have spent hours trying to squeeze out a couple of extra runs by lineup management ... but they would have been much, much better off figuring out that Rodney Scott's offense was so bad, he shouldn't have been a starter.
It works that way in your personal life, too. You can spend a lot of time and money picking out the perfect floral bouquet for your date, but you're probably better off checking if you have bad breath and taking the porn out of the glove compartment.
If it's true that sabermetrics helps teams win, I'd bet that most of the benefit comes from the "negative" side: having a framework that flags bad decisions before they get made.