I’ve gotten some interesting feedback with regard to my first noise simulation study (see here and here), and wanted to touch base on some ideas, and then get to a point I actually consider important. I’ve been talking about draft noise, or scouting error, or draft error, without exactly defining the phrase. I’m probably not going to satisfy the definition hounds here either, but I’ll approach the notion of value and draft value and then say what I’m calling draft noise might also be fruitfully called draft variability, since really it’s the variability of the value of a potential draftee that we’re after.
We know this exists. We know draft value, in this context, changes. This value could be related to athletic potential, but that’s another abstract quantity that isn’t measurable, and misses the point that the draft is a market. And as it’s a market, the currency is the draft slot used to ‘purchase’ a player. Pretty simple, huh? Just as you might pay 50 cents to purchase an apple, Reggie Bush is worth the #2 draft slot. That’s Reggie Bush’s value, in the context of the draft.
One approach for getting at the value and variability of players, we discussed here in the context of Monte Carlo draft simulations. Simply collect a group of scouts and let them all rank the players, take the mean and calculate the standard deviations of their estimates and you’ve got a normally distributed estimate of the draft variability of the player. Another was the ‘model the envelope of the apparent noise’ approach of the first simulation study. Rick Reilly uses a third approach in his recent re-draft article, which I believe really isn’t valuing the draft as a marketplace. He’s assessing the player’s performance after they were drafted. Mel Kiper’s language in this article is not only true, but utterly on the mark.
the obvious complaint we always hear is, “You can’t really grade a draft for a few years.” Not true. You can’t assess the performance of the players drafted for a few years, but you can assess the degree to which teams maximized value while filling needs.
To flip back to an analogy, I buy an apple. I pay 50 cents for the apple. I bite into the apple and get bruised flesh in my bite, and I say, “That was a lousy apple for the price.” That assessment doesn’t change the price. The price was 50 cents. Since the price exists, the kinds of analysis that Mel does is legitimate. He’s not analyzing the post draft performance of a player, he’s analyzing the market, and too much “draft” analysis forgets what the draft is.
One critique of my previous noise model is that there wasn’t a unique valuation for each individual team. Well, this is the deal. That’s per-player variability built into the model, and misses the point of the original study, which was to get a rough measure of the variability. However, the critique points out something else, and that draft noise, no matter how accurately teams scout, is never completely going away. In a word, it’s irreducible.
Let’s pick a player out of the blue, call him, oh Von Miller, and say that scout A from team A and scout B from team B each rank Von Miller as the 4th best athlete of 2011. But the scout from team A says, “We look for large linemen and large linebackers and so Von Miller isn’t big enough. On our scale, he’s worth a 15th pick.” Scout B from team B says, “We play a Miami 4-3, and we optimize our teams for pursuit and speed, and Von Miller is faster than greased lightning. We rank him 3rd, because he is a perfect fit to our requirements.” And therein lies a source of variability that will never go away.
Teams have physical and athletic requirements based on the offenses they play and the kinds of players they know can play in them. Scouts are taught to seek and value players based on those requirements. This creates variability in the market, hence noise. As we’ve said previously, that noise can be exploited. And that noise is never going away, no matter how accurately scouts slot players to their system.