Draft Accuracy Matters

Super short summary: more accurate drafting is more effective drafting.

Summary for Statheads: Improving the draft accuracy of a single team improves the quality of draft choices picked across the entire draft. Simulations at draft error levels of 0.8 and 0.6 rounds respectively show that the effect is on the order of 7 and 5 picks. In other words, someone picking 12th at a noise level of 0.8, that picks twice as accurately as the norm, has picks equivalent to a team slotted into the 5th position. At a noise level of 0.6, their picks would be equivalent to someone picking in the 7th position. The implications of these findings based on PFR’s approximate value stat and draft round are discussed.

Recently, we posted data showing that the draft error of NFL teams can be estimated based on the kinds of reaches observed in the draft, and our estimated range of error was from 0.5 to 1.0 rounds of error per draft pick. Taking these ideas further, I wanted to examine what would happen to a team that picked twice as accurately as its peers. By accurately, the error of its scouts are half  that of all the other teams. What advantages would they gain?

Figure 1. Pick improvement as a function of round at draft error = 0.8 round

Figure 2. Pick improvement as a function of round at draft error = 0.6 round.

The charts above plot “effective draft position” (i.e. improvements in the value of draft picks, as ranked by the draft position, or slot, they should have been picked) as a function of round, for teams with improved drafting ability. This term can be converted, using Pro Football Reference’s formula for estimated approximate value per slot, into a difference in estimated approximate value for such a choice, and those plots are given below.

Figure 3. AV improvement as a function of round at draft error = 0.8 round.

Figure 4. AV improvement as a function of round at draft error = 0.6 round.

That these results are not unique to these particular error levels is also true, as we calculated estimated AV improvements for a team picking 10th and one picking 20th at error levels of 0.4 as well. 0.4 is so low, in my opinion, as to be unbelievable, but even  then, you can see advantages to the team that drafts well.

One last point. Notice the jump in advantage from the 3rd to 4th round using our model of drafting? That jump is a function of less intense drafting of those players whose first ranking is less than 8.0, and therefore a product of a specific feature in the model. The notion that good teams improve as scouting resources become more scarce is not.

Good teams should  be expected to do markedly better the fewer scouting resources are applied to each player. Where that happens in the real NFL is beyond the scope of this study, but that it almost certainly does happen seems evident. Teams that are expert at drafting will show their expertise more and more as the draft goes on. Or, said another way, anyone with a copy of USA Today or an ESPN Insider subscription can draft a first rounder. It takes really good teams to take best advantage of late round draft choices.

 

Head scratching moments: just how noisy is the NFL draft?

Super short summary: Head scratching moments in the NFL draft are useful clues to the average error in the draft.

Summary for statheads: A simple, efficient market model of drafting can account for commonly observed reaches in the first round if the average error per draft pick is between 0.8 and 1.0 round. The model yields asymmetric deviances from optimal drafting even when the error is itself described by a normal distribution. This model cannot account for busts or finds; players such as Terrell Davis, Tony Romo, or Tom Brady are not accounted for by this model. I conclude that drafting in the most general sense is not efficient even though substantial components of apparent drafting behavior can be analyzed by this model.

Introduction

 There are 4 typical ways to describe a draft choice. The first is by the number of the choice (Joe Smith is the 39th player chosen in the draft). Second, by a scale, usually topping at 10, and going down one point for every round of change. In such a situation  the ideal player is a 10.0, a very promising player a 9.0, a first of the third round a 8.0, and so forth. Ourlads uses a similar device to rank players as draft candidates.  The third way to rank a draft candidate is by the market value of the slot taken, and the best known representative of that kind of methodology is Jimmy Johnson’s trade value chart. The fourth way to rank a draft choice is by the historically derived value of players drafted at that position, and Pro Football Reference has done that here. Note: another interesting attempt at an AV value chart is here.

Jimmy Johnson's trade value chart

Every draft has a moment where you see a player drafted, and you wonder what drove a team to take this player. In the 2011 draft I can recall off the top of my head at least three four head scratching moments: the draft by San Francisco of Aldon Smith (Ourlads  8.99, but rising), by Tennessee of Jake Locker (Ourlads 9.15, considered by many to be late first, second round) , by Seattle of James Carpenter (Ourlads 7.05),  and the draft by New England of Ras-I-Dowling (Ourlads 7.82, but perhaps scheme related). All four left me wondering. Perhaps the same do to you, perhaps they don’t. But what I’m getting at is the number of these moments defines an error level by its recognizable tails, and using that, we can back track to an estimate of the actual error involved in selecting players.

If, say, the first round of 2011 was typical of all rounds of the NFL draft, and there was at least one truly puzzling reach in every round of the draft, and let’s say the puzzlers involved a reach of at least a round or more of value in the draft, then any noise model of the NFL draft has to be at least that noisy, else it is unrealistic. If, for the sake of argument we’ll assume the baseline draft model is efficient, then we add the assumptions that there are no systematic errors in drafting, and that drafting errors are normally distributed. So, if there is 1 error per round of 1.0 rounds or more, then there should be 7 in the whole draft, and 7000 in 1000 simulated drafts. We set out to build a simulator and test  these principles.