The NFL draft is a kind of auction, with auction-like dynamics. It’s also akin to a marriage. It only takes one, not a crowd, to get married and the opinion of the one outweigh the many. When analyzing the draft, I’ve been known to say things like between three players of the same true value, the one that gets drafted is the one whose value is most overestimated (1). I’ve also said things like one scouting opinion isn’t important, but the envelope of opinions is. The distribution of those opinions is crucial to knowing when a player can be drafted (2).

The distribution of player rankings can affect the possible draft positions of a player.

The distribution of player rankings can affect the possible draft positions of a player. Hand drawn curves on a brand new pen tablet, so they’re not perfect curves. Imagine the purple curve with more extensive tails.

In the diagram above, there are three distributions, with different peaks, means and spreads. Player A, in black, has a tight distribution of values and barring any issues with uniqueness of position, there is a consensus where he will be drafted. Player B, in red, has a broader distribution, but is unlikely to suffer more than half a round of variance in draft position. Player C, in purple, has an envelope encompassing two whole rounds of the draft. It’s the player C types that create a lot of controversy.


Travis Frederick and ZZ Top: Separated at birth?

The player the Dallas Cowboys drafted in the first round, Travis Frederick, is exactly one of those types. He was highly ranked before the combine, but suffered because of his bad 40 time. People like Gil Brandt, who had him ranked 27th best at the time, dropped him because of his 40 time. Perusing various links, such as this one, you see rankings ranging from 31 (Gary Horton) into the 90s. Now please note that draft pundits really don’t count, NFL teams do. But for the sake of argument, we’re using media scouts as an estimator of the envelope of NFL opinions. And that envelope of values encompass two whole rounds of variance.

So, what happens when you must have a player whose valuation envelope is a broad distribution? This player must be taken pretty far from the mean, in the tails of the “high value” side, or else you risk losing him (3). What is guaranteed though, is that the pundits on the other side of the mean from you will undoubtedly scream bloody murder. That’s because a draft pundit’s opinion is his life’s blood, and they make their money validating and defending that opinion, usually in print, and sometimes on television. That it’s one of many doesn’t matter if that’s how you make your living. So of course pundits will scream.

2013 was a draft with few good players. If estimates are valid that there were only 16 or so players truly worth a first round pick, then by default you’re overdrafting your quality by a half round by the middle of the first round. If the span of Frederick’s valuations really ran from, say, mid second to the beginning of the fourth, then the so-called overdraft is not, it’s entirely the function of three things: first, the perceived need for the player and second, such a broad valuation envelope that Dallas had to draft him in the tails of the distribution. Third factor, the lack of talent overall in the draft that led to overdrafting in general.


1. Jonathan Bales, before he became a New York Times contributor, favored this comment (common sense, IMO) and used it to help validate a pet drafting theory of his. I never saw enough rigor in his theory to separate it from the notions of BPA or need, as it was more a collective efficiency concept. IMO the notion hardly led to the invalidation of BPA or needs based drafting.

2. In the early 2000s, I wrote a Monte Carlo simulator of the draft, which explicitly used those distributions to estimate where players would be drafted. More discussion of that code, released as a Sourceforge project, is here.

3. Let me note that in “must have” situations, teams whose draft record no one complains about .. New England say .. will draft players above their worth. Belichick’s rationale, given in the link, is instructive. An excerpt is:

Now, the question is always, “How much do they like him and where are they willing to buy?’ I’m sure for some teams it was the fourth round. For some teams it was the third round. But we just said, ‘Look we really want this guy. This is too high to pick him, but if we wait we might not get him, so we’re going to step up and take him.’

PS – tskyler, a Cowboys Zone forum contributor, has a very nuanced fan analysis of the Frederick draft here, one worth reading.

These are numbers that have been published before, but not presented as artfully as this. PFR has the average draft value of a draft pick on a per draft slot basis. They then find a representative player with that AV. Also, they’ve listed the best picks in the slot as well. Looking to pick a world beater?

The url for this page suggests that  the page is temporary. Hopefully it will become a permanent part of Pro Football Reference.

When trying to value drafts, we tend to think only in one direction:  how to get as much talent for the kinds of draft picks we have.There is another kind of optimization that often goes under the radar, and that is having the coaching talent and foresight to construct a winning offense that doesn’t require extreme athletes. If, for example, you can get the same caliber running game out of 4th round draft choices as other coaches would with a mid first round choice, you’ve lowered the cost of the offense (Mike Shanahan and his zone blocking-cutback running scheme). If you can get high quality play out of quarterbacks with modest physical skills by making their reads simpler and jobs easier, you’ve lowered the cost of your quarterbacks (the West Coast offense). If by looking for smaller players with plenty of speed, drafting linebackers from strong safety-linebacker tweeners, putting linebackers at defense end and defensive ends at defense tackle,  you markedly  increase  your team speed. Further, because you’ve fruitfully used so many tweeners, you’ve cut the cost of your defense (Miami 4-3, notes here and here and here. Coach Hoover talks about it here, defending the flexbone, and I’m pretty sure the Penn State defense, described here, is a derivative of this defense as well).

You can probably formalize the cost of an offense (or defense) by treating the draft as a market and assigning the players on a team their draft value, either by methods we touched on here, or a fit to a Weibull distribution, as shown in figure 1 of this manuscript, or by analogy using AdamJT13’s chart here. To note, the cost of a free agent in this context is zero, since no draft choice was spent purchasing them. I don’t claim ideas like these are original to me. On the site LiveBall Sports – very nice multisport site with a nice analytics bent – they  have a 2 part series (NFC and AFC) evaluating the usage of free agents, and the language of the author, Greg Trippiedi, makes it clear he’s thinking in terms of draft value. How valuable are these no-cost free agents? Please recall that in this article, we quote Bobby Beathard as saying the first Super Bowl team under his watch with the Redskins  had 26 free agents on the roster. But it also had excellent coaches, who could turn sow’s ears into.. well.. Hawgs.

Since a player  that makes a roster is occupying a slot that others could also occupy, I suspect a true valuation of the cost of a player would also have to include development time. If it takes 5 years for a player to become a starter (or major rotation player), there is the cost of his draft choice and the time cost of his development. Both need to be assessed in terms of his cost. A player that never starts, never plays and occupies space becomes a dead weight cost.

One final issue. Dynasties can’t be constructed with expensive players. Think about it. Dynasties don’t have particularly good draft position. Winning in the early years guarantees that. The average player lasts about four years. So in general, they will have a few elite players with long careers and a large corps of pretty good, inexpensive players. If costs of the team model can’t be lowered adequately, sustained winning can’t be achieved. Replacement players will simply come at an unsustainable cost.

I’ve gotten some interesting feedback with regard to my first noise simulation study (see here and here), and wanted to touch base on some ideas, and  then get to a point I actually consider important. I’ve been talking about draft noise, or scouting error, or draft error, without exactly defining the phrase. I’m probably not going to satisfy the definition hounds here either, but I’ll approach the notion of value and draft value and then say what I’m calling draft noise might also be fruitfully called draft variability, since really it’s the variability of the value of a potential draftee that we’re after.

We know this exists. We know draft value, in this context, changes. This value could be related to athletic potential, but that’s another abstract quantity that isn’t measurable, and misses the point that the draft is a market. And as it’s a market, the currency is the draft slot used to ‘purchase’ a player. Pretty simple, huh? Just as you might pay 50 cents to purchase an apple, Reggie Bush is worth the #2 draft slot. That’s Reggie Bush’s value, in the context of the draft.

One approach for getting at the value and variability of  players, we discussed here in the context of Monte Carlo draft simulations.  Simply collect a group of scouts and let them all rank the players, take the mean and calculate the standard deviations of their estimates and you’ve got a normally distributed estimate of the draft variability of the player.  Another was the ‘model  the envelope of the apparent noise’ approach of the first simulation study. Rick Reilly uses a third approach in his recent re-draft article, which I believe really isn’t valuing the draft as a marketplace. He’s assessing the player’s performance after they were drafted. Mel Kiper’s language in this article is not only true, but utterly on the mark.

 the obvious complaint we always hear is, “You can’t really grade a draft for a few years.” Not true. You can’t assess the performance of the players drafted for a few years, but you can assess the degree to which teams maximized value while filling needs.

To flip back to an analogy, I buy an apple. I pay 50 cents for the apple. I bite into the apple and get bruised flesh in my bite, and I say, “That was a lousy apple for the price.” That assessment doesn’t change the price. The price was 50 cents.   Since the price exists, the kinds of analysis that Mel does is legitimate. He’s not analyzing the post draft performance of a player, he’s analyzing the market, and too much “draft” analysis forgets what the draft is.

One critique of my  previous noise model is  that there wasn’t a unique valuation for each individual team. Well, this is the deal. That’s per-player variability built into the model, and misses the point of the original study, which was to get a rough measure of the variability. However, the critique points out something else, and that draft noise, no matter how accurately  teams scout, is never completely going away. In a word, it’s irreducible.

Let’s pick a player out of the blue, call him,  oh Von Miller, and  say that scout A from team A and scout B from team B each rank Von Miller as the 4th best athlete of 2011. But the scout from team A says, “We look for large linemen and large linebackers and so Von Miller isn’t big enough. On  our scale, he’s worth a 15th pick.” Scout B from team B says, “We play a Miami 4-3, and we optimize our teams for pursuit and speed, and Von Miller is faster than greased lightning. We rank him 3rd, because he is a perfect fit to our requirements.” And therein lies a source of variability that will never go away.

Teams have physical and athletic requirements based on the offenses they play and the kinds of players they know can play in them. Scouts are taught to seek and value players based on those requirements. This creates variability in the market, hence noise. As we’ve said previously, that noise can be exploited. And that noise is never going away, no matter how accurately scouts slot players to their system.

Once you have the concept of a drafting error in hand, and a fairly large one at that, you can ask questions  that have almost Murphy’s Law implications for hoary old theories such as BPA. Consider this scenario: you have three players in the middle rounds you are considering, whose “true career value” is about the same. We’ll assume drafting is an efficient market for now, so our estimation of the value of these picks is a normally distributed estimate whose mean is based off their true career value. Which one of these men do we draft? We draft the player whose value we have overestimated the most. Consequently, we draft the player most likely to underachieve our expectations.

Since in most drafts there are very few times a true BPA falls into the lap of teams (i.e. players where one is wildly superior to all other candidates), it would seem that BPA is a way of optimizing how heartbroken a team will be over the draft choices it actually picks. Though this approach would appear to gather the best athletes, in a draft with a large error, and multiple situations where you’re picking from nearly equivalent athletes, perhaps all BPA will get you is maximally suffering from buyer’s remourse.

Update: Brian Burke, on the blog Advanced NFL Stats, talks about exactly the same issue, except with free agents. This issue has a name, the “Winner’s Curse” and is a well documented problem with auction style transactions.

Super short summary: more accurate drafting is more effective drafting.

Summary for Statheads: Improving the draft accuracy of a single team improves the quality of draft choices picked across the entire draft. Simulations at draft error levels of 0.8 and 0.6 rounds respectively show that the effect is on the order of 7 and 5 picks. In other words, someone picking 12th at a noise level of 0.8, that picks twice as accurately as the norm, has picks equivalent to a team slotted into the 5th position. At a noise level of 0.6, their picks would be equivalent to someone picking in the 7th position. The implications of these findings based on PFR’s approximate value stat and draft round are discussed.

Recently, we posted data showing that the draft error of NFL teams can be estimated based on the kinds of reaches observed in the draft, and our estimated range of error was from 0.5 to 1.0 rounds of error per draft pick. Taking these ideas further, I wanted to examine what would happen to a team that picked twice as accurately as its peers. By accurately, the error of its scouts are half  that of all the other teams. What advantages would they gain?

Figure 1. Pick improvement as a function of round at draft error = 0.8 round

Figure 2. Pick improvement as a function of round at draft error = 0.6 round.

The charts above plot “effective draft position” (i.e. improvements in the value of draft picks, as ranked by the draft position, or slot, they should have been picked) as a function of round, for teams with improved drafting ability. This term can be converted, using Pro Football Reference’s formula for estimated approximate value per slot, into a difference in estimated approximate value for such a choice, and those plots are given below.

Figure 3. AV improvement as a function of round at draft error = 0.8 round.

Figure 4. AV improvement as a function of round at draft error = 0.6 round.

That these results are not unique to these particular error levels is also true, as we calculated estimated AV improvements for a team picking 10th and one picking 20th at error levels of 0.4 as well. 0.4 is so low, in my opinion, as to be unbelievable, but even  then, you can see advantages to the team that drafts well.

One last point. Notice the jump in advantage from the 3rd to 4th round using our model of drafting? That jump is a function of less intense drafting of those players whose first ranking is less than 8.0, and therefore a product of a specific feature in the model. The notion that good teams improve as scouting resources become more scarce is not.

Good teams should  be expected to do markedly better the fewer scouting resources are applied to each player. Where that happens in the real NFL is beyond the scope of this study, but that it almost certainly does happen seems evident. Teams that are expert at drafting will show their expertise more and more as the draft goes on. Or, said another way, anyone with a copy of USA Today or an ESPN Insider subscription can draft a first rounder. It takes really good teams to take best advantage of late round draft choices.

This is a follow up piece to my previous post on draft trends and football teams. I have some new charts, some new ways of looking at the data. I’ve found some new analysis tools (such as the fitting machine at What I don’t have — I’ll be upfront about this — is one true way to draft. The data that I have don’t support that.

We’ll start with some comments from Chris Malumphy of, almost all constructive. I wrote him about my work and he replied. He says in part:

What would also be interesting is how many “compensatory” picks are included in each team’s totals. I believe that New England is among the leaders in receiving compensatory picks (which were first awarded in 1994 or so). I’ve frequently suggested that compensatory picks are contrary to the initial purpose of the draft, which was to award high picks to the poorest teams, in the hope that they would improve. Compensatory picks typically go to teams that decide not to sign their own free agents, which often means teams like New England let relatively good, but perhaps overpriced or problematic players go, knowing that they will get extra draft picks in return. Typically, poor teams aren’t in the position to make “business” decisions like that.

Yes, that’s a really good point. I probably won’t be able to get to anything like that off the bat, unless someone suggests a good exhaustive resource for all compensatory picks ever awarded. Anyone have a guess?

I resheeted these data via rounds and it’s not as visually interesting a spreadsheet. In part, it doesn’t make the point that top 10 picks are almost inversely related to winning. The second issue is it appears there is something “magical” about the 181 and down draft bin that just sticks out when it is sheeted. And perhaps it’s a side effect of this “compensation pick” issue that Chris raises.

When I plot the data, we get an interesting trend line, but the confidence interval of the fitted slope parameter isn’t significant.

The fit is, as says (nice online tool for fitting small sets of data):

y = a + bx

Fitting target of sum of squared absolute error = 1.0770302448975281E+01

a = 6.5737988655760553E+00
b = 2.9998264270189301E+00

and the error analysis is:

Degrees of freedom (error): 30.0
Degrees of freedom (regression): 1.0
R-squared: 0.143164205216
R-squared adjusted: 0.114603012057
Model F-statistic: 5.01254287301
Model F-statistic p-value: 0.0327335138151
Model log-likelihood: -27.9829397908
AIC: 1.87393373693
BIC: 1.96554223085
Root Mean Squared Error (RMSE): 0.58014821514

Coefficient a std error: 2.13460E+00, t-stat: 3.07964E+00, p-stat: 4.40691E-03
Coefficient b std error: 4.23686E+00, t-stat: 7.08030E-01, p-stat: 4.84392E-01

Coefficient Covariance Matrix
[ 12.69186065 -24.87944877]
[-24.87944877 50.00139868]

I don’t know much statistics, but I do know that when the relative error of a fitted parameter exceeds 100% (and 4.237/3.00*100 = 141.2%), it’s not significant.

Take home? These data are useful for examining the draft strategies of select winning teams. They are not a mantra for how to draft. If you want to look in depth at the draft strategies of the Vikings versus the Patriots .. probably the two most extreme cases in the data set, you’re likely to glean some insight. But the draft methods of one team.. or even three or four.. aren’t the one and only way to win in the NFL.


Get every new post delivered to your Inbox.

Join 244 other followers