This has been part of an ongoing conversation among Dallas fans, and perhaps among any of the 9 teams, from the Redskins to Patriots to the Vikings, that traded up in the first round of the 2012 NFL draft. There are some new tools for the analyst and the fan, and these include: (1) Pro Football Reference’s average AV per draft choice list, (2)  Pro Sports Transactions’ NFL draft trade charts, and (3) The Jonathan Bales’ article on Dallas Cowboys.com where he analyzes a series of first round trades up from 2000 to 2010. He concludes that in general, the trade up does not return as much value as it gives.

I suspect that Jonathan’s conclusion is also evident in the fantasydouche.com plot we reposted here. The classic trade chart of Jimmy Johnson really does overvalue the high end draft choices. You’re not paying for proven value, but rather potential when you trade up. I suspect by the break even metric we chose, comparing relative average AVs, that many draft trades never pay off, in part because people pay too much for the  value they receive. This is most evident in trading a current second or third and a future first for a current first round draft choice. These trades tend almost to be failures by design, and smack ultimately of desperation, true even when the player obtained (e.g. Jason Campbell) actually has some skills.

That said, how many of these players exceed the average abilities of the slot in which they were drafted? Now that we have the PFR chart, this is another question that can be asked of the first round players. Note that Jonathan Bales’ study doesn’t really answer the question of how good the player becomes, in part because the time frame chosen doesn’t allow the player adequate development. I started in the year 2000 1995, ended in the year 2007. I identified 67 players in that time frame, and I compared the AV for each player as given by the weighted average on the PFR player page. I’ll note that the player page and the annual draft pages do not agree on players’ weighted career accumulated value, so I assumed the personal pages were more accurate.

As far as a scale, we’re using the following:

AV relative to average Ranking
-25 AV or more Bust
-24 to -15 AV Poor
-14 to -5 AV Disappointing
-4 to +4 AV Satisfactory
+5 to +14 AV Good
+15 to +24 AV Very Good
+25 AV and up Excellent

 
Note there are some issues with the scale. Plenty of players from 1995 through 2007 are still playing, and their rankings are almost certainly going to change. In particular, Eli Manning at +24 and Jay Cutler at +23 have a great chance to end up scored as Excellent before the next season is over. Jason Campbell is at +19, and if he starts for a team for one season, he will end up with a ranking of Excellent. Santonio Holmes (+19) also has a shot at the Excellent category.

Players in the years 2006 and 2007 in lower categories (Manny Lawson at +7, Joe Staley at +4, Anthony Spencer at 0 ) could end up as Very Good, perhaps even Excellent if their careers continue.

The scoring ended up as

Scale Number Percent as Good Percent as Bad
Excellent 14 20.9 100.0
Very Good 9 34.3 79.1
Good 13 53.7 65.7
Satisfactory 10 68.7 46.3
Disappointing 7 79.1 31.3
Poor 5 86.6 20.9
Bust 9 100.0 13.4

 
Data came from the sources above. A PDF of these raw data is here:

NFL Trade Ups

Update: Increased the dates of players considered from 2000-2007 to 1995-2007. Moved Ricky Williams back to 1999.

There were, of course, two substantial trades of Ricky Williams. The first netted the Washington Redskins the whole of the Saints 1999 draft, plus the Saint’s first and third round picks of 2000. Three years later, Ricky was traded to the Miami Dolphins for a pair of first rounders, plus change. The first was obviously not paid off. How did the Miami Dolphins fare in their trade, using our new risk metrics?

Risk Ratio no longer makes sense as a term when you’re talking about someone already drafted. The important term becomes the net risk term, 52 AV. That’s 1 more AV than the typical #1 draft choice, and that’s the amount of AV Ricky had to generate in order for this trade to break even. And note, these calculations are derived from weighted career AV, not raw AV. So any raw AV we apply to these numbers is a rough approximation (A typical career summing to, say, 95 AV, might end up around 76 or so WCAV).

That said, Ricky Williams had a great first season with the Dolphins, generating 19 AV in that season alone. His total ended up somewhere around 57 AV. I’d suggest the second trade approximately broke even.

End notes: I’ve seen a lot of discussion around  this set of data, discussing the quality of draft picks on a per pick basis, posted in of all places, a Cav’s board. If this board isn’t the original source of these graphs, please let me know. An excellent resource for high quality NFL draft trade information is here. And finally, a reader named Frank Dupont writes:

I wrote a book about decision making in the NFL.  It’s sort of a pop science book because it seeks to make what happens in the NFL understandable via some work that people like David Romer, Richard Thaler, and Daniel Kahneman have done.  But because all pop science books make their point through narrative, I spend a lot of time looking at why football coaches are so old, but other game players like chess players and poker players are so young (Tom Coughlin is 65 and yet the #1 ranked chess player in the world is 21, the world’s best poker players are 25-ish).

The link for the book is here, if this topic sounds interesting to you. I’ll only note in passing  that while physics prodigies are common, biologists seem to hit their stride in their 60s.  Some areas of knowledge do not easily lend themselves to the teen aged super genius.

In my last post, I introduced ways to determine the risk of NFL trades, using Pro Football Reference’s average AV per draft slot metric to assess the relative risk of the trade. I wish to continue the work done in the first post, by also taking a look  at the Eli Manning trades, and also the Robert Griffin III trades.

Eli's debt will be paid off in 2 more years of his current level of play.
RG3 will have to have a Sonny Jurgenson-like career, all in Washington, to pay off the value of the picks used to select him.

In the Eli trade, the New York Giants assumed a ‘AV debt’ comparable to that of Michael Vick, and a relative risk approximately the same as Michael Vick or Julio Jones. Looking, Eli has  rolled up perhaps 87 AV at this point, netting 12-15 AV a year. So, in two years, in purely AV terms, this trade would be even. Please note  that NFL championships appear to not net any AV, so if the value of the trade is measured in championships instead, I’d assume the New York Giants would consider themselves the outright winners of this trade.

The appropriate comparison with the Robert Griffin III trade is actually the Earl Campbell trade. The risk ratio is about the same, assuming the Washington Redskins go 8-8 and 9-7 the next two years, and end up with the #16 pick in 2013, and  the #20 pick in 2014. The lowest  value the AV Given column could total is 106, if the Washington Redskins ended up with the #31 pick twice. In any event, the Redskins are betting that RG3 will have a Sonny Jurgenson-esque career, and not just his Washington Redskin career, but his Eagles and Redskins career, in order to pay back the ‘AV debt’ that has been accrued by this trade.

There were eight trades in the first day involving the first round of the 2012 NFL draft. Most of them involved small shifts in the primary pick, with third day picks added as additional compensation. The one outlying trade was that of the St Louis Rams and the Dallas Cowboys, which involved a substantial shift in  the #1 pick (from 6 to 14) and the secondary compensation was substantial. This high secondary compensation has led to criticism of the trade, most notably by Dan Graziano, whose argument, boiled to its essence, is that Dallas paid a 2 pick price for Morris Claiborne.

Counting  picks is a lousy method to judge trades. After all, Dallas paid a 4 pick price for Tony Dorsett. Was that trade twice as bad a trade as the Morris Claiborne trade?  The Fletcher Cox trade saw Philadelphia give up 3 picks for Fletcher Cox. Was that trade 50% worse than the Morris Claiborne trade?

In order to deal with the issues raised above, I will introduce a new analytic metric for analyzing trade risk, the risk ratio, which is the sum of the AV values of  the picks given, divided by the sum of the AV values of the picks received. For trades with a ratio of 1.0 or less, there is no risk at all. For trades with ratios approaching 2 or so, there is substantial risk. We are now aided in this kind of analysis by Pro Football Reference’s new average AV per draft pick chart. This is a superior tool to their old logarithmic fit, because while the data may be noisy, they avoid systematically overestimating the value of first round picks.

The eight first round trades of 2012, interpreted in terms of AV risk ratios.

The first thing to note about the 8  trades is that the risk ratio of 6 of them is approximately the same. There really is no difference, practically speaking, in the relative risk of the Trent Richardson  trade, or the Morris Claiborne trade,  or the Fletcher Cox trade. Of the two remaining trades, the Justin Blackmon trade was relatively risk free. Jacksonville assumed an extra value burden of 10% for moving up to draft the wide receiver. The other outlier, Harrison Smith, can be explained largely by the noisy data set and an unexpectedly high value of AV for draft pick 98. If you compensate by using 13 instead of 23 for pick #98, you get a risk ratio of approximately 1.48, more in line with the rest of the data sets.

Armed with this information, and picking on Morris Claiborne, how good does he  have to be for this trade to be break even? Well, if his career nets 54 AV, then the trade breaks even. If he has a HOF career (AV > 100), then Dallas wins big. The same applies to Trent Richardson. For the trade to break even, Trent has to net at least 64 AV throughout his career. Figuring out how much AV Doug Martin has to average is a little more complicated, since there were multiple picks on both sides, but Doug would carry his own weight if he gets 21*1.34 ≈ 28 AV.

Four historic trades and their associated risk ratios.

By historic measures, none of the 2012 first round trades were particularly risky. Looking at some trades that have played out in  the past, and one  that is still playing out, the diagram above shows the picks traded for Julio Jones, for Michael Vick, for Tony Dorsett, and also for Earl Campbell.

The Julio Jones trade has yet to play out, but Atlanta, more or less, assumed as much risk (93 AV) as they did for Michael Vick (94 AV), except for a #4 pick and a wide receiver. And although Michael is over 90 AV now, counting AV earned in Atlanta and Philadelphia, he didn’t earn the 90+ AV necessary to balance out the trade while in Atlanta.

Tony Dorsett, with his HOF career, paid off the 96 AV burden created by trading a 1st and three 2nd round choices for the #2 pick. Once again, the risk was high, the burden was considerable, but it gave value to Dallas in the end.

Perhaps the most interesting comparison is the assessment of the Earl Campbell trade. Just by the numbers, it was a bust. Jimmie Giles, the tight end that was part of the trade,  had a long and respectable career with Tampa Bay. That, along with the draft picks, set a bar so high that only the Ray Lewis’s of the world could possibly reach. And while Campbell was a top performer, his period of peak performance was short, perhaps 4   years. That said, I still wonder if Houston would still make the trade, if somehow someone could go back in to the past, with the understanding of what would happen into the relative future. Campbell’s peak was pretty phenomenal, and not entirely encompassed by a mere AV score.

Super short summary: more accurate drafting is more effective drafting.

Summary for Statheads: Improving the draft accuracy of a single team improves the quality of draft choices picked across the entire draft. Simulations at draft error levels of 0.8 and 0.6 rounds respectively show that the effect is on the order of 7 and 5 picks. In other words, someone picking 12th at a noise level of 0.8, that picks twice as accurately as the norm, has picks equivalent to a team slotted into the 5th position. At a noise level of 0.6, their picks would be equivalent to someone picking in the 7th position. The implications of these findings based on PFR’s approximate value stat and draft round are discussed.

Recently, we posted data showing that the draft error of NFL teams can be estimated based on the kinds of reaches observed in the draft, and our estimated range of error was from 0.5 to 1.0 rounds of error per draft pick. Taking these ideas further, I wanted to examine what would happen to a team that picked twice as accurately as its peers. By accurately, the error of its scouts are half  that of all the other teams. What advantages would they gain?

Figure 1. Pick improvement as a function of round at draft error = 0.8 round

Figure 2. Pick improvement as a function of round at draft error = 0.6 round.

The charts above plot “effective draft position” (i.e. improvements in the value of draft picks, as ranked by the draft position, or slot, they should have been picked) as a function of round, for teams with improved drafting ability. This term can be converted, using Pro Football Reference’s formula for estimated approximate value per slot, into a difference in estimated approximate value for such a choice, and those plots are given below.

Figure 3. AV improvement as a function of round at draft error = 0.8 round.

Figure 4. AV improvement as a function of round at draft error = 0.6 round.

That these results are not unique to these particular error levels is also true, as we calculated estimated AV improvements for a team picking 10th and one picking 20th at error levels of 0.4 as well. 0.4 is so low, in my opinion, as to be unbelievable, but even  then, you can see advantages to the team that drafts well.

One last point. Notice the jump in advantage from the 3rd to 4th round using our model of drafting? That jump is a function of less intense drafting of those players whose first ranking is less than 8.0, and therefore a product of a specific feature in the model. The notion that good teams improve as scouting resources become more scarce is not.

Good teams should  be expected to do markedly better the fewer scouting resources are applied to each player. Where that happens in the real NFL is beyond the scope of this study, but that it almost certainly does happen seems evident. Teams that are expert at drafting will show their expertise more and more as the draft goes on. Or, said another way, anyone with a copy of USA Today or an ESPN Insider subscription can draft a first rounder. It takes really good teams to take best advantage of late round draft choices.

Follow

Get every new post delivered to your Inbox.

Join 243 other followers