This is something I’ve wanted to test ever since I got my hands on play-by-play data, and to be entirely honest, doing this test is the major reason I acquired play-by-play data in the first place. Linearized scoring models are at the heart of the stats revolution sparked by the book, The Hidden Game of Football, as their scoring model was a linearized model.
The simplicity of the model they presented, the ability to derive it from pure reason (as opposed to hard core number crunching) makes me want to name it in some way that denotes the fact: perhaps Standard model or Common model, or Logical model. Yes, scoring the ’0′ yard line as -2 points and the 100 as 6, and everything in between as a linearly proportional relationship between those two has to be regarded as a starting point for all sane expected points analysis. Further, because it can be derived logically, it can be used at levels of play that don’t have 1 million fans analyzing everything: high school play, or even JV football.
From the scoring models people have come up with, we get a series of formulas that are called adjusted yards per attempt formulas. They have various specific forms, but most operate on an assumption that yards can be converted to a potential to score. Gaining yards, and plenty of them, increases scoring potential, and as Brian Burke has pointed out, AYA style stats are directly correlated with winning.
With play-by-play data, converted to expected points models, some questions can now be asked:
1. Over what ranges are expected points curves linear?
2. What assumptions are required to yield linearized curves?
3. Are they linear over the whole range of data, or over just portions of the data?
4. Under what circumstances does the linear assumption break down?
We’ll reintroduce data we described briefly before, but this time we’ll fit the data to curves.
Linear fit is to formula Scoring Potential = -1.79 + 0.0653*yards. Quadratic fit is to formula Scoring Potential = 0.499 + 0.0132*yards + 0.000350*yards^2. These data are "all downs, all distance" data. The only important variable in this context is yard line, because this is the kind of working assumption a linearized model makes.
Fits to curves above. Code used was Maggie Xiong's PDL::Stats.
One simple question that can change the shape of an expected points curve is this:
How do you score a play using play-by-play data?
I’m not attempting, at this point, to come up with “one true answer” to this question, I’ll just note that the different answers to this question yield different shaped curves.
If the scoring of a play is associated only with the drive on which the play was made, then you yield curves like the purple one above. That would mean punting has no negative consequences for the scoring of a play. Curves like this I’ve been calling “raw” formulas, “raw” models. Examples of these kinds of models are Kieth Goldner’s Markov Chain model, and Bill Connelly’s equivalent points models.
If a punt can yield negative consequences for the scoring of a play, then you get into a class of models I call “response” models, because the whole of the curve of a response model can be thought of as
response = raw(yards) – fraction*raw(100 – yards)
The fraction would be a sum of things like fractional odds of punting, fractional odds of a turnover, fractional odds of a loss on 4th down, etc. And of course in a real model, the single fractional term above is a sum of terms, some of which might not be related to 100 – yards, because that’s not where the ball would end up - a punt fraction term would be more like fraction(punt)*raw(60 – yards).
Raw models tend to be quadratic in character. I say this because Keith Goldner fitted first and 10 data to a quadratic here. Bill Connelly’s data appear quadratic to the eye. And the raw data set above fits mostly nicely to a quadratic throughout most of the range.
And I say mostly because the data above appear sharper than quadratic close to the goal line, as if there is “more than quadratic” curvature less than 10 yards to go. And at the risk of fitting to randomness, I think another justifiable question to look at is how scoring changes the closer to the goal line a team gets.
That sharp upward kink plays into how the shape of response models behaves. We’ll refactor the equation above to get at, qualitatively, what I’m talking about. We’re going to add a constant term to the last term in the response equation because people will calculate the response differently
response = raw(yards) – fraction*constant*raw(100 – yards)
Now, in this form, we can talk about the shape of curves as a function of the magnitude of “constant”. As constant grows larger, the more the back end of the curve takes on the character of the last 10 yards. A small constant and you yield a less than quadratic and more than linear curve. A mid sized constant yields a linearized curve. A potent response function yields curves more like those of David Romer or Brian Burke, with more than linear components within 10 yards on both ends of the field. Understand, this is a qualitative description. I have no clues as to the specifics of how they actually did their calculations.
I conclude though, that linearized models are specific to response function depictions of equivalent point curves, because you can’t get a linearized model any other way.
So what is our best guess at the “most accurate” adjusted yards per attempt formula?
In my data above, fitting a response model to a line yields an equation. Turning the values of that fit into an equation of the form:
AYA = (yards + α*TDs – β*Ints)/Attempts
Takes a little algebra. To begin, you have to make a decision on how valuable your touchdown is going to be. Some people use 7.0 points, others use 6.4 or 6.3 points. If TD = 6.4 points, then
delta points = 6.4 + 1.79 – 6.53 = 1.79 + 0.07 = 1.86 points
α = 1.86 points/ 0.0653 = 28.5 yards
turnover value = (6.53 – 1.79) + (-1.79) = 6.53 – 2*1.79 = 2.95 points
β = 2.95 / 0.0653 = 45.2 yards
If TDs = 7.0 points, you end up with α = 37.7 yards instead.
It’s interesting that this fit yields a value of an interception (in yards) almost identical to the original THGF formula. Touchdowns are more close in value to the NFL passer rating than THGF’s new passer rating. And although I’m critical of Chase Stuart’s derivation of the value of 20 for PFR’s AYA formula, the adjustment they made does seem to be in the right direction.
So where does the model break down?
Inside the 10 yard line. It doesn’t accurately depict the game as it gets close to the goal line. It’s also not down and distance specific in the way a more sophisticated equivalent points model can be. A stat like expected points added gets much closer to the value of an individual play than does a AYA style stat. In terms of a play’s effect on winning, then you need win stats, such as Brian’s WPA or ESPNs QBR to break things down (though I haven’t seen ESPN give us the QBR of a play just yet, which WPA can do).
Update: corrected turnover value.
Update 9/24/11: In the comments to this link, Brian Burke describes how he and David Romer score plays (states).
