Early Season Power Rankings - Follow Up
This is a follow up to last week's post on early season power rankings. In that post, I attempted to measure the accuracy of various NFL power rankings by comparing the team rankings after week 4 of the season to each team's remaining wins for weeks 5-16.
The original dataset was for seasons 2009-2012, and just looked at the week 4 rankings. I have now added seasons 2007 and 2008, plus a look at how the rankings after week 8 correlated with future wins. As a reminder, higher percentages are better, with 100% meaning perfect correlation between the ranking and future wins (the metric used is the Spearman rank correlation coefficient).
It's counter-intuitive, but has a simple explanation. As you get further into the season, most poll-based rankings tend to become retrodictive, rather than predictive. In other words, they "reward" teams for past wins, rather than judge them on their ability to create future wins. Bill Parcells statements not withstanding, a team's win-loss record is not always the best predictor of future success.
The original dataset was for seasons 2009-2012, and just looked at the week 4 rankings. I have now added seasons 2007 and 2008, plus a look at how the rankings after week 8 correlated with future wins. As a reminder, higher percentages are better, with 100% meaning perfect correlation between the ranking and future wins (the metric used is the Spearman rank correlation coefficient).
| Week 5 Ranking Correlation to Future Wins | |||||||
|---|---|---|---|---|---|---|---|
| ranking | average | 2007 | 2008 | 2009 | 2010 | 2011 | 2012 |
| espn | 47% | 55% | 42% | 51% | 55% | 43% | 39% |
| dvoa | 50% | 57% | 45% | 47% | 46% | 41% | 65% |
| ans | 40% | 50% | 42% | 51% | 15% | 49% | 32% |
| market | 54% | 68% | 36% | 67% | 45% | 52% | 56% |
| srs | 52% | 70% | 46% | 47% | 38% | 56% | 54% |
| Week 8 Ranking Correlation to Future Wins | |||||||
|---|---|---|---|---|---|---|---|
| ranking | average | 2007 | 2008 | 2009 | 2010 | 2011 | 2012 |
| espn | 46% | 58% | 42% | 51% | 43% | 41% | 41% |
| dvoa | 53% | 77% | 53% | 59% | 44% | 35% | 46% |
| ans | 49% | 70% | 47% | 57% | 38% | 39% | 42% |
| market | 55% | 62% | 50% | 62% | 56% | 55% | 42% |
| srs | 53% | 75% | 51% | 55% | 46% | 42% | 50% |
Keep it Simple
The performance of the Simple Ranking System (SRS) continues to impress, either matching or outperforming the more sophisticated DVOA and ANS models. The market ranking still appears to be the most accurate.
More data makes you dumber
Interesting to note that while all of the quantitative rankings improved going from week 4 to week 8 (particularly the Advanced NFL Stats efficiency model), the qualitative ESPN ranking took a small step backward. It's hard to read too much into such a small difference, but it is consistent with Ken Pomeroy's observations on the AP rankings for college basketball. For the 1990-2010 academic years, he compared tournament performance of the pre-season AP #1 to the AP #1 immediately prior to the tournament. In general the pre-season #1 fared better in the tournament (out of 21 seasons, the pre-season #1 won it all 6 times, compared to just 3 for the pre-tournament #1).It's counter-intuitive, but has a simple explanation. As you get further into the season, most poll-based rankings tend to become retrodictive, rather than predictive. In other words, they "reward" teams for past wins, rather than judge them on their ability to create future wins. Bill Parcells statements not withstanding, a team's win-loss record is not always the best predictor of future success.
Leave a Comment