How do you judge a football team?
Wins and losses are the best way to rank them, but there has to be more involved in the best way to rate them. Games are made up of more than 100 plays; the best rating systems take every one of those snaps into account, not just the final score they produced.
This is one reason we use Bill Connelly’s S&P+ system (along with KenPom for basketball) a lot on this site. We spent the better part of a decade at SB Nation, where his Football Study Hall continues to provide a wealth of statistical knowledge.
A full explanation of S&P+ is available here; essentially it uses five factors to determine a team’s strength:
- Efficiency (measured by success rate: gaining 50% of the needed yardage on first down, 70% on second down, and 100% on third/fourth down)
- Explosiveness (measured by points-per-play of successful plays)
- Finishing Drives (measured by points per trip inside the 40 yard line)
- Field Position
- Turnover Margin
S&P+ should not be used to rank teams because it doesn’t care about outcomes. Right now, Ohio State is number one in its ratings. The system doesn’t dock the Buckeyes for having two losses; its only concern is the data from every snap. And play-for-play, it still views Ohio State as the best team in the country. This same system rated the 1993 Vols as the best of Tennessee’s decade of dominance; they tied Alabama and lost by seven in Gainesville, but had a school-record offense en route to winning nine games by an average of 37 points. The 1998 Vols, among others, were more successful. But the 1993 Vols were better play-for-play.
In coaching searches, we tend to focus on just the wins and losses. This is a good starting point, and even a good ending point when circling back around. But a good search is more thorough than simply one’s record. How well your team played (or didn’t play) en route to that record can separate a great prospect from a good one.
For example:
Butch Jones in S&P+
Five years ago, there wasn’t enough time between Charlie Strong’s no and Butch Jones’ yes for fans to form more than an off-the-cuff opinion of Cincinnati’s coach. But one of the initial positive reactions after his hire was, “He knows how to win close games.” And that was true. Moving on from Derek Dooley, the ability to win not just close but meaningful games was important. Butch Jones beat Virginia Tech 27-24 in 2012 and beat Louisville 25-16 in 2011. And Jones has won his share of close, meaningful games at Tennessee.
But looking a little closer, the number of close games Jones was playing should have stood out. In 2011, six of Cincinnati’s last eight games were decided by nine points or less. At Central Michigan, Jones and the Chippewas were in nine one possession games in 2008. Kudos to Jones for winning enough of them to get Cincinnati to 10-3 in both 2011 and 2012. But, as we found out at Tennessee and Florida found out with Jim McElwain, winning a bunch of close games is not the best indicator of one’s coaching ability. In S&P+, Derek Dooley’s 2012 Vols rated higher than Butch Jones’ 2012 Bearcats.
Using S&P+’s percentile performance for each game (data from Tennessee’s advanced statistical profiles at Football Study Hall and SB Nation’s Tennessee season previews), what were Butch Jones’ best performances at Tennessee?
- 2014 wins over Utah State, Kentucky, and Iowa all scored in the 99th percentile. The Vols won those three games by a combined 82 points, essentially representing the best Tennessee could play against those opponents relative to expectation.
- Bowl victories against Northwestern (96%) and Nebraska (94%) also stand out. Last year’s Music City Bowl is a good example of what S&P+ reveals: though the Vols won by only two touchdowns, they dominated statistically. Tennessee averaged 6.9 yards per play while giving up just 4.5 yards per play.
- In bigger games, Jones’ Vols had an 80+ percentile performance rating in the 2015 win over Georgia, the Battle at Bristol, and the 2016 win over Florida. They also showed up well in the 2014 loss at Georgia (another way S&P+ is useful: showing how well you actually played in a loss).
S&P+ also includes a win expectancy statistic: using those five factor stats in a given game, how often should you expect to win that game? Georgia in ’15 and Florida in ’16 both featured amazing comebacks, but statistically the Vols were superior in the end. The Vols had a 72% win expectancy against Georgia and 81% against Florida; Tennessee wasn’t lucky to finish those comebacks, they were the better team over the course of those sixty minutes.
But win expectancy cuts the other way against Butch Jones a lot, especially this year. The Vols got the win against Georgia Tech despite a 24% win expectancy in S&P+; given those five factor stats, Georgia Tech wins that game 76% of the time. You need wins like this here and there; good teams tend to also be fortunate. But win expectancy shows the rest of Tennessee’s close games this year weren’t really that close: it gave the Vols an 18% chance to win at Florida, 23% against South Carolina, and 33% against Kentucky. Even though all three of those games (plus Georgia Tech) came down to the final play, the Vols shouldn’t be considered unlucky to have lost them because statistically, they weren’t that close at all.
Walk it back to 2016, and you’ll find something similar. It’s not surprising to see a win expectancy below 50% against Georgia when you won on a hail mary. But close games with Texas A&M (22%) and South Carolina (19%) really weren’t that close on a play-for-play basis. Outcomes like these repeatedly gave Butch Jones a “one play away” narrative. But when you start taking every play into account, the Vols weren’t nearly as close, and usually ended up with the outcome they deserved.
The lesson: look beyond wins and losses, and look out if you find a bunch of close wins. Hire a coach who does a better job winning every single snap.
Two of Tennessee’s top targets do a good job of this:
The Long-Term: Dan Mullen
Here’s the Mississippi State advanced statistical profile. From 2014-2017, Butch Jones went 29-20 at Tennessee. Over the same span, Dan Mullen is 32-16 at Mississippi State. In those four years Jones has an average S&P+ percentile performance rating of 61.9% (which does not include data from the loss at Missouri). Mullen’s S&P+ percentile performance average is 69.2% (which does not include data from the loss to Alabama). Using this metric, the Vols were actually better play-for-play relative to expectation in 2014 (69%) and 2015 (70.3%) than 2016 (65.9%).
With percentile performance, the top team isn’t at 99% every week. No one is that perfect. Right now Alabama is at an 87.2% average for the season (90+% in every game but Florida State (80%), Texas A&M (68%), and LSU (68%); Mississippi State will join that list). But Dan Mullen has done a good job getting more from his teams on every play, and this has been true over a number of years.
At Mississippi State, there have been times the Bulldogs have simply been over-matched. We saw that with Auburn and Georgia this year (win expectancy: 0%). And yes, they weren’t sharp against UMass (58% percentile performance with 65% win expectancy). But in every other win this season, their win expectancy is either 99 or 100%. Last season they struggled in going 6-7, losing a game to South Alabama they should have won and beating Miami (OH) in a bowl game they should have lost.
2015 is a good example of the Mulllen resume: four losses to the four most talented teams on their schedule (LSU, Texas A&M, Alabama, Ole Miss). But at a decided talent disadvantage, the Bulldogs played well relative to expectation, including a 76% percentile performance in a two point loss to LSU. And in eight of their nine wins, they performed at 80+%.
In 2014, the year Mullen took them to number one, the Bulldogs took full advantage in their 9-0 start: other than a 65% percentile performance in a 14 point win over Kentucky, MSU scored between 82-98% every Saturday. They still were above 50% at 61% in the loss to Alabama, then hit 99% in beating Vanderbilt the next week. They fell off in the Egg Bowl and the Orange Bowl, but still, the Bulldogs earned every bit of the success they found in 2014.
All of this to say: Dan Mullen’s teams have consistently been better play-for-play than what Butch Jones has done at Tennessee or any point in his career. Other than last year’s loss to South Alabama, when they get beat it’s largely because the other team is better. But when they win – and Mullen has won a lot for Starkville – those wins are earned on every snap. The Bulldogs haven’t been lucky. They’ve simply been good relative to expectation. Ask Vegas: in nine years Mullen has only lost as a favorite seven times, three of those as a favorite of three points or less.
In the current full S&P+ ratings, Mississippi State is 18th nationally and the highest-rated 7-3 team in the nation. They’ve put a better product on the field every snap than Washington State (21st), Virginia Tech (24th), Memphis (30th), West Virginia (33rd), and Michigan State (39th), among others. They have the same record as Kentucky, but the Wildcats are 76th in S&P+. This system knows.
In 2015 the Bulldogs finished 13th in S&P+, the strongest 9-4 team in the nation (the Vols were second at 20th overall). In 2014 they finished ninth overall. His teams tend not to get rewarded in the final AP poll because they play in the SEC West with a handful of losses to teams significantly more talented than them. But even in those games, MSU’s performance has often been noteworthy. And in the games they do win, they have done a much better job taking advantage on every snap than what we’ve seen in Knoxville the last five years.
This is the whole question with Mullen: will Tennessee make the difference? Can you make up the talent gap in Knoxville to compete not so much with Alabama right now, but Florida and Georgia? Does Mullen believe moving from Starkville to Knoxville significantly raises the ceiling? What does getting the most out of a Tennessee team look like?
If he believes Tennessee will make the difference, I believe he can make the difference for Tennessee.
The Short-Term: Scott Frost
Small sample size, but if you want to see what taking full advantage on every snap looks like, it’s Central Florida this year.
This guy isn’t the mid-major flavor of the month. His team is 8-0 with an average percentile performance of 88% (better than Bama). Their win expectancy has been 93-100% every week. They beat Mike Norvell and Memphis 40-13. Against Chad Morris and SMU the score was only 31-24, but they dominated statistically and still had a 78% percentile performance, and that’s their worst of the season.
It’s only eight games on top of the 13 last year where he moved Central Florida from 0-12 to bowl eligibility. But Scott Frost does not mess around.
They won at Navy 31-21 earlier this year; the Midshipmen are pesky and that was a solid road win. In the postgame, Frost was asked if he enjoyed seeing how his team reacted in a tight game:
“Heck no. I lost a year off my life tonight,” Frost said Saturday after the Knights’ 31-21 victory over Navy. “You need close games because you need to see how your kids experience those situations, but they’re not fun. It was stressful.”
This is a game they won by 10, on the road.
First of all, a coach who says, “Heck no,” will do great at Tennessee. Second of all, this quote is a great example of why Frost gets it, and why any team would be fortunate to get Frost: every snap matters. You coach to get the most out of all of them. Butch Jones coached to get enough to give his team a chance to win at the end; sometimes they did, sometimes they didn’t. The best coaches don’t leave it up to the last play because they start trying to beat you on the first play and don’t stop until the game is over.
Wins and losses will tell you how to rank them. But if you’re looking for how to rate them – and if you’re looking for effectiveness play-for-play – S&P+ is the more revealing tool. Right now, the only teams doing that better than Scott Frost and Central Florida are Ohio State, Alabama, Washington, and undefeated Wisconsin. (Again, how did you perform relative to expectation? The Badgers haven’t had the opportunity to get a signature win yet, but play-for-play they’ve done exactly what a good team should have done against their schedule to date). S&P+ has Central Florida fifth in their ratings overall.
If you’re looking for a larger sample size, what Dan Mullen has done at Mississippi State also fits the bill. Mullen may also be a longer-term answer for Tennessee, if Frost has dreams of the NFL. Teams looking for new coaches often make pendulum swing hires. The most important way to do that in moving on from Butch Jones is to look at more than just wins, losses, and especially close games. Who’s getting the most out of his team every play, every week, relative to expectation? Dan Mullen or Scott Frost would be a win in that department, and either would be a win for Tennessee.