When it comes to analytics in the NFL, the Baltimore Ravens are No. 1.
That’s the take of analytics staffers around the league, as polled by ESPN in our first NFL analytics survey. Of course, it’s a tricky question. There’s some visible evidence of analytics work — on fourth-down decisions or draft-day trades, for example — but there’s even more that’s invisible to those outside each organization. There’s secrecy and competitive advantage involved, so no one knows exactly what everyone else is doing. Which is why we ask, because if anyone has a good sense of how analytics are playing a role, it’s the analytics staffers themselves.
So we surveyed 26 people who are current NFL analytics staffers or have been in the past year on a variety of topics during this past offseason — including the most and least analytically inclined teams — in what is no longer a niche area of football. (My list of analytics staffers around the league currently includes 85 people.) Some staffers left additional comments, and others were called by ESPN for contextual follow-ups. And all were granted anonymity so they could speak freely.
Note: 26 people were surveyed, though a few abstained from some questions. Participants were allowed to select their own team where applicable.
Which NFL team is the most analytically advanced?
1. Baltimore Ravens (15 votes)
2. Cleveland Browns (7)
T3. Philadelphia Eagles (1)
T3. Houston Texans (1)
Two voters abstained.
Which team most incorporates analytics into its decision-making?
1. Baltimore Ravens (14)
2. Cleveland Browns (6)
3. Philadelphia Eagles (2)
T4. Houston Texans (1)
T4. Indianapolis Colts (1)
Two voters abstained.
I grouped these two questions into one category because the Ravens won both and the runners-up were pretty similar. And given their fourth-down and 2-point aggressiveness — longtime staples of the analytics playbook — this was probably the chalk result.
“You saw how aggressive the Ravens were on fourth down last year. It’s because [head coach John] Harbaugh trusts the numbers, he trusts the analytics there. What Harbaugh has done is truly amazing. He has changed the culture of the team to believe in this stuff,” one staffer said, pointing out the video of a fired-up Lamar Jackson encouraging his coach to go for it on fourth-and-2 against the Seahawks last season and Harbaugh listening.
The ratio of Ravens win probability gains to win probability losses stemming from fourth-down decision-making was substantially higher than that of any other team in the league last season, according to ESPN’s model. The abridged version: They made better fourth-down decisions than anyone else. Houston and Philadelphia, which each received a single vote for most analytically advanced team, ranked second and fourth in the category, respectively (the Panthers were third).
Baltimore last season also ran play-action at the highest rate, another offensive choice that data analysis has revealed to be a significant advantage for the offense. Yes, the team’s style of offense leaves it less clear that the choice can truly be attributed to analytics, but it’s still something that we can look to as further evidence. The 49ers and Rams, two teams viewed as being on the upper end of the analytics spectrum (see below), ran play-action on the second- and third-highest percentage of plays, respectively.
Then there is the fact that the Ravens have one of the largest known analytics staffs in the league. They were early adopters, hiring their first director of analytics in 2012. A large staff can mean more specialization, and a longer history of analytics in the organization often leads to better work and better integration of data analysis into decision-making.
“It’s very rare for teams to start with analytics and move backwards,” said one high-level team staffer.
“For those teams in particular, there’s a commitment at the highest levels of the organization,” another NFL analytics veteran, lumping Baltimore in with the two other teams regularly considered to be in the top tier of NFL analytics, the Browns and Eagles. “Cleveland with Paul DePodesta, Philly with Jeffrey Lurie … and then in Baltimore with Eric DeCosta. Those guys are publicly advocating for analytics, and then they back that up with head count.”
But Baltimore was not a unanimous selection for either of these categories.
“I think most places, analytics acts like the 10th scout,” said one staffer, referring specifically to personnel decisions. “You’ll go around the room, and there’s nine scouts, and analytics will act like the 10th scout. Whereas I think the only two places where the analytics are the process instead of being just a cog in the wheel of the process are Cleveland and the Eagles. Maybe the Vikings.”
To that end, Minnesota pulled off the best draft pick-for-draft pick trade in the 2020 draft, according to our model, and cumulatively accrued the value of an additional second-round pick through its draft pick trades this year.
Which NFL analytics team produces the highest level of work?
1. Cleveland Browns (8)
2. Baltimore Ravens (5)
3. Philadelphia Eagles (4)
4. Buffalo Bills (3)
T5. Dallas Cowboys (1)
T5. Atlanta Falcons (1)
T5. Houston Texans (1)
T5. Indianapolis Colts (1)
Two voters abstained.
This is where the Browns stood out, beating out the Ravens, Eagles and Bills. Though the general belief is that analytics usage was dormant during the John Dorsey regime from 2017 to ’19, the Browns largely maintained the analytics group they grew under Sashi Brown, who was the team’s executive vice president from 2013 to ’17. And now many believe that group will play a larger role under new general manager Andrew Berry.
“I think that Cleveland is probably positioned to be the most analytically advanced in 2020 and beyond,” one staffer wrote. “They have a robust infrastructure in this area that survived multiple regime changes and is now aligned philosophically with their head coach and general manager.”
“Andrew Berry, he’s a true believer,” said another. “He’s one of these rare guys you’ll come across … in life where you think to yourself, ‘Man, this guy thinks at a different level. Just pure genius.’ He’s one of them.”
Which teams are among the top five most analytically advanced?
1. Baltimore Ravens (23)
2. Cleveland Browns (20)
3. Philadelphia Eagles (18)
T4. Buffalo Bills (7)
T4. San Francisco 49ers (7)
T6. Minnesota Vikings (6)
T6. Seattle Seahawks (6)
T8. Indianapolis Colts (5)
T8. Los Angeles Rams (5)
T8. New England Patriots (5)
T11. Dallas Cowboys (4)
T11. Miami Dolphins (4)
T11. New York Giants (4)
T14. Houston Texans (3)
T14. Jacksonville Jaguars (3)
T16. Atlanta Falcons (2)
T16. Kansas City Chiefs (2)
T18. Carolina Panthers (1)
T18. Denver Broncos (1)
T18. Green Bay Packers (1)
T18. New York Jets (1)
We asked voters to give their No. 1 choice and then pick the next four in any order. Two voters selected only four teams.
The goal here was to push past the generally agreed upon first tier of Baltimore, Cleveland and Philadelphia. Who’s next?
The answers were far more spread out, but some consensus did emerge, with the Bills, 49ers, Vikings and Seahawks earning the most votes among the others. What became clear is that there are so many aspects of a football team that data analysis can affect, and that different teams rely on analytics in different ways. One staffer surmised that the Rams were particularly analytically inclined in the draft because they have selected several players that their own team’s projection model has been high on. Some additional nuggets from survey-takers on other teams:
- “I think the 49ers have sneakily put together one of the top analytics teams in the league.”
- “The Seahawks … their sports performance is I think possibly the best in the league. I know people think they couldn’t be bought in to analytics because of some of their strategies, but their sports performance is phenomenal.”
- “The Cardinals are not known for analytics but have shown more commitment of late with some recent hires and an analytically inclined head coach.”
- “The New England one is interesting to me. I think they get credit from people that don’t actually know what it is they do. They have a lot of extremely intelligent people on staff. They’re able to come up with things similar to what analytics would produce, without actually doing what we’d call formal analytics.”
Which team is the least analytically advanced?
1. Washington Football Team (6)
2. Tennessee Titans (5)
T3. Cincinnati Bengals (3)
T3. New York Giants (3)
5. Pittsburgh Steelers (2)
T6. Houston Texans (1)
T6. Kansas City Chiefs (1)
T6. Las Vegas Raiders (1)
T6. Los Angeles Chargers (1)
T6. New Orleans Saints (1)
T6. Tampa Bay Buccaneers (1)
One voter abstained.
Many staffers said this was a more difficult question to answer than identifying the most analytically advanced teams.
“There’s a handful of teams, probably about 10, that I know essentially nothing about their analytics. Which leads me to believe they’re not doing a whole lot,” one experienced analytics staffer said. “So it could have been any of those 10 teams.”
Washington, which received the most votes, has one staffer listed in a hybrid analytics/scouting assistant role. The team ranked second worst behind the Packers in fourth-down decision-making in 2019, though that may change with new coach Ron Rivera, who is famous for his aggressive fourth-down choices. Carolina, his former team, had the third-best ratio of win probability gained to win probability lost on fourth downs last season.
Tennessee garnered the second-most votes, and it lists zero full-time analytics staffers on its website (though, to be fair, the Titans do run play-action at a high rate). The Giants were maybe the most stratified team in the survey, receiving four votes as one of the five most analytically advanced teams in the NFL and three votes as the least advanced.
While it may not be particularly clear who the least analytically inclined team might be, multiple staffers expressed the opinion that the difference between teams in consideration for the bottom of the list and those in the top tier (Baltimore, Cleveland and Philadelphia) or even the second tier (Buffalo and San Francisco, among others) was vast.
“There is a large spectrum of analytics work being done. You’ve got [a staffer on another team] with a Ph.D. And then you’ve got folks with analytics titles and they’re … not Ph.D.s,” said one person. “There really is such a disparity in technical capabilities.”
One further note: The Buccaneers, who received one vote in this category, recently hired Jacqueline Davidson as director of football research, and some surveys were taken before that.
If you had to guess, how many teams are building and using metrics based on player-tracking data?
9-16 (14)
0-8 (9)
17-24 (2)
25-32 (1)
Does your team build and use metrics based on player-tracking data?
Yes (24)
No (1)
One voter abstained.
Nearly 90% of respondents said 16 or fewer teams built and used metrics for player-tracking data, yet all but one staffer said their own team did. That’s quite the disconnect!
“Sounds like some bias there,” said one amused survey-taker upon hearing the results.
“We all think we’re better than everyone else,” added another. “I think it also shows people don’t want to say they’re not doing it. I wish I could know how much all those teams actually are utilizing them.”
Some important context has to be given to these numbers, however. This was most certainly not a random sample. A team without any known analytics staffers, for example, cannot have had anyone take the survey. And teams with more analytics staffers were more likely to participate in this survey — in some cases, multiple staffers from the same team took the survey. (That being said, the survey-takers did span more than half the teams in the NFL.)
Another caveat: The degree of usage matters. For example, a team might isolate speed measurements to similar situations in order to get apples-to-apples comparisons between players. Some might count that as building and using player tracking, while others may have felt that something more advanced — say, quantifying a cornerback’s ability to stick with his receiver in man coverage — was necessary to qualify.
The person who noted the bias earlier said they did not buy that other teams’ player-tracking usage was that significant. “There’s a difference between building and using it,” said the skeptic. “So we can build it, but is anyone using it? So the teams that answered yes, they’re partially telling the truth, I think, maybe?”
Either way, these numbers are a pretty clear indication that leaguewide adoption of player-tracking metrics is a little further along than many previously thought.
Which area of the game is most affected by analytics across the league?
Game management (10)
Coaching and opponent scouting (8)
College personnel evaluation (5)
Performance/strength and conditioning (1)
Pro personnel evaluation (0)
Salary-cap and contract analysis (0)
Two voters abstained.
Game management is the most visible usage of data analysis in the NFL, so it’s not too surprising that it took this category — though coaching/opponent scouting and college personnel evaluation did garner plenty of votes.
“In terms of points added, the most points you can add in terms of unit of time invested is probably like fourth-down decision-making or first-and-10 decision-making or something like that. But getting that from a meeting room in the facility onto the field is a pretty difficult task,” said one staffer, who noted that influencing front-office decisions maybe had less of an effect but was easier to achieve.
“They’re all valuable to different degrees based on the decision-makers,” said a second survey-taker. “You can have the best game-management person, but if the head coach isn’t listening, engaged, responsive … that’s not going to do your organization any good. Ultimately, the game-management decisions impact your product on the field on game day, which should be high organizational impact.”
That respondent also shared some insight on why staffers voted for college personnel as the most impacted area, but not a single person voted for pro personnel. He noted that a list of 1,200 yard-receivers in the NFL in a given season would all be pretty good players. But a list of 1,200-yard receivers would be much more of a mixed bag when thinking about how they translated to the pros.
Data analysis lends itself to adjusting for the talent disparity in college, which is less necessary in the more even playing field of the NFL.
If you had to guess, what do you think will happen to the number of analytics staffers employed by NFL teams over the next three years?
Increase by 10-50% (19)
Increase by greater than 50% (6)
Either increase or decrease by less than 10% (1)
Decrease by 10-50% (0)
Decrease by more than 50% (0)
Staffers were relatively bullish on growth in their field over the next few seasons, with most believing there would be a 10% to 50% increase — substantial. But since all of the people surveyed are in analytics, they to some extent have a vested interest in that growth.
Multiple people wondered if COVID-19 and the anticipated hit to NFL finances could slow down hiring in the short term. But ultimately, most figured teams would make more hires because it’s just worth it.
“In a salary-cap league, anything you can do outside of the salary cap is kind of a force multiplier,” said one survey-taker. “It’s not affecting your limited resources. And then even beyond that, if you look at what’s the marginal cost of a win in the NFL, it’s in the millions and millions of dollars. If you can bring in a team that can help you add even a small portion of that, they’re probably paying for themselves.”
Credit: Source link