I saw this article today, published a few days ago: PFF owner Cris Collinsworth on player grade criticism: ‘Thanks for the attention’.
We occasionally discuss PFF in this forum, and there are those like me who value their grades and others who don’t believe in them. So I thought it might be interesting for people to read the article and watch this video linked in the article: Collinsworth on Up and Adams show this week.
I think he makes a lot of credible points that support PFF’s grading system. I also believe him when he says that he welcomes critics in to see their process and believe that many of them realize it is a quality system.
I doubt anything in those links is going to change anyone’s opinion on PFF grades, but thought some of you might find them interesting.
I definitely think there’s value to what PFF does, but I personally take anything Chris Collinsworth says on the matter with a huge grain of salt. He’s way, wayyy too deeply invested – I don’t really expect him to come out and agree with players’ criticism of his platform that is a huge cash cow for him.
I tend to side with players when they take strong stances on this subject. If JJ Watt says he couldn’t give out defensive line grades without speaking to the player and coaches to understand the scheme and assignment of a given play, I trust him way more than I trust anyone within PFFs umbrella.
I thought Palazzolo and Monson did about as good of a job breaking this down as I’ve seen from someone who gained much of their notoriety from the success of PFF.
It’s probably easier to grade when things are “good” that fully understand what caused something to go “wrong” and who’s fault it is, particularly of the defensive side and sometimes in offensive blocking schemes. I like the idea of using PFF to showcase which players are ranked in the “Top 10” of certain categories, but I’ve seen too many inconsistencies with their grades and what I’ve seen on tape to believe it’s a solid measuring stick or grading rubric on it’s own.
Ryan Watkins has also made a great point that while he’s working on scouting reports, he constantly sees player grades “be revised” throughout his scouting process as he revisits games from the previous season, often times resulting in substantial swings. It provides further evidence that the grades are incredibly subjective to the grader – and thus inherently inconsistent.
Good post.
I think PFF benefits from the absence of any other source of grading player performance. (Tracking statistics does not equal grading player performance, since it lacks context.)
IMO PFF suffers in public opinion because they literally grade every snap for every player, and they treat every snap equally.
That does not align well with most fans who are just watching TV broadcasts and noticing the particularly good or bad plays by players but not really tracking the “every snap” performance. This results in PFF grades that seem non-intuitive.
I agree – this is PFF’s greatest strength and truly a unique core competency that no other outlet replicates.
The best way I can describe how I value it is similar to how we’ve talked about Approximate Value (AV) in the past. I don’t find AV particularly useful on a season-by-season basis, since even a weak offensive lineman can post a solid single-season AV simply by being the “worst starter” on a high-performing unit. However, career AV tends to reveal much more… players who start consistently, year after year, on productive teams get rewarded, even if they aren’t piling up traditional stats or accolades.
I view PFF’s single-season grades as a stronger snapshot of performance than AV, but I share similar reservations about single-game or small-sample grades. The play-by-play inconsistencies in their scoring system tend to smooth out over time, which makes broader data sets far more trustworthy.
That said, as you pointed out, it’s still the best public measurement tool available. But if I see a low or average PFF grade that conflicts with a film breakdown from someone like Ryan (or any analyst who clearly understands scheme, assignment, and context) I’ll almost always side with them and the film.
I’d probably feel differently if PFF’s grading process were more transparent. The anonymity of the graders is, in my opinion, the biggest driver of skepticism. If analysts “signed off” on their grades, we could at least understand their biases and tendencies toward certain schemes or styles of play. That context would add valuable framing to the data. Likewise, brief notations explaining why grades were adjusted and by whom would go a long way in boosting credibility and trust in their system.
I don’t recall if I have posted my views on AV in this forum, but I have posted that elsewhere.
I applaud Doug Drinen for creating it, and I have interacted with him about it. It is a fine proxy for large samples (e.g., career), but, as you say, not for smaller samples.
It is inherently flawed in a number of ways. For example, here is a snippet from a post in another forum in which I addressed why it was a flawed metric in general and, in some cases, specifically for Ronde Barber:
* For defense, a given team’s total defense points to be divided up among the defensive players is based exclusively on defensive points allowed per drive
— No elements for yards allowed, points allowed total (not per drive) or any other defensive metric
* 1/3 of those points are assigned to defensive backs and 2/3 to the rest of the position groups, independent of how good those groups are on any given defense
— This disproportionately hurts a player like Barber, who played with another HOF defensive back
— This methodology is also dated, since teams play 5+ DBs more often than they play just 4, so this splitting of the points shorts the DB group nowadays
* Defensive backs are given zero credit for tackles
— Tackles was a big strength for Barber, and he gets zero credit for it
* Defensive players get points for interceptions but not for passes defended
— PD was a big strength for Barber, and he gets zero credit for it
* Defensive players get points for fumble recoveries, but not for forced fumbles
* Defensive players get points for sacks, but not for other pressures (QB hits, hurries)
* AV accounts only for regular season games; it ignores postseason gamesOffense has a similar set.
Again, it is fine as long as one recognizes its utility.
That is the issue with many of these grading or assessment systems. Unless you understand the sport on a deeper level and have some personal basis to create your own opinions, most people just look at the PFF grading thrown on the screen to determine their view on the player.
Example: 94th best OT out of 101
It happens when you watch so called “analysts” break down teams to pick games. They often just look at the overall records and who they have beaten or lost to and make a snap judgement. They don’t take things like injury or travel into consideration.
Also, who is making these PFF evaluations? Do they have an extensive coaching or scouting breakdown? Do they take the “dirty work” of the sport into consideration in grading like a LB taking on a lead blocker in the hole to let his teammate make a tackle or the WR running traffic just to free up another WR in mesh?
As long as PFF or any grading system is used as another tool to build a complete picture, and not the end all of evaluation, PFF is fine for what it is.
From Learn about PFF Player Grades:
More info here, with specific articles on various aspects of the game (e.g., pass blocking, run blocking, run defense): How PFF grades all positions at the NCAA and NFL levels.
I’m sure they do. They assign a grade to every player on every play. They would understand these examples as being assignments for players on those plays and grade them accordingly based on how well they executed those assignments.
PFF’s objectives are incredibly worthwhile.
I’ve questioned aspects of the system (appropriate weighting of plays, the distribution they choose which grades too many players < 50.0, etc).
A more core question mark for me is the quality of the graders. Think of the effort that goes into grading every single player across the League, on every play, within 24 hours of the game having finished. It’s ludicrous to think about.
And then you hear the complaints from people who’ve formerly worked at PFF about the work expectations and abysmal pay.
Then you start hearing about attrition rates. And that the process for applying to become a grader has been accelerated to keep up with demand.
And then the conspiracy theories start circulating online around “Uber drivers” becoming graders within 24 hours, and insufficient QA/QC.
So I think Kyle’s transparency point is still pretty relevant. They write some lovely stuff on their website. But who is actually doing the grading. We will never really know their quality. They could be that Mile High guy from BFTB giving Hyatt +2.0 on every snap 😀
This is a very fair post. Impossible to know if not with inside information.
I am somewhere in the middle with them. I don’t put 100% stock into them and I don’t ignore them completely. I use it as a supplement to what I am seeing. As someone who was in the education field previously who taught English, grading a subject that has no concrete answer is very subjective. Just look at scouting as an example. We could watch the same 10-15 plays from a prospect and view them completely different. You might peg him as a first round talent and I could see him as a day 3 selection.