Maybe we should develop a grading system and start grading the PFF graders.Mothman wrote:I understand why some people like what PFF provides but in the end, I find a lot of what they do more like statistical editorializing, the stat/grading equivalent of a columnist's take on a game or performance. Like ESPN's QBR, the further removed from the raw data and the more subjective the analysis becomes, the less reliable the grades/numbers become. Consequently, the rankings and +/- numbers at the top of this thread strike me as borderline useless. I question whether they're really painting a more accurate picture of individual performances or just creating the illusion of greater accuracy. Are they really providing us with more substantive information when their numbers are derived from hundreds of little subjective choices made by analysts with no clear qualifications at all beyond a willingness to do the work?
I'd be curious how many graders there are and if they all grade different games. To me it's at least a bit less subjective if you're consistently getting different eyeballs looking at different players and all coming to a relatively similar conclusion at the end of the year, particularly if the graders have no prior opinion of a team or player going in.
[edit: I found this: "Each game of analysis goes through at least three sets of eyes, just for the grading." https://www.profootballfocus.com/about/grading/ ]
I'd be curious to see their grades over, say, a five-year period and then see how many us fans would really disagree with. Maybe even take into account their grades and players' signing contract extensions or free agent contracts to see if maybe if there's some kind correlation. Other than that, I don't know. Maybe you just have to become one of their graders and use their system to determine for yourself to what degree it's flawed. I suppose this is what it is: a tool. And until a better one emerges, it's a bit more information when evaluating a player.
Here's one "expert" that knows the game and buys into what they're doing (literally and figuratively).They claim interaction with NFL teams leads them to feel strongly about the accuracy of their grades but it would be far more interesting to know how one of those NFL teams actually felt about that accuracy. It would be nice if PFF provided even one quote from an NFL Personnel Dept. that reinforced their claims.
So this spring, Collinsworth was looking to improve his data collection, and signed up for an annual insider’s subscription to PFF. The free part of the site looked real, but he wanted to know more, and so he filled in the email contact form asking for someone to get back to him. Hornsby, with his thick English accent, phoned up. “I thought I’d been scammed,’’ Collinsworth said. “So I figured I would ask him a bunch of questions, you know, to expose him … and after about five minutes, I could tell he was absolutely legit, so I just shut up and listened to Neil talk about football at a very high level.”
At first, Collinsworth said he wanted to simply get to know more about all 32 teams without having to do the kind of film study and painstaking research that would take a couple of days at least—in his words, “to be in position Monday morning to know what would normally take me till Tuesday night to know.’’ Collinsworth said he was actually thinking of engaging some people to watch NFL tape and grade it for him until he realized PFF already did it, and did it accurately. So last week, he reached agreement with Horsnby to buy a significant stake in the company. Hornsby and his crew of 12 full-time graders continue, and Collinsworth will use his influence to help grow the company and seek out new business for it.
There was one final test for Collinsworth. He and Hornsby separately graded a game. When they compared notes, their grades were very close. “And when we went back over them,’’ Collinsworth said, “I’d say on 50 percent of the discrepancies, they were right and I was wrong.”
The deal came together in a couple of months this summer, culminating last week. “What really impressed me,’’ said Collinsworth, “is the fact that 13 NFL teams have contracted with Pro Football Focus for their data. I mean, I have been around the NFL for over 30 years, I know how hard it is to get behind the wall of those teams. And they’ve got 13 teams to trust their data. That’s huge.’’
http://awfulannouncing.com/2014/cris-co ... focus.html
