The Myth of Performance Testing

[Riddlewire]

I'm posting this in this forum because this issue is much more important at the college level than the NFL. It's the high school to college transition where the overwhelming bulk of discrimination occurs. And it's these performance metrics that are used as the prime discriminators.

Here is an excellent article about the inherent flaws in maximal performance testing (I saw it at Smart Football). I'm going to post a large chunk of it here. Feel free to read the intro at the website.

We live in a society obsessed with maximum performance. Think of exams like the SAT and the GRE. Though these tests take only a few hours, they're supposed to give schools and companies a snapshot of an individual's abiding talents.

Or consider the NFL Scouting Combine, in which players entering the draft perform short physical and mental tasks, such as the 40-yard dash. The Combine is meant to measure physical ability; that's why teams take the results so seriously.

It's easy to understand the allure of such maximal measures. They don't take very long, so we can quantify many people. Also, they make assessment seem relatively straightforward, reducing the uncertainty of selecting a college applicant or football player.

But as Mr. Sackett demonstrated with those supermarket cashiers, such high-stakes tests are often spectacularly bad at predicting performance in the real world. Though the SAT does a decent job of predicting the grades of college freshmenâ€"the test accounts for about 12% of the individual variation in grade point averageâ€"it is much less effective at predicting levels of achievement after graduation. Professional academic tests suffer from the same flaw. A study by the University of Michigan Law School, for instance, found that LSAT scores bore virtually no relationship to career success as measured by levels of income, life satisfaction or public service.

Even the NFL Combine is a big waste of time. According to a recent study by economists at the University of Louisville, there's no "consistent statistical relationship" between the results of players at the Combine and subsequent NFL performance.

The reason maximal measures are such bad predictors is rooted in what these tests don't measure. It turns out that many of the most important factors for life success are character traits, such as grit and self-control, and these can't be measured quickly.

Consider grit, which reflects a person's commitment to a long-term goal. As Angela Duckworth, a psychologist at the University of Pennsylvania, has demonstrated, levels of grit consistently predict levels of achievement, such as graduation from West Point and success in the National Spelling Bee.

The problem, of course, is that students don't reveal their levels of grit while taking a brief test. Grit can only be assessed by tracking typical performance for an extended period. Do people persevere, even in the face of difficulty? How do they act when no one else is watching? Such traits often matter more than raw talent. We hear about them in letters of recommendation, but hard numbers take priority.

The larger lesson is that we've built our society around tests of performance that fail to predict what really matters: what happens once the test is over.

Basically, this article invalidates Rivals' entire reason for existing. Their combines (which are disproportionately attended by negroes) send flawed signals to college coaches. They give maximal effort during these tests because they know they are being tested for specific numerical results. Whereas the great white player who didn't happen to attend a combine doesn't have the superior test results to sell himself. He only has his peformance on the field. But excellent physical test numbers don't have any correlation with effort during a game (or during preparation). So your 'physical freak' afflete is no more effective (and likely he is less so) than your run of the mill "high motor" guy. White players like Zach Barket (one of the most productive high school runningbacks of all time) will most likely perform better for you on game day than "dynamic playmakers" who have "game breaking speed" (code words for 'black kid with a fast 40').

 

[Observer]

Another thing about a one-off test is that they sometimes test exactly the opposite quality to that which they think they actually want. A simple example is that top-end running speed is almost inversely proportional to quickness (unless the individual is very strong). And also, both speed and quickness are often inversely proportional to endurance.

I would think a good running test would be to have a player run 10 dashes of 20 yards of so, with 30 seconds between. And then, count not only the total time, but the slowest time -- and also, the number of strides taken. A short strider who can still run fast tends to be much better at "game speed".

But even a fairly simple-to-implement test like this would not at all test how a player responds when he is somewhat tired and under stress.

And that is why past performance would certainly seem to be the best indicator of future success.

Sometimes, I really wonder what the average IQ is of the talent evaluators. I mean that seriously. Are they so stupid, or is the political pressure so great that they do not feel confident in evaluating a player on criteria that have a subjective aspect involved?Edited by: Observer