The moderator problem

Months, maybe years, are spent analyzing a job (or jobs), talking with subject matter experts, documenting, carefully creating assessment instruments, and administering exams in a standardized manner. You gather performance data post-hire, run your correlations, and...wham! Your numbers look terrible. You're not predicting anything.

The test(s) didn't work...right? Maybe, maybe not.

The "holy grail" of test validity is showing a strong correlation between how someone did on the test(s) and how they perform on the job. This type of evidence is often referred to as "criterion-related validity."

Sounds great, but there are some big hurdles to jump when collecting this type of data. I'll mention three.

First we have to grapple with what is considered "strong." In some circles the correlations lauded in professional selection would be scoffed at. For example, .3 is often considered to be a respectable value (e.g., DOL's guide).

Second is what has been called "the criterion problem". This phrase covers quite a bit of ground, but the essential idea is when we measure job performance, there are a few questions to be asked and answered. What is "performance"--number of customers helped? Opinions of supervisors? Extra-role behaviors? More than one of these? All of these? How do we measure performance? Number of customers helped per hour? Supervisory performance ratings? If you use ratings, what factors are rated, using what rating scales?

Third, what, and how, will you correlate with test score? A single rating? Multiple ratings, combined somehow? Will you be correcting any of the statistics for (un)reliability, range restriction, etc.?

Related to these issues is what I'll call the moderator problem. We know that many factors go into job performance, not just someone's competencies they bring to the job. These include things like motivation, mood, and home life on the employee's side and supervisor, co-workers, physical work environment, and resources on the part of the organization--just to name a few.

So here's the deal: Your job analysis may be sound. Your tests may be great. Your pass point(s) perfect. The problem could be something entirely different--it could be something happened with the individual(s) being measured that had nothing to do with work. It could be that it had everything to do with work--say, a horrible supervisor, or unsupportive co-workers.

The point is that the information is not being captured, assumptions are being made, and a lot of hard work may be thrown out. Simply because we're not taking a broad picture of job performance.

How do we solve this problem? Think very carefully before using criteria correlation as a source of support for tests. Don't simply plug numbers into Excel and run the correlation. We need to think about what aspects of performance really are important for the job--just like we thought about competencies. We need to measure in a way that captures these important aspects. Above all, we need to treat this exercise as a complex procedure--because it is.