Research update: Political skill, stereotype threat, and NFL players

A few research articles for us...

First up, several articles from the latest issue of Human Performance:

Lee & Dalal demonstrate in a policy-capturing study that performance "troughs" exceed "peaks" in their influence on performance ratings.

Next, a fascinating study by Meurs et al. where they show how political skill (or networking ability) moderates the relationship between the HEXACO factor of sincerity and task performance. In other words, for individuals high on political skill the authors found a positive relationship between sincerity and task performance (and a negative relationship for those low on the skill).

Are you recruiting highly educated graduates? Then you'll want to read Jaidi, et al.'s piece. In it, they describe a study where recruitment advertising and positive word of mouth related positively to job pursuit intention and behavior. Somewhat surprisingly, on-campus presence related negatively to these measures.

If you like football and/or physical ability tests, you'll be interested in the study by Lyons, et al. of NFL players. In it, they demonstrate that collegiate game performance out-predicted physical ability tests administered during the NFL Combine when looking at future NFL performance. And unlike physical ability, past performance remained a consistent predictor across four years of performance, although the criterion coefficients deteriorated over time, similar to what we find with cognitive ability scores.

Finally, over in the Journal of Applied Social Psychology, Nadler & Clark report the results of research on stereotype threat. The results of their meta-analysis indicated that attempts to nullify stereotype threat (e.g., by dismissing it or disguising the task) resulted in a moderate improvement in score (d=.52) for both African Americans and Hispanic Americans, and there appeared to be no difference between the groups in terms of the effect.

Sidenote: those of you with an interest in HR technology and talent management might want to check out the six sessions being streamed live from Bersin & Associates' IMPACT 2011 conference on April 27th and 28th.

Is your test the 40-yard dash?

A fascinating article came out in the last couple days about how the 40-yard dash is used to evaluate NFL prospects. And while I'm not a huge fan of equating sports teams with other organizations, sometimes the comparison works.

The article starts by describing the story of Brice McCain, a former defensive back from Utah. McCain was considered "too small" by football scouts until they came to town and saw him run the 40-yard dash; he did it in 4.30 and 4.34 seconds. As the article states, "Suddenly, his size (5-foot-9) was less of concern."

Consider these other points from the article:

- McCain's 40-yard dash time wasn't his only strength; he got high marks in other drills that assess quickness rather than overall speed (multiple-method convergence, anyone?)

- Scouts often feel you can teach things like catching, but teaching prospects to run faster is difficult (abilities vs. skills?)

- The 40-yard dash is considered more relevant for evaluating receivers and defensive backs than linemen, where scouts are looking more at foot speed and agility (job analysis informs assessment choice)

- Times for the dash vary with surface (grass v. artificial surface) and runners are never clocked wearing their uniform (beware fidelity of the test instrument)

- The importance of the test is debated given that few players ever run 40 yards during a game (some criterion-related validation might be in order)

- Businesses have sprouted up that provide physical training to prospects to help them perform better in front of scouts (test prep industry expands its reach)

There's quite a lot here that overlaps with assessment in general.

Where doesn't the comparison work? Well, non-sports organizations almost never have the wide variety of statistics available to them to use in judging applicants that sports scouts do. Many organizations also don't have recruiters constantly traveling around the country evaluating groups of applicants. And course there's that whole draft thing.

Still, an example of how assessment can be found in all kinds of situations, and how sticking to best practices pays off in a variety of situations.

Police depts relax hiring standards

In response to serious recruiting challenges, many U.S. police departments are "lowering" their standards for hiring.

The reasons behind the shortage are many, including a strong job market, the Iraq war, and a high number of retirements.

Departments are using whatever means they have at their disposal, including upping their advertising. Case in point: while driving down 880 the other day in Oakland, CA, I noticed a sign promoting the $69,000 starting salary for Oakland Police Officers (and people wonder why it's hard to hire in the Bay Area).

The article cited above describes many steps departments are taking, some of which may initially seem like cause for concern. Let's take a look at them:

1. Forgiving minor criminal convictions, particularly old ones. If someone got busted 10 years ago for doing Ecstasy in college, and hasn't been in trouble since, is that still relevant?

2. Relaxing the 2-year college degree requirement, or allowing experience substitutions. I'm familiar with some research indicating a relationship between college education and officer performance, but if an officer has relevant experience (and performed well), this seems like a wash.

3. Raising the age limit. Age and job performance has been a hot topic in I/O psychology for a long time. While there are some declines over age (e.g., working memory), my reading is that they aren't practically significant in most situations. And we're talking about raising the limit to 40 or 44, not 85.

4. Relaxing fitness requirements. To me this comes back to plain 'ol validation. Granted, it's not always easy to determine where a pass point should be set (do they have to run 300 meters in 55 seconds or 56 seconds?), but do the study. Find out where a reasonable point would be. Run the numbers. See if it makes sense.

A lot of the concerns that go along with these changes--hiring people with low integrity, hiring people physically or mentally unable to perform the job--can be mitigated with good assessment, such as memory tests, physical ability testing, integrity testing, and reference and background checks.

Overall, I think this is a good thing--minimum qualifications (MQ) are often barriers to employment for certain ethnicities, women, and individuals with disabilities. And the situation is even worse when they aren't based on any rigorous study of the necessity for the MQs to being with.

On the other hand, I have heard anecdotally that similar changes in standards for U.S. Army recruits has resulted in more challenges for training.

What do you think--big deal or not?

Summer 2007 Personnel Psychology + free content

The Summer 2007 issue of Personnel Psychology (v. 60, #2) is here and it's got some good stuff, so let's jump right in!

First off is the aptly titled, A review of recent developments in integrity test research by Berry, Sackett, and Wiemann, the fifth in a series of articles on the topic. This is an extensive review of research on integrity tests since the last review, which was done in 1996. There's a lot here, so I'll just hit some of the many highlights:

- It appears that integrity tests can vary in their cognitive load depending on which facets are emphasized in the overall score.

- It is likely that aspects of the situation impact test scores (in addition to individual differences); more research is needed in this area.

- Although there have been no significant legal developments in this area since the last review, concerns have been raised over integrity tests being used to identify mental disorders. The authors do not seemed concerned, as these tests (e.g., Reid Report, Employee Reliability Index) were not designed for that purpose thus likely do not violate EEOC Guidelines.

- Research on subgroup scores (e.g., Ones & Viswesvaran, 1998) indicate no substantial differences on overt integrity tests; no research has addressed personality-based tests.

- Test-takers do not seem to have particularly positive reactions to integrity tests, although this appears to depend upon the type of test, items on the test, and response format.

Next, Raymond, Neustel, and Anderson investigate certification exams and whether re-taking the same exam or a parallel form results in different score increases. Using a sample of examinees taking ARRT certification exams in computed tomography (N=79) and radiography (N=765), the authors found no significant difference in score gains between the two types of tests, suggesting exam administrators may wish to re-think the importance of alternate forms for certification, particularly given the cost of development (estimated by the authors at between $50K and $150K). The authors do point out that the generalizability of these results is likely limited by test type and examinee characteristics.

Third, Henderson, Berry, and Matic investigate the usefulness of strength and endurance measures for predicting firefighter performance on physically demanding suppression and rescue tasks. Using a sample of 287 male and 19 female fire recruits hired by the city of Milwaukee, the authors found that both measures (particularly strength measures such as lat pull-down and bench press) predicted a variety of criteria, including a roof ladder placement exercise, axe chopping, and "combat" test. The authors suggest continued gathering of data to support the use of these types of tests (while acknowledging the ever-present gender differences), and discuss several problems with simulated suppression and rescue tasks, now used by many municipalities in light of previous legal challenges to pure strength and endurance measures.

Lastly, LeBreton, et al. discuss an alternate way of demonstrating the value of variables in I/O research. Traditionally researchers have focused on incremental validity, essentially the amount of "usefulness" that a variable adds to other variables already in the equation. (Allows you to do things like determine if a personality test would help you predict job performance above and beyond the test(s) you already use.) Instead, the authors present the idea of relative importance, which shifts the focus to the importance of each variable in the equation. Fascinating stuff (and far more than I can describe here), and something I'd like to see more of. I believe the authors are correct in stating it would be much easier to talk to managers about how useful each test in a battery is rather than the fact that overall they predict 35% of performance. The article includes a fascinating re-analysis of Mount, Witt, & Barricks' 2000 study of the use of biodata with clerical staff.

----

This issue also includes reviews of several books, including the third edition of Ployhart, Schneider and Schmitt's Staffing Organizations (conclusion: good but not great), Weekley and Ployhart's Situational Judgment Tests (conclusion: good as long as you already know what you're doing), and Griffith and Peterson's A Closer Examination of Applicant Faking Behavior (conclusion: good for researchers, not so good for managers).

---

But wait, there's more...the Spring 2007 issue, which had some interesting stuff as well, is free right now! So get those articles while you can. Hey, it's worth surfing over there just for McDaniel et al.'s meta-analysis of situational judgment tests!