Spring '11 P-Psych: Leadership, interviews, competency modeling

It's journal season--this time let's take a look at the Spring 2011 issue of Personnel Psychology:

First up, Derue, et al. with an important meta-analysis on leadership effectiveness. After looking at 59 studies, they found that leader traits and behaviors explained a minimum of 31% of leadership effectiveness. Interestingly, group performance (which I would argue is the most important criterion) was the most difficult to predict. Behaviors tended to explain more than traits, but the authors suggest a model where behavior mediates the relationship between traits and effectiveness is warranted. Not surprisingly, the best trait predictor depended on the criterion: conscientiousness predicted leader effectiveness, group performance, and follower job satisfaction the best, while satisfaction with leader was best predicted by leader agreeableness (reminds me of a recent IJSA study). The same was true with leader behaviors, although consideration was a good predictor across the criteria. A must for anyone interested in leadership research, and you can read an in-press version here.

Next, a piece by Melchers, et al. on whether more interview structure really leads to better rating quality (I'll ruin it for you: yes). Specifically, using a sample of primarily undergraduates, the authors found that providing subjects with frame-of-reference (FOR) training and descriptively anchored rating scales led to substantial increases in rater accuracy and interrater reliability. You can read an in-press version here.

Those of you interested in the concept of core self-evaluations will want to read the study by Ferris, et al.

One area that we don't see enough research in is newcomer adaptation. The longitudinal study by Wang, et al. of a group of Chinese subjects helps fill that hole by exploring the relationship between adaptability, person-environment fit, and work-related outcomes.

Last but most certainly not least, Campion, et al. provide us with a review of best practices in competency modeling. Specifically, 20 of them. Of particular note to you may be that they distinguish competency modeling from job analysis. Want to read the whole thing? Me too! Wish I could have found an in-press version but no such luck.

Sidenote: while not specifically related to recruitment or selection, Christian, et al.'s piece on engagement may be of interest to several readers.

"Grit": An example of will do

Personnel psychologists often make a distinction between factors that indicate a person can do a particular task and those that indicate they will do. Usually "can do" facets include things like cognitive and physical abilities--baseline traits that a person must possess to even be able to perform the task. "Will do" facets are related to motivation and interest and get at whether the person is likely to perform the task, regardless of ability.

In the March 2011 issue of Fast Company, Dan and Chip Heath (Switch) write about the concept of "true grit" and its importance for successful performance. They point to the recent movie remake of True Grit. While many might assume the title refers to the crusty gunslinger (Rooster Cogburn), it actually refers to Mattie, a teenage girl who hires Cogburn to avenge her father's death.

The Heaths describe several examples where organizational leaders and innovators refused to give up in the face of failure or long odds and go on to impressive success. They even cite research conducted several years ago by Angela Duckworth and her colleagues who found that scores on a measure of grit predicted retention in West Point (a prestigious U.S. military academy).

What they don't point out is that the retention finding was specific to a summer training program and scores on the measure of grit were not superior to other predictors when the criterion was first-year cadet GPA or performance ratings. In addition, the percentage of variance accounted for across the studies was around 4% and the measure correlated highly with a measure of conscientiousness. However, grit demonstrated incremental validity beyond IQ and conscientiousness and it's still a fascinating study that you can read here.

To be sure, "will do" factors are often overlooked when talking about selection. Often we focus exclusively on ability factors, either because we're unsure how to measure motivational factors or we're afraid to. But that doesn't mean they're not important. We know in some situations (e.g., jobs with low entry and ability requirements) noncognitive measures can out-predict ability measures.

This article also raised two other issue for me. First, laypeople often conceptualize KSAPs as dichotomous: you're either smart or you're not, you either have integrity or you don't. The reality is practically anything you can think of measuring lies on a continuum--so we talk of degrees of personality characteristics, or levels of ability. With respect to the topic of this post the situation is the same: there are shades of grit.

The second issue has to do with having too much of a good thing. One can be too smart for a job; it's not that you can't do it, it's that you'll likely get bored after a short period of time. Similarly, one can have "too much" (or be too far on either end) of a personality continuum. Take grit. Imagine someone who was so determined that not only do they persist in the face of obstacles, they refuse to give up, even when presented with overwhelming odds. Now they're bordering on obsessional and/or delusional.

So what does this all mean? Back to basics:

1) Know the job and its requirements
2) Pick critical, necessary-at-entry KSAPs to measure
3) Select and/or develop high quality measures
4) Know your applicant pool and the likely range of scores you will obtain
5) Recognize that the relationship between tests and job performance is probably not linear (particularly when your concept of job performance is multifaceted)

And finally, back to true grit. The best thing we can do as assessment professionals is demonstrate it ourselves by not taking the easy way out and not folding in front of obstacles such as the desire for speed over quality, or ignorance. No guns required.

March IJSA: So much good stuff

The March 2011 issue of the International Journal of Selection and Assessment is out, and it's a doozie. Check it out:

- Beaty, et al. present evidence that unproctored Internet tests (noncognitive: personality and biodata) generally had similar criterion-related validities across a spectrum of job performance criteria compared to the same administered in a proctored setting. Mmmm....UIT...

- Generally when people re-take a cognitive ability test, they do better. But do they do better on other tests of cognitive ability? Matton, et al. describe a study that looked at just that and found the answer to be "no."

- One advantage that is frequently claimed about personality inventories is that usage results in less adverse impact compared to, say, ability or knowledge tests. But might the AI depend on how hires are made (e.g., top-down, compensatory)? Turns out the answer is yes--at least according to some results by Risavy and Hausdorf.

- Hey, government agencies, still debating whether to put more resources into your career web portal? Maybe this will convince you. Selden and Orenstein show that governments with more usable portals as well as better available content not only attract more applicants per opening, but have less voluntary turnover of new hires.

- With advances in technology and changes to the work environment, clerical jobs have changed a lot over the last 30 years and the old ways of selecting for these jobs (namely g-loaded tests such as perceptual speed and verbal ability) likely need to be re-thought...right? Well, not so much, at least according to a meta-analysis by Whetzel and her colleagues. In fact, the criterion-related validity values met or exceeded those found 30+ years ago. The more things change...

- De Goede, et al. present the results of a study that explores the relationship between P-O fit and organizational websites but include the concept of person-industry fit. One implication: if you're trying to attract a more diverse group of candidates, work on making your portal more attractive.

- We spend a lot of time trying to make sure interviews are loaded with job-relevant content. But how much attention do we pay to the impact of impression management tactics on the part of applicants? Huffcutt's results make a compelling argument that we ignore the latter to our detriment as it may have more to do with interview ratings than the job-relevant content.

- How does one determine managerial potential? Well, it depends who you ask. Thomason, et al. present results that indicate when supervisors are asked, they focus on task-based personality traits (e.g., conscientiousness), whereas peers focus on contextual traits such as agreeableness. Given that leadership is ultimately about achieving things through subordinates, I wonder what we should be paying attention to....hmmm...how about both?

- Thinking about using self-ratings of political skill as part of the application process? I can certainly see situations where this skill may be helpful, but might this method be susceptible to inflation? Not so much, at least according to results from Blickle, et al.

- Last but definitely not least, Carless & Hetherington with some data on the impact of recruitment timeliness on applicant attraction. The longer we make applicants wait, the less attracted to the organization they will be, right? Not so fast. According to this research, it is perceived timeliness that matters, not actual timeliness (hence the importance of communication). In addition, this relationship is partially mediated by job and organizational characteristics.

Research update: From item context to signaling theory and more

Here are some research articles from the last couple months:

Grand et al.'s study showed that adding a job-relevant context to test items--even under explicit stereotype threat--had either beneficial or no effects on test performance and test perceptions among female test takers. More evidence of the benefit of tailoring items to the particular position being tested for.

Impression management during assessments is often considered to be a negative thing--i.e., a source of error. But as Kleinmann and Klehe point out in their study of interviewee behavior, it may be an additional source of validity, and can be related to performance. At the very least it indicates that the person knows enough to alter their behavior to fit the job!

Celani and Singh provide a literature review of the role of signaling theory in applicant attraction (making inferences about important aspects of the job/organization from characteristics of the recruitment) and how social identity interacts to impacts attraction outcomes.

Last but not least, Soto et al. with a fascinating study of age and personality characteristics. Over a million individuals participated over the web, and the authors highlight several key results, such as late childhood/early adolescence being key periods for age trends, strong maturity and adjustment trends over adulthood, and the importance of looking at facet-level results.

We can make assessments "fun"...should we?

Remember when tests were fun?

Neither do I. Tests and assessments have a long history of being about as popular as the dentist. Starting in grade school many come to dread them as lifeless--and often inaccurate--judges of worth. (Of course doing well on them that tends to improve your view)

Tests don't have to be boring. We write structured interview questions and multiple-choice questions because that's what we've always done. And we know how to do it right.

But there are plenty of ways of making them more interesting, from the way they're written (try: "You are in a maze of twisty passages, all alike"), to their presentation (e.g., animation, video), to the way people progress (e.g., adaptive testing), to the way results are given to you ("You've got the high score!"). Today more than ever before we have the flexibility to take those dry, monochromatic presentations and turn them into something eye-catching and even...dare I say...fun?

(10 bonus points to those of you who caught the Adventure reference in the preceding paragraph)

The question is: Should we?

There's been quite a bit written lately about the "gamification" of assessments. Heck, I've been on that bandwagon for years. America's Army, first published in 2002 as the U.S. Army's foray into first-person shooters, was an early example of the potential marriage between staffing and entertainment--yes it's not technically a personnel assessment but the recruiting mission is obvious as is the potential use of the results. Since then we've seen a steady stream of innovation, from the use of branching video to realistic job preview-type assessments presented online.

There are several reasons why we might want to make assessments a wee bit more entertaining:

A) Because we can. Please don't use this reason.

B) As a recruiting tool to help you distinguish yourself from competitors ("Look, we're fun and cool! Join us!")

C) To encourage candidates to complete the assessment ("Yes it's a little long, but the time will fly by!")

D) As part of a realistic job preview ("Not sure if you want the job? Find out virtually!"). Nothing wrong with that. Self-selection out is good.

E) Because it helps us measure more accurately. Ah-ha. Now we're getting somewhere. To the extent that entertainment/interactivity helps us overcome candidate fatigue or other error, or otherwise helps us measure relevant KSAPs more accurately, we've won the game.

(10 bonus points to those of you who saw noticed the irony of me using a traditional multiple-choice presentation in that last part)

(50 bonus points to those of you who have noticed the use of bonus points in this post)

So we have a number of reasons we might want to make assessments more fun. But there are some reasons why we might not want to. Or at least that should make us pause:

1) Tests are serious. No, really. They have an enormous impact on people's life. We don't want to water down their nature so much that we disrespect our applicants.

2) The "tests" out there right now that are the most "fun" are also not ones you'd want to use to select people (e.g., which animal are you?), although there are some surprising hybrids (find your Star Wars twin).*

3) Those assessments that are kinda-pitched-as-actual-assessments-but-not-really-don't-hold-us-to-that, have already started blurring the line (e.g., True Colors). We want to draw a clear distinction between assessments purely for fun and "okay, you need to take this seriously, it's for a job."

4) We can easily mess this up. All it takes is a high-level manager getting into his/her head that we need to "make these test things more fun" and suddenly we're pressured into creating an expensive mess that doesn't deliver (like Daikatana).

(10 bonus points to those of you who noticed I switched response options from letters to numbers)

(50 bonus points to those of you that completed all three of the example assessments)

(100 bonus points if you know what Daikatana is)


So is there room for us to be a little more creative and investigate alternate--more immersive, interactive--ways of assessing candidate qualifications? Absolutely. But should we use caution to make sure we don't have a big-budget flop on our hands? You bet.


Now count up your points from this blog post. How did you do?

0 points: Wait...you did READ this, right?

10-20 points: Okay, maybe you're tired.

20-130 points: The force is strong with you...but you are not a Jedi, yet.

More than 130 points: Call me.


* I'm an owl. Or maybe a penguin. Oh, and kinda like Darth Vader. But also Princess Leia. And Mon Mothma. Now I'm confused.

New blog: Select Perspectives

There's a new blog on the block, and this time it's the folks over at Select International and the name of their blog is Select Perspectives. Three posts over a span of one-and-a-half weeks (they started at the end of January) is promising. I particularly enjoyed the post about talking to fifth graders about I/O psychology.

Here's to hopin' that we're witnessing the birth of a valuable addition to our blog roll! Welcome.

Oh, and the RSS feed is riiiggghhht here.

Job ads: Choose your words wisely

Back in 2007 I began a project through this website to collect survey data from passers-by on words found in job ads. You may have seen a link to the survey on this blog's home page--or even taken the survey itself.

I was motivated to undertake this project because at the time (and this may still be the case) there was very little research on how attractive/effective certain words are in job advertisements. Seems like a simple question, and I was curious.

After 3.5 years I've decided to share what the data shows. My hope was to get a large sample, and I'm settling for just under 150, so take that into account. And of course the generalizability of the results is questionable, although the fact that it was gathered over such a long period of time and the sample group is fairly diverse may help us feel a little more comfortable.

Method

SurveyMonkey. Four questions. Began collecting data in June of 2007, last data point was November of 2010. Questions were followed by a series of fifteen words or phrases generated pretty much off the top of my head with an "Other" option. Unfortunately I didn't think to randomize the presentation of options, so keep that in mind. I had to use two different collectors in SurveyMonkey since I have a free account and they max at 100 responses. When graphics are presented below they are for the first collector only (couldn't download/combine the data sets because of free account), which has more responses but is older data. You can see/take the survey here.

Participants

One hundred and forty-seven blog visitors. I expected most to primarily identify as HR professionals or academics, but most (43%) chose job seeker. The respondents generally fell evenly into the other categories, including just being interested in the subject matter.

Results

Word Frequency


The first question asked participants what words they see most often in job ads.

The #1 answer? By far....."Motivated" Selected by 70-80%, depending on the collector, also the first choice presented.

Other frequent answers (in the 50% vicinity):

- Professional
- Organized
- Works Well Under Pressure

Interestingly, "Motivated" seems to have become more frequent over time, as has "Works Well Under Pressure" and "Flexible". "Independent", "High Energy", and "Friendly" were among the words becoming less frequent.

How about least frequent?

- Conscientious
- Smart
- Friendly

Both "Conscientious" and "Smart" became less frequent over time.

Emotional Response


The second question asked participants to rate their emotional response to the same words or phrases presented in the first question on a seven-point rating scale from "Very Negative" to "Very Positive". Generally most words/phrases received positive responses, and the difference in means between the best-liked ones and least-liked ones was less than one point.

So which received the most positive ratings?

- Motivated
- Reliable
- Professional
- Independent

How about least positive?

- Works Well Under Pressure
- High Energy
- Detail Oriented

I'm guessing that to the extent that there is a difference here, the most popular words seem to describe jobs that would allow applicants a fair amount of flexibility over their work and involve a stimulating work environment. The least positive words are likely associated with fast-paced (hey that would have been a good option) jobs, such as customer service, that may not be particularly stimulating.

Application Intentions


The last question asked respondents to select how likely they would be to apply for a job that contained the same list of words/phrases (knowing no other details). As with emotional response, most responses were associated with high intentions to apply, and the difference between most likely and least likely means was less than one point.

So which led to the highest application intentions? "Professional" and "Reliable" were consistent across the two collectors. "Motivated" became more positive over time, as did "Independent". This is consistent with emotional response.

How about the ones least likely to lead to application intentions? Matching the emotional responses, "High Energy" and "Detail Oriented" were bottom on the list. "Smart" has become less associated with application intentions, as has "Works Well Under Pressure".

Discussion

Judging by this data, it appears that organizations wishing to distinguish themselves using job advertisements should feel comfortable using words that directly speak to applicant personality, such as "conscientious" and "friendly"--these were less frequently found and not associated with particularly negative responses. Organizations should try to use words that imply an environment that allows applicants to use their own judgment when making decisions and stay away from words that imply a hectic, always-on work environment.

Of course all this depends on the particular job being advertised. As we know, presenting candidates with a realistic job preview is immensely helpful (for them as well as the organization), and if the job is heavy customer service, well, it just is. In addition, this data says nothing about the quality of applicants--it may be that higher performing employees prefer different words than lower-performing ones.


I hope at the very least you found these results interesting. It's something to think about when crafting your job ads, and of course one could run a much more sophisticated study by including things like occupation and demographics.

Jan TIP gems: HRO, UIT, SIOP, and VII

Yes, my goal was to create a blog post title using words no longer than four letters.

Anyway, for those non-SIOP'ers out there, or SIOP'ers that may have missed 'em, there were some gems in the latest issue of TIP:

How I/O can shape the practice of strategic human resources outsourcing (HRO)

A great little study on perceptions of various ways of mitigating cheating on unproctored Internet testing (UIT)

The difference between academics and practitioners in terms of what topics are valued at the SIOP conference (e.g., the latter were more interested in job analysis, staffing, and strategic HR)

Last but not least, a point, counter-point on whether the addition of sex as a protected category under Title VII was a joke

January, 2011 J.A.P.: Interests, trainability tests, interns, and personality tests

The January issue of the Journal of Applied Psychology is out with some great content, so let's jump right in.

First up, an intriguing study by Van Iddekinge, et al. on an interest inventory. Even though many suspect vocational interest plays a part in motivating work behavior, historically the published relationship between interest and job performance has not been strong. The authors of this study created a new measure of interest and using a decent sample size (418) found surprisingly high (corrected) correlation values between scores and various criteria, including job knowledge, job performance, and continuance intentions (mean R = .31). Scores also predicted additional variance beyond cognitive ability and measures of the Big 5 personality dimensions. Could we be on the cusp of a revolution in measures of interest? This could help bridge the gap between KSAs and discretionary effort.

Next, Roth et al. with an update on trainability tests--their predictive validity as well as Black-White score differences. As a refresher, trainability tests are a sub-category of work sample tests that involve a structured period of learning for applicants and are designed to measure how well they can learn a new skill. Previous (limited) research indicated they predict training performance fairly well but job performance less so, and this decreases over time. However, the authors of the current study show using data from a recent video-based trainability exam that the validity may be higher than we thought. Unfortunately it also showed a high level of mean differences between Black and White applicants, matching or exceeding that typically found for cognitive ability (which the test correlated highly with).

Did someone say personality testing? (no, but you knew it was coming) Le et al. are up next with an update on the curvilinear relationship between personality scores and job performance. Using two different samples, the authors found not only the hypothesized curvilinear relationship but that the inflection point (after which the relationship disappears) occurs later in jobs that are more complex--similar to the relationship between experience and performance. So for example, scores on Conscientiousness may correlate with job performance (and OCBs, and CWBs) for higher scores than, say, a retail clerk. Important for anyone making assumptions about what personality inventory scores imply.

Next up is your second personality test article, this time by Landers et al., who provide a warning about faking. The authors noticed a new trend in responses, which they label "blatant extreme responding" (BER; not listed as an X Games sport), indicated by answering all "1"s or "5"s on an inventory. They hypothesize that this is due to a coaching rumor, which seems to have been supported by the fact that internal retesters showed a higher prevalence of BER than the general sample. On the plus side, an interactive warning seems to have reduced the spread. Hard to tell if this is anything new, since we know faking does indeed occur--the debate is over its impact on validity.

Last but not least, Zhao and Liden write about internship programs and the impression management that occurs on the part of interns as well as the organizations. Not surprisingly, interns that wished to get hired by the organization were more likely to use self-promotion and ingratiation, which increased the likelihood of receiving a job offer. Perhaps more interesting is the finding that organizations wishing to hire the interns permanently exhibited more openness to creativity on the part of the interns, which in turn increased the likelihood that interns would apply. Lesson? If you have a good intern that you want to bring on full-time, solicit and be open to their suggestions.

Supreme Court upholds NASA background checks

Yesterday, in a unanimous 8-0 decision*, the U.S. Supreme Court handed federal employers a victory when they upheld a background check process used by the National Aeronautics and Space Administration (NASA).

The checks, which have been standard for millions of federal employees since the 1950's, were challenged by a group of contract employees at the Jet Propulsion Laboratory (JPL) in California who previously had not been required to undergo them. The employees claimed the questions asked during the background checks infringed on their "constitutional right to information privacy" rights by asking about things like drug treatment.

In their decision, the court held that "the Government has an interest in conducting basic background checks in order to ensure the security of its facilities and to employ a competent, reliable workforce to carry out the people’s business" and that the challenged questions were "reasonable, employment-related inquiries that further the Government’s interests in managing its internal operations." It also held that there was no meaningful difference between contract employees and civil service employees.

With respect to questions regarding drug use, the court pointed out that the government used follow-up questions regarding treatment as a "mitigating" factor to separate out drug users from those who are taking steps to address and overcome their problems.

They gave the federal government a low hurdle to jump when it comes to business necessity, rejecting the argument that "the Government has a constitutional burden to demonstrate that its employment background questions are “necessary” or the least restrictive means of furthering its interests."

They also touched on reference checks, writing "Asking an applicant’s designated references broad questions about job suitability is an appropriate tool for separating strong candidates from weak ones. The reasonableness of such questions is illustrated by their pervasiveness in the public and private sectors."

Importantly, the court pointed out several times that concerns regarding the collection of personal information by the government were mitigated by the substantial controls in place to protect such information.

You can read the decision here. You can also see the challenged forms here and here.

*Justice Kagan took no part in the decision

Why aren't more supervisors held accountable for their hiring mistakes?

Before I begin this rant...err...post, let's create a fantasy world.

No, not the kind with elves and magic (maybe in a future post).

The kind where organizations run efficiently, consistently, and rationally (see I told you it was going to be a fantasy).

So we'll set it up with two assumptions:

1) Job performance is defined for every individual in the organization in clear, meaningful ways.

2) All employees are held accountable for their critical decisions.

Okay, have this fantasy world in your mind? Great. Now that we have that out of the way, I ask you: Why aren't more supervisors in this world held accountable for their hiring decisions?

Think about it. What would you do with a supervisor who repeatedly:

- Made bad decisions about what technology to purchase/implement

- Made ineffective choices regarding how to divvy up team assignments

- Spent time inefficiently and chose to focus on low-priority projects

- Conducted meetings that were too long and had little payoff

Presumably write them up or get rid of them. All of these decisions have to do with the use of resources. Yet I could make a strong argument that the most important resource decision supervisors make is about who to bring into the organization.

So honestly, why aren't more supervisors held accountable? I have several theories:

1) The hires are blamed, not the supervisor. In some cases this makes sense. But when a hiring supervisor demonstrates a pattern of getting it wrong, someone needs to be looking at the decision-makers.

2) The connection between the decision and the consequences is distant. It's easier when a decision has immediate consequences, but when the employee has been around for 2 months, it's a lot easier to focus on their mistakes rather than how they got there to begin with.

3) It's harder to hold supervisors accountable. I think most organizations simply have a harder time quantifying or otherwise seeing supervisory performance in objective ways. Even when performance is measured, too often it focuses on how they manage their employees rather than their selection decisions.

4) People have a hard time giving negative feedback, especially to supervisors. It's hard enough to tell someone they're not meeting specific production goals; it's even harder to pinpoint exactly what the supervisor is doing wrong when hiring people--and then communicate that.

5) It's political. Organizations are typically even more committed to supervisors than rank-and-file because the decision of who to put there likely involved folks higher up in the organization. Thus, to admit a mistake in the supervisor is to admit a mistake in who hired them. Not to mention that supervisors are part of the very management structure that has issues in the first place.

6) It's not clearly measured as part of the job of a supervisor. Okay, I realize this is somewhat against the rules of our fantasy world, but I suspect this is often a big problem. Supervisors are usually measured against things like "leadership" or "administrative ability", not "the percentage of new hires who receive excellent performance ratings after 6 months."

So what can we do about this? Well I think some of the answers present themselves when we see what some of the problems might be above. Clearly document that hiring success will be considered a key performance metric for supervisors. Provide them training on the topic and feedback. Have consequences for those who keep getting it wrong (which may simply be taking the decision largely out of their hands).

Should HR be more involved? Almost certainly. It is after all HR's duty to maximize human performance in an organization. We're supposed to be the experts on this stuff--if we can't articulate the issue and suggest alternatives, why would we expect line management to?

Finally, let me end on something else to think about: Does your organization praise supervisors who repeatedly make good hiring decisions? We know from basic psychology that positive reinforcement is more likely to result in repeat behavior than punishment or negative reinforcement. But which do we normally use?

Now back to reality...

IPAC Call for Proposals Open

The Call for Proposals for the 2011 IPAC Conference is now open until January 31st. The Program Committee is seeking proposals for symposia, panel discussions, paper presentations, tutorials, and pre-conference workshops.

The conference will be held July 17-20 in Washington, D.C. at the Dupont Hotel and promises to match previous events in terms of quality and networking opportunities.

You can find the Call here, and more details about the conference here.

Relax, researchers

Researchers make a lot of choices: What to study. How to measure. What statistic(s) to use. Researchers conducting meta-analyses (essentially a study of studies) face similar choices. So it's nice to know that any agonizing meta-analysts go through may be largely unimportant in terms of the results.

In the January 2011 issue of the Journal of Management, Aguinis et al. report the results of their study of 196 meta-analyses published in places like Journal of Applied Psychology, Academy of Management Journal, and Personnel Psychology. Specifically, they looked at the impact that various methodological choices had on the overall effect sizes. Things like:

- Should we eliminate studies from the database?

- Should we include a list of the primary-level studies used in the meta-analysis?

- What meta-analytic procedure should we use?

Here, in their words, were the results:

"Our results, based on 5,581 effect sizes reported in 196 published meta-analyses, show that the 21 methodological choices and judgment calls we investigated do not explain a substantial amount of variance in the meta-analytically derived effect sizes. In fact, the mean effect size...for the impact of methodological choices and judgment calls on the magnitude of the meta-analytically derived effect sizes is only .007. The median [effect size] value is an even smaller value of .004."

So not only does this suggest researchers should spend less time worrying about methodological choices, it raises a question about the value of including too much of this history in published articles--if it doesn't make any difference, do I really need to read about why you chose to eliminate studies due to independence issues or missing information?

The article has another, perhaps even more interesting, finding. And it's something that us personnel research nerds like to argue over: corrections. Things like correcting for range restriction (e.g., because you only hire people that get the top scores) and criterion unreliability (e.g, because supervisor performance ratings are less than perfect). For every "corrections were necessary to understand the conceptual-level relationships" you'll get a "that's great, but in the real world the data's the data."

So that's why it's fairly amusing to note the differences these types of corrections tended to make. Again, in the authors' own words:

"across 138 meta-analyses reporting approximately 3,000 effect sizes, the use of corrections improves the resulting meta-analytically derived effect size by about .05 correlation coefficient units (i.e., from about .21 to about .25). Stated differently, implementing corrections for statistical and methodological artifacts improves our knowledge of variance in outcomes by about only 25%."

So are all going to stop arguing and hold hands? Not likely. For a variety of reasons, including the fact that opinions are hard to change, and social science researchers are not immune. You could also argue that in some cases an increase from .21 to .25 has significant implications--and to be sure that's true. But I agree with the authors that the number of cases where this greatly increases the practical usefulness of a theory is small.

Does this mean we should forget about refining our results to make them more accurate? Absolutely not. It just means that our current approaches may not warrant a lot of hand-wringing. It also means we should focus on high-quality primary studies to avoid the garbage-in, garbage-out problem.

So let's take a deep breath, congratulate ourselves for a productive year, and look forward to 2011, which I'm sure will bring with it all kinds of exciting and provoking developments. Thanks for reading.

By the way, you can access the article (at least right now) here.

Haste makes waste

"Here you go, way too fast
don't slow down you're gonna crash
you should watch -- watch your stay here
don't look out you're gonna break you neck"
- The Primitives

There are many pieces of advice I could give the average employer when it comes to recruiting and hiring the right way. Make your job ads more attractive and concise. Use realistic job preview technology. Conduct thorough job analyses. Reduce your over-reliance on interviews.

But I'd be hard pressed to come up with a more important piece of advice than this: SLOW DOWN.

Too often organizations rush through the recruitment and selection process, relying on past practice and not giving it the attention it deserves. The result is often poor applicant pools and disappointing final selection choices.

Here are some classic warning signs things are going the wrong way:

"We don't have time to re-do the advertisement"

"Let's just do interviews like we did last time"

"The questions we used last time will be fine"

When you hear these types of comments, your reaction should be: "Because they worked so well last time?" Okay, maybe that's what you think. What you say is: "What information do we have about their prior success?" Hiring decisions are too important to be left to hunches and cognitive laziness--we all know this. Yet it's surprising how often folks fail to put in the effort they should.

Why do people fall into this trap? Mostly because it's easier that way (although naivete and lack of organizational processes play a role). Decision makers naturally gravitate toward the path of least resistance (we do love our heuristics), and it takes resilience to put in the effort each time. But it's not just because humans are lazy. It's because we're busy, and because other factors tend to overshadow sound selection--like organizational politics or feeling that someone is "owed" the job.

Decision making as a field of study tends to be overlooked when it comes to hiring, and that's a shame (ever heard of escalation of commitment?). Fortunately there is a large body of research we can learn from, and this cross-fertilization is the subject of the first focal article by Dalal, et al. in the December 2010 issue of Industrial and Organizational Psychology. They point out that the field of I/O psychology is hurting itself by not taking advantage of the theories, methods, and findings from the field of judgment and decision-making.

One of their main recommendations is to make "a concerted effort to consider the benefits of adopting courses of action other than the favored one." This could mean things like having devils advocates and group decision making automatically part of your hiring plan. It could even be as simple as a checklist that hiring supervisors fill out to ensure they're not rushing it. Or--heaven forbid--we could hold supervisors accountable for the quality of their hiring decisions.

There are several interesting commentaries following the article, making many different points. One of my favorites is from Kristine Kuhn, who I'll now quote from liberally:

"...evidence-based recommendations to use statistical models to select employees rather than more holistic, subjective assessments meet substantial resistance. The ambiguous criterion of "fit" is advanced by many experienced practitioners as a reason for not relying solely on validated predictive indices."

"Despite considerable evidence that typical interviews do not add predictive validity, managers often resist attempts to impose even minimal structure." (Consider this the next time you follow-up a structured interview with a more casual unstructured one)

"Some managers may be receptive to training and even willing to implement structural changes in selection procedures. But this will only be the case if the primary goal is in fact to hire the people most to likely to perform well and not those with whom they will be most comfortable interacting."

Amen.

As for me, I'm going to make a New Years resolution to take more deep breaths and slow down. Most involved in hiring would do well to do the same.

I should point out there is another focal article in this issue by Drasgow, et al. that you methodology folks will like. In it, the authors argue that rating scale methods derived from Likert's approach (the 5-point response scale) are inferior to ones that have evolved from the (older) ideas of Thurstone.

In a nutshell, the authors describe how the latter approach focuses on an "ideal point" that describes an individual's standing on a particular trait. It involves, as part of the rating scale design, asking people to provide ratings that might seem unnaturally forced or incongruous (do you like waffles or Toyotas?). But the authors argue strongly that this approach offers tangible improvements for things like personality inventories.

The commentators are...shall we say...skeptical. But the back-and-forth makes for some interesting reading if this is your cup of tea.

More on personality: Empathy and genetic links

The research on personality inventories continues unabated with two new studies.

The first, by Taylor et al. in the December 2010 JOOP, found that empathy plays an important role in explaining the relationship between Big 5 traits and organizational citizenship behaviors.

The second, by McCrae, et al. in the December 2010 Journal of Personality and Social Psychology, was an attempt to better explain the genetic underpinnings of personality. The effects found in this study were small but significant, suggesting further research is needed to better understand this relationship.

Speaking of personality, don't miss Bob Hogan's most recent post to his blog; it features a wonderfully simply explanation of the value of even "small" correlations between assessment instruments and job performance.

Research potpourri

Here's a very quick round-up of several recent pieces of research:

Bott et al. on how individual differences and motivation impact score elevation on non-cognitive measures (e.g., personality tests)

Ackerman, et al. on cognitive fatigue during testing

Robie, et al. on the impact of coaching and speeding on Big Five and IM scores

Evers, et al. on the Dutch review process for evaluating test quality (pretty cool; in press version here)

And now, back to figuring out why I eat too much on Thanksgiving. Every year.

New issues of Personnel Psych and IJSA: Muslim applicants, selection perceptions, and more

Two of the big journals have come out with new issues, so let's take a look at the highlights:

First, in the Winter 2010 issue of Personnel Psychology:

King and Ahmad describe the results of several experiments that found interviewers and raters altered their behavior for confederates exhibiting obvious Muslim-identified behavior (e.g., clothing) depending on whether applicants exhibited stereotype-inconsistent behavior. For those that didn't, reactions were shorter and more negative. On the other hand, no difference was found in offers between those dressed in Muslim-identified clothing and those that weren't. So behavior--specifically its stereotypicality--and not simply something obvious like dress, may be key in predicting/preventing discriminatory behavior.

Do you consistently read the "Limitations" section of journal articles? Brutus et al. did, for three major I/O journals from 1995 to 2008 and found that threats to internal validity were the most commonly reported limitation. Interestingly, they also found that the nature of limitations reported changed over time (e.g., more sampling issues due to volunteers, variance issues). You can see an in press version here.

Next up, Henderson reports impressive criterion-related validity (combined operational = .86) for a test battery consisting of a g-saturated exam and a strength/endurance exam after following a firefighter academy class for 23 years. He suggests that employers have considerable latitude in choosing exams as long as they are highly loaded on these two factors, and also suggests approximately equal weighting.

Struggling to communicate the utility of sound assessment? Last but not least, Winkler et al. describe the results of sharing utility information with a sample of managers and found that using a casual chain analysis--rather than simply a single attribute--increased understanding, perceived usefulness, and intent to use.


Let's switch now to the December 2010 issue of the International Journal of Selection and Assessment (IJSA). There's a lot of great content in this issue, so take a deep breath:

Interested in unproctored internet testing? You'll want to check out Guo and Drasgow's piece on verification testing. The authors recommend using a Z-test over the likelihood ratio test for detecting cheating, although both did very well.

Walsh et al. discuss the moderating effect that cultural practices (performance orientation, uncertainty avoidance) have on selection fairness perceptions.

Speaking of selection perceptions, Oostrom et al. found that individual differences play a role in determining how applicants respond--particularly openness to experience. The authors recommend considering the nature of your applicant pool before implementing programs to improve perceptions of the assessment process.

Those of you interested in cut scores (and hey, who isn't) should check out Hoffman et al.'s piece on using a difficulty-anchored rating scale and the impact it has on SME judgments.

Back to perceptions for a second, Furnham and Chamorro-Premuzic asked a sample of students to rate seventeen different assessment methods for their accuracy and fairness. Not surprisingly, panel interviews and references came out on top in terms of fairness, while those that looked the most like a traditional test (e.g., drug, job knowledge, intelligence) were judged least accurate and fair. Interestingly, self-assessed intelligence moderated the perceptions (hey, if I think I'm smart I might not mind intelligence tests!).

And now for something completely different (those of you that get that reference click here for a trip down memory lane). A study by Garcia-Izquierdo et al. of information contained in online job application forms from a sample of companies found on the Spanish Stock Exchange. A surprisingly high percentage of firms asked for information on their applications that at best would be off-putting, at worst could lead to lawsuits, such as age/DOB, nationality, and marital status. The authors suggest this area of e-recruitment is ripe for scientist-practitioner collaboration.

Last but not least, a piece that ties the major topics of this post together: selection perceptions and recruitment. Schreurs et al. gathered data from 340 entry-level applicants to a large financial services firm and found that applicant perceptions, particularly of warmth/respect, mediated the relationship between expectations and attraction/pursuit intentions. This reinforces other research that has underlined the importance of making sure organizational representatives put your best foot/face forward.

Observer ratings of personality: An exciting possibility?

Using measures of personality to predict job performance continues to be one of the most active areas in I/O psychology. This is due to several things, including research showing that personality measures can be used to usefully predict job performance, and a persistent interest on the part of managers in tools that go beyond cognitive measures.

Historically the bulk of research on personality measures used for personnel assessment has been done using self-report measures—i.e., questionnaires that individuals fill out themselves. But there are other ways of measuring personality, and a prominent method involves having other people rate one’s personality. This is known as “other-rating” or “observer rating.”

Observer ratings are in some sense similar to self-report measures that ask the rater to describe their reputation (such as the Hogan Personality Inventory) in that the focus is on visible behavior rather than internal processes; in the case of observer ratings this takes on even greater fidelity since there is no need to “guess” at how behavior is perceived.

Research on observer ratings exists, but historically has not received nearly the attention given to self-ratings. Fortunately, in the November 2010 issue of Psychological Bulletin, Connelly and Ones present the results of several meta-analyses with the intent of clarifying some of the questions surrounding the utility of observer ratings and, with some caveats, largely succeed. The study was organized around the five-factor model of personality, namely Openness to Experience, Conscientiousness, Extraversion, Agreeableness, and Emotional Stability.

The results of three separate meta-analyses with a combined 263 independent samples of nearly 45,000 individuals yielded several interesting results, including:

- Self- and other-ratings correlate strongly, but not perfectly, and are stronger for the more visible traits, such as Extraversion and Conscientiousness, compared to those less visible (e.g., Emotional Stability).

- Each source of rating seems to contribute unique variance, which makes sense given that you’re in large part measuring two different things—how someone sees themselves, their thoughts and feelings, compared to how they behave. Which is more important for recruitment and assessment? Arguably other-ratings, since we are concerned primarily not with inner processes but with behavioral results. And it certainly is not uncommon to find someone who rates themselves low on a trait (e.g., Extraversion) but is able to “turn it on” when need be.

- Self/other agreement is impacted by the intimacy between rater and ratee, primarily the quality of observations one has had, not simply the quantity. This is especially true for traits low in visibility, such as Emotional Stability.

- Inter-rater reliability is not (even close to) perfect, so multiple observers is necessary to overcome individual idiosyncrasies. The authors suggest gathering data from five observers to achieve a reliability of .80.

- Observer ratings showed an increased ability to predict job performance compared to self-ratings. This was particularly true for Openness, but for all traits except Extraversion, observer ratings out-predicted self-ratings.

- Just as interesting, observer ratings added incremental validity to self-ratings, but the opposite did not hold true. This was especially true for Conscientiousness, Agreeableness, and Openness.

- Not only were observer ratings better at predicting job performance, the corrected validities were considerably higher than have been documented for self-report measures, and in one case—Conscientiousness—the validity value (.55) was higher than that reported previously for cognitive ability (.50), although it should be noted that the corrections were slightly different. Values before correcting for unreliability in the predictor were significantly lower.

Why do observer ratings out-perform self-ratings? Perhaps other-ratings are more related to observable job performance. Perhaps they do not contain as much error in the form of bias (such as self-management tendencies). Or perhaps because the measure is completely contextualized in terms of work behavior, whereas the questions in most self-report measures are not restricted to the workplace.

Despite these exciting results, some caution is in order before getting too carried away. First, the authors were unable to obtain a very large sample for observer ratings—sample sizes were around 1,000 for each trait. Second, there are currently a limited number of assessment tools that offer an observer rating option—the NEO-PI being a prominent example. Finally, there is an obvious logistical hurdle in obtaining observer ratings of personality. It is difficult to see how in most cases an employer would obtain this type of data to use in assisting with selection decisions.

However, even with that said the results are promising and we know of at least one way that technology may enable us to take advantage of this technique. What we need is a system where trait information can be gathered from a large number of raters relatively quickly and easily and the information is readily available to employers. One technology is an obvious solution, and it’s one of my favorite topics: social networking websites. Sites like Honestly (in particular), LinkedIn, and (to a lesser extent) Facebook have enormous potential to allow for trait ratings, although the assessment method would have to be carefully designed and some of the important moderators (such as degree of intimacy) would have to be built in.

This is an intriguing possibility, and raises hope that employers could soon have easy access to data that would allow them to add substantial validity to their selection decisions. But remember; let’s not put the cart before the horse. The question of what assessment method to use should always come after an analysis of what the job requires. Don’t jump headlong into other-ratings of personality when the job primarily requires analytical and writing skill.

This is just another tool to put in your belt, although it’s one that has a lot of exciting potential.

Multimedia on the cheap: Xtranormal and Toondoo

When it comes to recruiting, one of the most important things an organization can do is make themselves stand out from other employers. And one of the best ways to do this is by using creative multimedia technologies that demonstrate both creativity and an openness to innovative approaches.

I've written before about some of the exciting technologies out there that employers can use for this purpose, such as simulations and realistic videos. But today I'd like to talk about two more abstract--but perhaps more fun--websites that are easy to use and have a lot of potential.

Oh yes, and they're free (for the basic version). Free is good.

The first is Xtranormal.com. You may have seen some of these videos on YouTube or elsewhere, as it seems to rapidly have become the technology of choice for creating quick animated videos. They have both a web-based design option ("text-to-movie") as well as a more fully featured downloadable version called State.

As the designer, you simply choose a setting, the number of actors, and type in the script. The website adds computer-generated voices for you. You also have control over other features, such as the camera angles for each scene, emotional expressions, and sounds.

Now I'm no designer--as you will quickly see--but I was able to make this video in all of about 10 minutes*.



If you're intrigued, also try GoAnimate.com which is similar but I found to be slightly more cumbersome to use.

The other technology is slightly more old school--panel cartoons. Yes, like the ones you see in the paper.

Here's an example, which took me about 10 minutes using Toondoo:

By BB808 | View this Toon at ToonDoo | Create your own Toon

Neither of these websites are perfect. You'll find there's a small learning curve, and you'll wonder why they made certain design decisions. But there really is no reason not to at least try some of these tools.

*Note: In my day job I'm an HR Manager for the California Attorney General's office, but this blog is neither sanctioned nor supported by my employer. For better or worse, these animations were entirely my creation. But what kind of recruiter would I be if I didn't use this opportunity to promote, promote, promote!

November '10 J.A.P.

The November 2010 issue of the Journal of Applied Psychology is out, so let's take a look at the relevant articles:

Woods & Hampson report the results of a fascinating study looking at the relationship between childhood personality and adult occupational choice. The participants (N ~ 600) were given a 5-factor personality inventory when they were between 6 and 12 years old, then reported their occupation 40 years later using Holland's vocational types. Results? Openness/Intellect and Conscientiousness scores were correlated with occupational choice as adults. Implication? Career choice may be determined fairly early on in some cases and be influenced by how we're hard-wired as well as through friends, family, etc.

Perhaps even more interesting, the authors found that for the most strongly sex-typed work environments (think construction, nursing), the results related to Openness/Intellect were moderated by gender, supporting the idea that gender stereotyping in these jobs is impacted by individual differences as well as other factors (e.g., cultural norms).


Next up, a study by Maltarich, et al. on the relationship between cognitive ability and voluntary turnover, with some (what I thought were) counter-intuitive results. The authors focused on cognitive demands as the moderating factor, so think for a second about what you would expect to find for a job with high cognitive demands (think lawyer)--who would be most likely to leave, those with high cognitive ability or those with low? I naturally assumed the latter.

Turns out it was neither. The authors found a curvilinear relationship, such that those low and high in cognitive ability were more likely to leave than those in the middle.

What about jobs lower in cognitive demands? Who would you expect to have higher voluntary turnover--those with high cognitive ability or lower cognitive ability? I assumed high, and again I was wrong. Turns out the relationship the authors found in that case was more of a straight negative linear relationship: the higher the cognitive demands, the less likely to leave.

What might explain these relationships? Check out I/O at Work's post about this article for more details including potential explanations from the authors. It certainly has implications for selection decisions based on cognitive ability scores (and reminds me of Jordan v. New London).


Do initial impressions of candidates matter during an interview? The next article, by Barrick et al. helps us tease out the answer. The authors found that candidates that made a better impression in the opening minutes of an interview received higher interview scores (r=.44) and were more likely to receive an internship offer (r=.22). Evaluations of initial competence impacted interview outcomes not only with the same interviewer, but with separate interviewers, and even separate interviewers who skipped rapport building.

But perhaps more interestingly, the authors found that assessments of candidate liking and similarity were not significantly related to other judgments made by separate interviewers. Thus, while these results support the idea that initial impression matter, they also provide strong support for using a panel interview with a diverse makeup, so that bias unrelated to competence is less likely to influence selection decisions.


Finally, Dierdorff et al. describe the results of a field study that looked at who might benefit most from frame-of-reference training (FOR). As a reminder, FOR is used to provide raters with a context for their rating and typically involves discussing the multi-dimensional nature of performance and rating anchors, and conducting practice sessions with feedback to increase accuracy and effectiveness (Landy & Conte, 2009).

In this case, the authors were interested in finding out whether individual differences might impact the effectiveness of FOR. What they found was that the negative impact of having a motivational tendency to avoid performance can be mitigated by having higher levels of learning self-efficacy. In other words, FOR training may be particularly effective for individuals that believe they are capable of learning and applying training, and overall results may be enhanced by encouraging this belief among all the raters.


References:

Landy, F. and Conte, J. (2009). Work in the 21st century: An introduction to industrial and organizational psychology (Third Edition). Hoboken, NJ: Wiley-Blackwell.