Considering employee testimonials? Go video.

In the September 2009 issue of the Journal of Applied Psychology, one study stood out: the authors studied employee testimonials shown on recruitment websites. Results strongly suggest that:

(1) Including some type of testimonials increases your attractiveness as an employer; and

(2) Using more complex multimedia (video with audio) is clearly superior to simply pictures and text in terms of both attractiveness and credibility. This also helps mitigate any perceptual differences that occur when you increase the number of testimonials from minorities.

This is great validation for organizations that have put the time and effort into putting quality videos on their site.

Check out these other studies while you're at it:

Does recruitment method impact turnover? (short answer: yes, in the short run)

Interested in P-E fit? Check out this review and model development.

Like vocational interest inventories and statistics? You'll like this.

R&A Software Failures Hurt Taxpayers, Too

We tend to think of successes and failures of applicant tracking systems and other recruitment- and assessment-related technologies as impacting businesses--much of what's written is about large organizations such as Microsoft and Google and what software they decide to adopt.

But public sector organizations are using these technologies as well. And when they fail, it hurts not only the organization but taxpayers as well.

Case in point: the state of Washington recently decided to abandon their efforts to implement SAP E-Recruiting after nearly three years and millions of dollars. The state will now go with a hosted solution which is estimated to be $700-800,000 a year cheaper (and hopefully much easier) to maintain.

Having been ringside for some of this, I can tell you the problem was not with motivation or energy, or even IT knowledge. I suspect that a lion's share of the problem was related to the complexity of the program. This would match reports I've read that a significant number of organizations are moving away from single-vendor HR solutions and going with simpler, targeted products. It's also possible that businesses find it easier to implement these programs because resources (particularly internal experts) are easier to move around and buy-off is easier to obtain.

I wish them luck on their next purchase, and hope they do due diligence in their research (you can often find others who have had problems). Some type of audit may help them determine exactly what went wrong and how to prevent it the next time around. It's not just a matter of time, energy, and expense on the part of the organization, these failures impact applicants, hiring supervisors, HR staff, and ultimately taxpayers.

SNWs for R&A professionals

In the August 2009 issue of IPAC's Assessment Council News I write about how recruitment and assessment professionals can take advantage of social networking websites (SNWs) such as LinkedIn, Facebook, and Ning, and offer some cautionary notes about using them.

The article covers what will be familiar ground to many of you, but I tried to also talk about how we can use these sites for professional development, not just for sourcing or selection.

The article ends with a very Web 1.0 idea--a solicitation of letters to the editor. I'll make the same request here: what do YOU think about these websites--flash in the pan or here to stay? Approach with kid gloves or jump right in?

Latest IJT and PP

The latest issue of the International Journal of Testing has some good stuff in it, particularly if you're in education. Here's a sample of what's available:

Correcting fallacies in (construct) validity, reliability and classification


Then head on over to Personnel Psychology to check out the Autumn issue. Here's some of what it covers:

Conscientiousness and KSAs predicts leadership in the military

Job component validation, meta-analysis, and DOT ratings

Personality variables and job search behavior and success

The importance of specificity and observability in job analysis ratings

Ricci presentation

On August 13th I gave a presentation at a PTC-NC luncheon about the Ricci decision. We had a great discussion about the implications (which remain to be seen) and dissected several passages from the decision.

One of the questions that came up had to do with the pass point. I didn't know the answer at the time, but looking back at the case it turns out that the City charter mandated a 70% pass point for these exams. Which is funny, because I made a joke about how 70% is the magic cutoff score given its ubiquity, particularly in the public sector.

In a perfect testing world would the pass point be based on an analysis of the minimum competency level required for the job? Yep. Did the 5-member majority in this case care? Nope.

You can view the (mostly visible) slides below.

September '09 IJSA

The September 2009 issue of IJSA (International Journal of Selection and Assessment) is chalk full of good stuff. Let's dive in.

1) An important update of the "guidelines and ethical considerations for assessment center operations"--a must for anyone interested in the appropriate use of assessment centers.

2) Speaking of assessment centers, here's a meta-analysis of how they correlate with cognitive ability and personality, as well as the proper way to weight the results.

3) Speaking of cognitive ability, curious about the correlation between ability and faking? Check out this large-sample study of faking on a biodata measure.

4) Worried about what your applicants think of your selection method? Frame it as select in (accept) rather than select out (reject).

5) Want to make sure your raters are rating accurately? You may want to re-think stocking your panel with agreeable people (sounds like a lot of fun for the exam analyst!).

6) Before you put the finishing touches on your new online job application system, make sure you pay attention to its features, user friendliness, and efficiency. I like to think of this as "Googley."

7) Looking for a measure of person-job fit that relates equity of contribution to reward? Check this out.

That's all for now!

Research update

Here are some research studies that you may have missed:

Predicting applicants' job pursuit behavior from their selection expectations: the mediating role of the theory of planned behavior

Complexity in the relationship among the subdimensions of extraversion and job performance in managerial occupations

The use of personality test norms in work settings: Effects of sample size and relevance

Networking as a job search behaviour: A social network perspective

Bonuses:

Avue recently published a white paper titled "The state of the federal workforce: A case for insourcing" that, while focusing on the federal government, has some good information and tips for all of us.

Can't get enough analysis of the Ricci decision? Check out this webinar on 8/14.

Reminder: Want more frequent updates? You can follow my shared items here.

July 2009 J.A.P.: SJTs and more

Situational judgment tests (SJTs) have a long tradition of successfully being used in employment tests. These types of (typically multiple-choice) items describe a job-related scenario then ask the test-taker to endorse the proper response. The question itself usually takes one of two forms:

1) What SHOULD be done in this situation? ("knowledge instruction")

2) What WOULD you do in this situation? ("behavioral tendency instruction")

What are the practical differences between the two? Previous meta-analytic research, specifically McDaniel et al.'s 2007 study, revealed that knowledge instruction items tend to be more highly correlated with cognitive ability, while behavioral tendency items show higher correlations with personality constructs. In terms of criterion-related validity, there appeared to be no significant difference between the two.

But there were limitations to that study, and two of them are addressed in a study found in the July 2009 issue of the Journal of Applied Psychology. Specifically, Lievens et al. addressed the inconsistency in stem content by keeping it the same while altering the response instruction, and also looked at a large population of applicants, rather than incumbents, which tended to dominate McDaniel et al.'s 2007 sample.

Results? Consistent with the 2007 study, knowledge instructions were again more highly correlated with cognitive ability, and there was no meaningful difference in criterion-related validity (the criterion being grades in interpersonally-oriented courses in medical school). Contrary to some research in low-stakes settings, there were no mean score difference between the two response instructions.

Practical implications? The authors suggest knowledge instruction items may be superior due to their resistance to faking. My only concern is that these items are likely to result in adverse impact in many applied settings. Like all assessment situations, the decision will involve a variety of factors, including the KSAs required on the job, the size and nature of the applicant pool, the legal environment, etc. But at least this type of research supports the fact that both response instructions seem to WORK. By the way, you can see an in-press version of this article here.

Other content in this journal? There's quite a bit, but here's a sample:

Content validity <> criterion-related validity

More evidence that selection procedures can impact unit as well as organizational performance

Self-ratings appear to be culturally bound

Ricci webcast on August 12

The Ricci v. DeStefano decision continues to generate a lot of interest. To help sort it all out, the Personnel Testing Council of Metropolitan Washington, D.C. (PTC-MW) will host Dr. James Outtz, renowned I/O psychologist and co-author of an amicus brief in the case, on August 12.

Not in D.C.? Not a problem. The luncheon presentation will be webcast at an extremely low price. Check out the website for details.

Coincidentally, a much less well known individual (yours truly) will also be presenting on the Ricci decision at PTC-Northern California (PTC-NC) at their August 13th luncheon.

By the way, check out some great commentary about the decision by several SIOP members here. I find it fascinating that SIOP came out strongly against the validity of the exam, to which the majority of the Supreme Court responded, "yawn."

HR, comic book style

You've always suspected that HR would make a great comic (graphic novel), right? Well turns out you're right.

Check out Super Human Resources

(first issue here)

(video preview is also here)

How can we improve executive selection?

Many of us would agree in the wake of recent financial meltdowns that much of the problem stemmed from poor decision making--presumably from the top down. We know a lot about how to select the right people, yet our best estimates peg leadership failures at around 50%. Are there ways we can use I/O expertise to improve this statistic?

This is the topic of the first focal article in the June 2009 issue of Industrial and Organizational Psychology, written by George Hollenbeck.

The author makes several excellent points, among them:

- The process of selecting executives is significantly dissimilar from how we select, say, entry-level hires. The decisions tend to be based more on "character"--essentially personality aspects with a little morality tossed in--more than standardized testing of competencies.

- I/O psychologists are rarely brought into the executive selection process, in large part because they don't "get" how selection decisions at this level are made. We tend to have an assessment or behavioral bent, whereas these decisions more often are holistic and highly subjective.

The author argues that we need to change our mindset to match more closely that of executives--we need to focus on character rather than competencies. The authors that provide subsequent commentaries agree that the focus on executive selection is timely, but some question the focus on character and others point out that predicting performance at this level is incredibly difficult given all of the environmental factors.

Yet after all this, I can't help but wonder (as do some of the commentary authors)...is it selection professionals that need to change their mindset, or should how we select executives look more like how we select entry-level hires? Maybe we'd all benefit from largely taking the judgment component out and relying more on standardized methods such as ability tests. But is that realistic? Are people at the top willing to admit that their judgment may be inferior to standardized tests?

How can we marry assessment expertise with the political and organizational realities inherent in executive selection? My bet is it lies with establishing quality relationships with the high-level decision makers. Become a trusted adviser, demonstrate the bottom-line value of sound assessment, and be flexible about applying our best practices. This is the kind of partnership that works with first-line supervisors; there's a good chance it will work all the way up the chain.

Ricci case: Full of sound and fury...

There's been a lot of hoopla over the last several days over the U.S. Supreme Court's decision in Ricci v. DeStefano. It's been described as a win for "reverse discrimination" cases, a rebuke of written tests, and judicial activism. The way I read it, the decision is completely unsurprising and will likely change absolutely nothing about employment testing.

For anyone who isn't familiar with the case, here's a very brief rundown: the City of New Haven, CT gave promotional tests for Lieutenant and Captain firefighter positions using written multiple choice tests and interviews. When they crunched the results it turned out--not surprisingly--that there was statistical evidence of adverse impact against the Black candidates. The City decided not to use the list, and the White and Hispanic candidates sued, claiming disparate treatment. The Supreme Court ruled in their favor.

A little unusual of a case in terms of who's on what side, and there's a lot of good reading in the decision for anyone wanting to know more about test validation. But the decision itself is totally consistent with three main themes from previous decisions:

(1) There really isn't "reverse discrimination"--there's just discrimination based on a protected classification, such as race, color, or sex. Majority groups are protected just like minority groups.

(2) Employers do not have to go to irrational lengths to validate their selection methods. Although the tests had flaws, the court continued to demonstrate that employers simply need to follow a logical process for developing the exam to show job relatedness; the exams don't have to win any awards.

(3) Disparate treatment by a government entity in order to avoid liability for adverse impact is legal only in certain very specific instances (when there is a "strong basis in evidence"). The court has been trending for years toward "color-blind" selection decisions.

About the only thing this case really points out is employers need to be ready to use the results from whatever test they administer, barring some enormous irregularities. That, and part of a defense against an adverse impact case might be that choosing not to use the exam would have been evidence of disparate treatment (I'll grant you that one's a little confusing).

All in all--and I'm certainly not the only one who feels this way--it doesn't appear to be anything to get excited about.

Want to know more? Check out the scotuswiki page.

SIOP Leading Edge Consortium

Feel like going to Denver in October?

That's when SIOP is going to have its annual Leading Edge Consortium. Previous consortiums have focused on executive coaching, innovation, talent, and leadership. This time we're fortunate that they've chosen to focus on Selection and Assessment in a Global Setting.

Speakers include individuals from companies like Cisco, Google, and Merck as well as consulting firms like SHL, Previsor, HumRRO, DDI, and Valtera.

Here are some of the session titles:

- "Global trends in HR"

- "Interviewing across cultures"

- "Cross border hiring"

- "Computerized adaptive testing"

Sure to be interesting stuff, particularly for anyone interested in attracting individuals from other countries and cultures.

Turn off qualified applicants in one easy step

Looking for a way to turn off qualified applicants in one easy step? The City of Bozeman, MT may have put its finger on it.

Turns out they have had--for several years--a requirement that all applicants seeking a position with the City must, after a conditional job offer that required a background check, turn over their ID and passwords for all social networks they're on, including Facebook and Twitter. After a firestorm of criticism, they decided to suspend the policy pending "a more comprehensive evaluation."

With all due respect to city officials...what were they thinking?

Put aside for the moment the potential problems of violating the terms and conditions of the social networking sites (which generally prohibit sharing passwords), and the potential legal issues inherent in finding information you shouldn't, what high-potential applicant worth her/his salt is going to give over their password information? Its akin to asking someone for their diary--and about as valid and relevant to job performance.

According to the City Manager, "choosing not to disclose log-in information did not hurt candidates’ chances of getting the job." Somehow I find that hard to believe.

I can appreciate wanting to perform your due diligence as part of the hiring process, and gathering as much information as you can, but there are tried and true methods of doing this, including detailed reference checks for every hire.

Maybe the proximity to great fishing interfered with judgment making.

Summer '09 Personnel Psychology

The Summer 2009 issue of Personnel Psychology covers a lot of ground. Take a look:

Kuncel & Tellegen demonstrate (with undergrads) that when inflating on personality inventories, people don't always max out their self-presentation; in fact for some traits a moderate level of endorsement is seen as more desirable.

Bledow & Frese describe how a situational judgment test can be used to predict not only overall job performance, but a particular construct--in this case, initiative. Participants were employees and supervisors at six banks in Germany.

This one particularly caught my eye. Yang & Diefendorff discovered (using ~200 employees in Hong Kong), among other things, that agreeableness and conscientiousness seem to moderate the relationship between negative emotions and counterproductive work behaviors (CWBs). Implication? If you're hiring for a job prone to negative emotions (e.g., customer service), consider adding a personality inventory to your screeening process to prevent CWBs.

De Pater, et al. studied both students and employees to determine that challenging job experiences reported by participants predicted promotability ratings above and beyond current job performance and job tenure. This has implications for both career development and performance management.

Want to know more about what executive coaches do? Then check out Bono et al.'s study of similarities and differences between practicing coaches that are also I/O psychologists versus those that aren't. (Turns out they do a lot of the same things)

Last but definitely not least, Aguinis et al. describe a web-based frame of reference training they used to decrease the amount of bias inherent in personality-based job analysis. The article describes in detail how the training was implemented, and it had quite dramatic effects. Useful stuff for anyone looking to add this tool to your assessment procedure (in this case they used Raymark et al.'s personality-related personnel requirements form, which they describe as superior to Hogan & Rybicki's performance improvement characteristics tool (which I've actually used and found quite user friendly).

Fast Company disses interviews

Those of you who know about research in personnel selection know that while interviews have been shown to be predictive of job success, several other types of selection mechanisms often out-perform them. Cognitive ability is often mentioned as the holy grail of predictors, but in terms of overall utility and defensibility, I recommend work sample exams. So do the authors of a recent article in Fast Company.

As the authors (who also penned Made to Stick) point out, interviewers are often snowed by candidate interview skills. Often only when you make them demonstrate their skills do their true strengths and weaknesses reveal themselves. (Of course if you're going to interview--and almost everyone does--make sure it's structured)

A couple strengths the authors leave out: work sample (sometimes called "performance") tests are easier to defend legally, since you're measuring an observable KSA rather than a construct like intelligence, and they give candidates a more realistic preview of the job. Heck, after doing a work sample a candidate may decide the job's not for him/her. Finally, they tend to be well received by candidates, more so than many other types of assessment.

This is my favorite quote:

"...figure out whether candidates can do the job. Research has consistently shown that one of the best predictors of job performance is a work sample. If you're hiring a graphic designer, get them to design something. If you're hiring a salesperson, ask them to sell you something. If you're hiring a chief executive, ask them to say nothing -- but reassuringly."

Enthusiasm? I'd rather see cautious optimism.

"I'm really excited about this job. Hurry up and pick me."

How would you feel if a candidate said that to you? A bit...confused? Well that's essentially what Paul Westphal told his new bosses in winning his bid to become the Sacramento King's new head coach.

Granted, Westphal had several things going for him:

1) He has been an NBA coach (Suns, Supersonics).

2) He's led a team to a winning season, something the Kings sure could use.

3) He's coached at the college level and also been an assistant coach; this should add some depth to his experience.

4) To his credit, he seemed to know what his new bosses wanted--i.e., enthusiasm and a reasonable salary. This could pay off in terms of his ability to get along with management (more about this below).

So what's the problem? Several, potentially. Here's how the article summed up the selection: "Westphal won the job largely on his NBA experience and enthusiasm for the job itself."

Here are my concerns:

1) Enthusiasm is not a proven predictor of job performance, yet his active pursuit of the job seems to have been a deciding factor. We know pure interest in the job does a horrible job of prediction. Pure experience isn't a great predictor either.

2) The search, according to the article, took only 47 days (which sounds quick to me). Yet apparently, "Westphal had grown impatient enough that sources say he was close to pulling out of the race." What does this say about an applicant? Maybe nothing. But it could signal something about personality (or desperation).

3) The screening seems to have relied primarily on interviews and "reputation." Is this the best way to pick a coach? What else might we do? (simulations, role plays, talking to previous players, etc.)

4) There's a big assumption being made here: that he was solely (or primarily) responsible for the wins of previous teams he coached. As we know, team performance doth not lie with leader alone. As one article commenter noted, the General Manager may be the common denominator leading in recent years to less-than-stellar team stats for the King. Will a new coach solve the real problem?

To be honest, I don't really want an applicant to be off the charts enthusiastic. It suggests overconfidence, a frightening lack of self-insight, or an attempt to snow me. Are there times where the enthusiastic candidate is the right one? Absolutely. All I'm suggesting is that we be wary. Personally, I'd rather see cautious optimism, which indicates an understanding that what they bring to the job is only part of the equation.

But heck, enough with the negativity. Here's hoping the Kings make it to the finals next year!

SIOP offers multimedia presentations

As part of its learning center, SIOP is now offering audio and video content from its conferences and events. Price ranges from $100-150 depending on membership status and whether or not you attended the event.

You can hear/see samples here, including presentations on personality in the workplace, reducing turnover using selection, and global talent management.

Good stuff. Hope other professional organizations follow their lead.

Emotional competence

I don't write a whole lot about emotional intelligence (EI), mostly because I still haven't seen a consensus around its conceptualization and measurement, but there continues to be significant interest in it. And on that note, there's an excellent article in a recent issue of JOB that I think is worth discussing.

In a nutshell, Kim, et al. studied nearly 200 matched subordinate-supervisor pairs in four South Korean hotels. The employees worked either at the front desk or were waiters--folks that likely would benefit from emotional competence.

Emotional competence, you say, not emotional intelligence? Yes, the authors prefer the term competence for several reasons:

1) Self-report inventories such as those used in this study may not be appropriate for measuring abilities.

2) Self-report measures usually measure typical behavior rather than maximally possible behavior, which an ability test hypothetically does.

3) Self-report measures of EC have low correlations with tests of cognitive ability.

I commend the authors for distinguishing between these concepts. In fact, it leads me to wonder whether we should go a step further and say emotional confidence or emotional report. However, this does raise some troubling issues with respect to the similarity and differences between the concepts and is a good illustration of why many I/O types shy away from this topic (it may also have something to do with there being many instruments that claim to measure EI).

Anyway, back the study. The measure they used for EC was a 16-item scale using 7-point Likert-type scales. An example question was, "I am sensitive to the feelings and emotions of others."

The authors found uncorrected correlations of .15 between EC and the two work performance measures of task effectiveness as well as social integration (both p<.05). In addition, they found support for their hypotheses that the relationship between EC and job performance was mediated by "interpersonal proactive behaviors", measured here by supervisors as the extent to which the employee engaged in feedback seeking behavior and relationship development with the supervisor. So not huge correlations, but useful. The strength of the correlation is in line with what we often see for uncorrected self-report measures such as personality inventories.

To their credit, the authors chose employees who likely would have need of some type of emotional awareness. This of course would be one of the big questions if one were considering this type of selection tool, and the decision as always would rest with the results of a detailed study of the job. What the Uniform Guidelines would have to say about supporting this measure using content validation is another story for another day!

Hybrid tests and the June '09 IJSA

What do you get when you combine a structured interview with a performance assessment? Perhaps some sort of hybrid with pieces from both sides. In the June 2009 issue of the International Journal of Selection and Assessment we find out more.

Morgeson et al. describe the development of a "performance interview" that combines a structured interview with an on-site performance demonstration. Essentially this involved going to the relevant work area (this study was for parts manufacturers) and asking a series of questions to determine promotability, such as "How do you set up this machine?" It's fascinating stuff, and it worked (using concurrent measures), although it might be challenging to use for less observable performance measures. For more details, check out the in press version here; the recipe book starts on page 12, and check out the example on page 41.

What else is in the issue? Take a look:

Predicting managerial readiness in Chinese workers

Is inflation in personality inventories necessarily a bad thing?

Do occupations tend to have their own personality? (yep)

Leadership effectiveness: Self- versus other-ratings (check out who tends to inflate)

CWBs: The organization plays a role

Biodata continues to shine (this time in healthcare organizations)

Is handwriting analysis popular among European employers? Not so much.

Can you predict military performance using personality inventories? Seems so.

Job experience v. personality measures in a small sample