R&A Software Failures Hurt Taxpayers, Too

We tend to think of successes and failures of applicant tracking systems and other recruitment- and assessment-related technologies as impacting businesses--much of what's written is about large organizations such as Microsoft and Google and what software they decide to adopt.

But public sector organizations are using these technologies as well. And when they fail, it hurts not only the organization but taxpayers as well.

Case in point: the state of Washington recently decided to abandon their efforts to implement SAP E-Recruiting after nearly three years and millions of dollars. The state will now go with a hosted solution which is estimated to be $700-800,000 a year cheaper (and hopefully much easier) to maintain.

Having been ringside for some of this, I can tell you the problem was not with motivation or energy, or even IT knowledge. I suspect that a lion's share of the problem was related to the complexity of the program. This would match reports I've read that a significant number of organizations are moving away from single-vendor HR solutions and going with simpler, targeted products. It's also possible that businesses find it easier to implement these programs because resources (particularly internal experts) are easier to move around and buy-off is easier to obtain.

I wish them luck on their next purchase, and hope they do due diligence in their research (you can often find others who have had problems). Some type of audit may help them determine exactly what went wrong and how to prevent it the next time around. It's not just a matter of time, energy, and expense on the part of the organization, these failures impact applicants, hiring supervisors, HR staff, and ultimately taxpayers.

SNWs for R&A professionals

In the August 2009 issue of IPAC's Assessment Council News I write about how recruitment and assessment professionals can take advantage of social networking websites (SNWs) such as LinkedIn, Facebook, and Ning, and offer some cautionary notes about using them.

The article covers what will be familiar ground to many of you, but I tried to also talk about how we can use these sites for professional development, not just for sourcing or selection.

The article ends with a very Web 1.0 idea--a solicitation of letters to the editor. I'll make the same request here: what do YOU think about these websites--flash in the pan or here to stay? Approach with kid gloves or jump right in?

Latest IJT and PP

The latest issue of the International Journal of Testing has some good stuff in it, particularly if you're in education. Here's a sample of what's available:

Correcting fallacies in (construct) validity, reliability and classification


Then head on over to Personnel Psychology to check out the Autumn issue. Here's some of what it covers:

Conscientiousness and KSAs predicts leadership in the military

Job component validation, meta-analysis, and DOT ratings

Personality variables and job search behavior and success

The importance of specificity and observability in job analysis ratings

Ricci presentation

On August 13th I gave a presentation at a PTC-NC luncheon about the Ricci decision. We had a great discussion about the implications (which remain to be seen) and dissected several passages from the decision.

One of the questions that came up had to do with the pass point. I didn't know the answer at the time, but looking back at the case it turns out that the City charter mandated a 70% pass point for these exams. Which is funny, because I made a joke about how 70% is the magic cutoff score given its ubiquity, particularly in the public sector.

In a perfect testing world would the pass point be based on an analysis of the minimum competency level required for the job? Yep. Did the 5-member majority in this case care? Nope.

You can view the (mostly visible) slides below.

September '09 IJSA

The September 2009 issue of IJSA (International Journal of Selection and Assessment) is chalk full of good stuff. Let's dive in.

1) An important update of the "guidelines and ethical considerations for assessment center operations"--a must for anyone interested in the appropriate use of assessment centers.

2) Speaking of assessment centers, here's a meta-analysis of how they correlate with cognitive ability and personality, as well as the proper way to weight the results.

3) Speaking of cognitive ability, curious about the correlation between ability and faking? Check out this large-sample study of faking on a biodata measure.

4) Worried about what your applicants think of your selection method? Frame it as select in (accept) rather than select out (reject).

5) Want to make sure your raters are rating accurately? You may want to re-think stocking your panel with agreeable people (sounds like a lot of fun for the exam analyst!).

6) Before you put the finishing touches on your new online job application system, make sure you pay attention to its features, user friendliness, and efficiency. I like to think of this as "Googley."

7) Looking for a measure of person-job fit that relates equity of contribution to reward? Check this out.

That's all for now!

Research update

Here are some research studies that you may have missed:

Predicting applicants' job pursuit behavior from their selection expectations: the mediating role of the theory of planned behavior

Complexity in the relationship among the subdimensions of extraversion and job performance in managerial occupations

The use of personality test norms in work settings: Effects of sample size and relevance

Networking as a job search behaviour: A social network perspective

Bonuses:

Avue recently published a white paper titled "The state of the federal workforce: A case for insourcing" that, while focusing on the federal government, has some good information and tips for all of us.

Can't get enough analysis of the Ricci decision? Check out this webinar on 8/14.

Reminder: Want more frequent updates? You can follow my shared items here.

July 2009 J.A.P.: SJTs and more

Situational judgment tests (SJTs) have a long tradition of successfully being used in employment tests. These types of (typically multiple-choice) items describe a job-related scenario then ask the test-taker to endorse the proper response. The question itself usually takes one of two forms:

1) What SHOULD be done in this situation? ("knowledge instruction")

2) What WOULD you do in this situation? ("behavioral tendency instruction")

What are the practical differences between the two? Previous meta-analytic research, specifically McDaniel et al.'s 2007 study, revealed that knowledge instruction items tend to be more highly correlated with cognitive ability, while behavioral tendency items show higher correlations with personality constructs. In terms of criterion-related validity, there appeared to be no significant difference between the two.

But there were limitations to that study, and two of them are addressed in a study found in the July 2009 issue of the Journal of Applied Psychology. Specifically, Lievens et al. addressed the inconsistency in stem content by keeping it the same while altering the response instruction, and also looked at a large population of applicants, rather than incumbents, which tended to dominate McDaniel et al.'s 2007 sample.

Results? Consistent with the 2007 study, knowledge instructions were again more highly correlated with cognitive ability, and there was no meaningful difference in criterion-related validity (the criterion being grades in interpersonally-oriented courses in medical school). Contrary to some research in low-stakes settings, there were no mean score difference between the two response instructions.

Practical implications? The authors suggest knowledge instruction items may be superior due to their resistance to faking. My only concern is that these items are likely to result in adverse impact in many applied settings. Like all assessment situations, the decision will involve a variety of factors, including the KSAs required on the job, the size and nature of the applicant pool, the legal environment, etc. But at least this type of research supports the fact that both response instructions seem to WORK. By the way, you can see an in-press version of this article here.

Other content in this journal? There's quite a bit, but here's a sample:

Content validity <> criterion-related validity

More evidence that selection procedures can impact unit as well as organizational performance

Self-ratings appear to be culturally bound

Ricci webcast on August 12

The Ricci v. DeStefano decision continues to generate a lot of interest. To help sort it all out, the Personnel Testing Council of Metropolitan Washington, D.C. (PTC-MW) will host Dr. James Outtz, renowned I/O psychologist and co-author of an amicus brief in the case, on August 12.

Not in D.C.? Not a problem. The luncheon presentation will be webcast at an extremely low price. Check out the website for details.

Coincidentally, a much less well known individual (yours truly) will also be presenting on the Ricci decision at PTC-Northern California (PTC-NC) at their August 13th luncheon.

By the way, check out some great commentary about the decision by several SIOP members here. I find it fascinating that SIOP came out strongly against the validity of the exam, to which the majority of the Supreme Court responded, "yawn."

HR, comic book style

You've always suspected that HR would make a great comic (graphic novel), right? Well turns out you're right.

Check out Super Human Resources

(first issue here)

(video preview is also here)

How can we improve executive selection?

Many of us would agree in the wake of recent financial meltdowns that much of the problem stemmed from poor decision making--presumably from the top down. We know a lot about how to select the right people, yet our best estimates peg leadership failures at around 50%. Are there ways we can use I/O expertise to improve this statistic?

This is the topic of the first focal article in the June 2009 issue of Industrial and Organizational Psychology, written by George Hollenbeck.

The author makes several excellent points, among them:

- The process of selecting executives is significantly dissimilar from how we select, say, entry-level hires. The decisions tend to be based more on "character"--essentially personality aspects with a little morality tossed in--more than standardized testing of competencies.

- I/O psychologists are rarely brought into the executive selection process, in large part because they don't "get" how selection decisions at this level are made. We tend to have an assessment or behavioral bent, whereas these decisions more often are holistic and highly subjective.

The author argues that we need to change our mindset to match more closely that of executives--we need to focus on character rather than competencies. The authors that provide subsequent commentaries agree that the focus on executive selection is timely, but some question the focus on character and others point out that predicting performance at this level is incredibly difficult given all of the environmental factors.

Yet after all this, I can't help but wonder (as do some of the commentary authors)...is it selection professionals that need to change their mindset, or should how we select executives look more like how we select entry-level hires? Maybe we'd all benefit from largely taking the judgment component out and relying more on standardized methods such as ability tests. But is that realistic? Are people at the top willing to admit that their judgment may be inferior to standardized tests?

How can we marry assessment expertise with the political and organizational realities inherent in executive selection? My bet is it lies with establishing quality relationships with the high-level decision makers. Become a trusted adviser, demonstrate the bottom-line value of sound assessment, and be flexible about applying our best practices. This is the kind of partnership that works with first-line supervisors; there's a good chance it will work all the way up the chain.

Ricci case: Full of sound and fury...

There's been a lot of hoopla over the last several days over the U.S. Supreme Court's decision in Ricci v. DeStefano. It's been described as a win for "reverse discrimination" cases, a rebuke of written tests, and judicial activism. The way I read it, the decision is completely unsurprising and will likely change absolutely nothing about employment testing.

For anyone who isn't familiar with the case, here's a very brief rundown: the City of New Haven, CT gave promotional tests for Lieutenant and Captain firefighter positions using written multiple choice tests and interviews. When they crunched the results it turned out--not surprisingly--that there was statistical evidence of adverse impact against the Black candidates. The City decided not to use the list, and the White and Hispanic candidates sued, claiming disparate treatment. The Supreme Court ruled in their favor.

A little unusual of a case in terms of who's on what side, and there's a lot of good reading in the decision for anyone wanting to know more about test validation. But the decision itself is totally consistent with three main themes from previous decisions:

(1) There really isn't "reverse discrimination"--there's just discrimination based on a protected classification, such as race, color, or sex. Majority groups are protected just like minority groups.

(2) Employers do not have to go to irrational lengths to validate their selection methods. Although the tests had flaws, the court continued to demonstrate that employers simply need to follow a logical process for developing the exam to show job relatedness; the exams don't have to win any awards.

(3) Disparate treatment by a government entity in order to avoid liability for adverse impact is legal only in certain very specific instances (when there is a "strong basis in evidence"). The court has been trending for years toward "color-blind" selection decisions.

About the only thing this case really points out is employers need to be ready to use the results from whatever test they administer, barring some enormous irregularities. That, and part of a defense against an adverse impact case might be that choosing not to use the exam would have been evidence of disparate treatment (I'll grant you that one's a little confusing).

All in all--and I'm certainly not the only one who feels this way--it doesn't appear to be anything to get excited about.

Want to know more? Check out the scotuswiki page.

SIOP Leading Edge Consortium

Feel like going to Denver in October?

That's when SIOP is going to have its annual Leading Edge Consortium. Previous consortiums have focused on executive coaching, innovation, talent, and leadership. This time we're fortunate that they've chosen to focus on Selection and Assessment in a Global Setting.

Speakers include individuals from companies like Cisco, Google, and Merck as well as consulting firms like SHL, Previsor, HumRRO, DDI, and Valtera.

Here are some of the session titles:

- "Global trends in HR"

- "Interviewing across cultures"

- "Cross border hiring"

- "Computerized adaptive testing"

Sure to be interesting stuff, particularly for anyone interested in attracting individuals from other countries and cultures.

Turn off qualified applicants in one easy step

Looking for a way to turn off qualified applicants in one easy step? The City of Bozeman, MT may have put its finger on it.

Turns out they have had--for several years--a requirement that all applicants seeking a position with the City must, after a conditional job offer that required a background check, turn over their ID and passwords for all social networks they're on, including Facebook and Twitter. After a firestorm of criticism, they decided to suspend the policy pending "a more comprehensive evaluation."

With all due respect to city officials...what were they thinking?

Put aside for the moment the potential problems of violating the terms and conditions of the social networking sites (which generally prohibit sharing passwords), and the potential legal issues inherent in finding information you shouldn't, what high-potential applicant worth her/his salt is going to give over their password information? Its akin to asking someone for their diary--and about as valid and relevant to job performance.

According to the City Manager, "choosing not to disclose log-in information did not hurt candidates’ chances of getting the job." Somehow I find that hard to believe.

I can appreciate wanting to perform your due diligence as part of the hiring process, and gathering as much information as you can, but there are tried and true methods of doing this, including detailed reference checks for every hire.

Maybe the proximity to great fishing interfered with judgment making.

Summer '09 Personnel Psychology

The Summer 2009 issue of Personnel Psychology covers a lot of ground. Take a look:

Kuncel & Tellegen demonstrate (with undergrads) that when inflating on personality inventories, people don't always max out their self-presentation; in fact for some traits a moderate level of endorsement is seen as more desirable.

Bledow & Frese describe how a situational judgment test can be used to predict not only overall job performance, but a particular construct--in this case, initiative. Participants were employees and supervisors at six banks in Germany.

This one particularly caught my eye. Yang & Diefendorff discovered (using ~200 employees in Hong Kong), among other things, that agreeableness and conscientiousness seem to moderate the relationship between negative emotions and counterproductive work behaviors (CWBs). Implication? If you're hiring for a job prone to negative emotions (e.g., customer service), consider adding a personality inventory to your screeening process to prevent CWBs.

De Pater, et al. studied both students and employees to determine that challenging job experiences reported by participants predicted promotability ratings above and beyond current job performance and job tenure. This has implications for both career development and performance management.

Want to know more about what executive coaches do? Then check out Bono et al.'s study of similarities and differences between practicing coaches that are also I/O psychologists versus those that aren't. (Turns out they do a lot of the same things)

Last but definitely not least, Aguinis et al. describe a web-based frame of reference training they used to decrease the amount of bias inherent in personality-based job analysis. The article describes in detail how the training was implemented, and it had quite dramatic effects. Useful stuff for anyone looking to add this tool to your assessment procedure (in this case they used Raymark et al.'s personality-related personnel requirements form, which they describe as superior to Hogan & Rybicki's performance improvement characteristics tool (which I've actually used and found quite user friendly).

Fast Company disses interviews

Those of you who know about research in personnel selection know that while interviews have been shown to be predictive of job success, several other types of selection mechanisms often out-perform them. Cognitive ability is often mentioned as the holy grail of predictors, but in terms of overall utility and defensibility, I recommend work sample exams. So do the authors of a recent article in Fast Company.

As the authors (who also penned Made to Stick) point out, interviewers are often snowed by candidate interview skills. Often only when you make them demonstrate their skills do their true strengths and weaknesses reveal themselves. (Of course if you're going to interview--and almost everyone does--make sure it's structured)

A couple strengths the authors leave out: work sample (sometimes called "performance") tests are easier to defend legally, since you're measuring an observable KSA rather than a construct like intelligence, and they give candidates a more realistic preview of the job. Heck, after doing a work sample a candidate may decide the job's not for him/her. Finally, they tend to be well received by candidates, more so than many other types of assessment.

This is my favorite quote:

"...figure out whether candidates can do the job. Research has consistently shown that one of the best predictors of job performance is a work sample. If you're hiring a graphic designer, get them to design something. If you're hiring a salesperson, ask them to sell you something. If you're hiring a chief executive, ask them to say nothing -- but reassuringly."

Enthusiasm? I'd rather see cautious optimism.

"I'm really excited about this job. Hurry up and pick me."

How would you feel if a candidate said that to you? A bit...confused? Well that's essentially what Paul Westphal told his new bosses in winning his bid to become the Sacramento King's new head coach.

Granted, Westphal had several things going for him:

1) He has been an NBA coach (Suns, Supersonics).

2) He's led a team to a winning season, something the Kings sure could use.

3) He's coached at the college level and also been an assistant coach; this should add some depth to his experience.

4) To his credit, he seemed to know what his new bosses wanted--i.e., enthusiasm and a reasonable salary. This could pay off in terms of his ability to get along with management (more about this below).

So what's the problem? Several, potentially. Here's how the article summed up the selection: "Westphal won the job largely on his NBA experience and enthusiasm for the job itself."

Here are my concerns:

1) Enthusiasm is not a proven predictor of job performance, yet his active pursuit of the job seems to have been a deciding factor. We know pure interest in the job does a horrible job of prediction. Pure experience isn't a great predictor either.

2) The search, according to the article, took only 47 days (which sounds quick to me). Yet apparently, "Westphal had grown impatient enough that sources say he was close to pulling out of the race." What does this say about an applicant? Maybe nothing. But it could signal something about personality (or desperation).

3) The screening seems to have relied primarily on interviews and "reputation." Is this the best way to pick a coach? What else might we do? (simulations, role plays, talking to previous players, etc.)

4) There's a big assumption being made here: that he was solely (or primarily) responsible for the wins of previous teams he coached. As we know, team performance doth not lie with leader alone. As one article commenter noted, the General Manager may be the common denominator leading in recent years to less-than-stellar team stats for the King. Will a new coach solve the real problem?

To be honest, I don't really want an applicant to be off the charts enthusiastic. It suggests overconfidence, a frightening lack of self-insight, or an attempt to snow me. Are there times where the enthusiastic candidate is the right one? Absolutely. All I'm suggesting is that we be wary. Personally, I'd rather see cautious optimism, which indicates an understanding that what they bring to the job is only part of the equation.

But heck, enough with the negativity. Here's hoping the Kings make it to the finals next year!

SIOP offers multimedia presentations

As part of its learning center, SIOP is now offering audio and video content from its conferences and events. Price ranges from $100-150 depending on membership status and whether or not you attended the event.

You can hear/see samples here, including presentations on personality in the workplace, reducing turnover using selection, and global talent management.

Good stuff. Hope other professional organizations follow their lead.

Emotional competence

I don't write a whole lot about emotional intelligence (EI), mostly because I still haven't seen a consensus around its conceptualization and measurement, but there continues to be significant interest in it. And on that note, there's an excellent article in a recent issue of JOB that I think is worth discussing.

In a nutshell, Kim, et al. studied nearly 200 matched subordinate-supervisor pairs in four South Korean hotels. The employees worked either at the front desk or were waiters--folks that likely would benefit from emotional competence.

Emotional competence, you say, not emotional intelligence? Yes, the authors prefer the term competence for several reasons:

1) Self-report inventories such as those used in this study may not be appropriate for measuring abilities.

2) Self-report measures usually measure typical behavior rather than maximally possible behavior, which an ability test hypothetically does.

3) Self-report measures of EC have low correlations with tests of cognitive ability.

I commend the authors for distinguishing between these concepts. In fact, it leads me to wonder whether we should go a step further and say emotional confidence or emotional report. However, this does raise some troubling issues with respect to the similarity and differences between the concepts and is a good illustration of why many I/O types shy away from this topic (it may also have something to do with there being many instruments that claim to measure EI).

Anyway, back the study. The measure they used for EC was a 16-item scale using 7-point Likert-type scales. An example question was, "I am sensitive to the feelings and emotions of others."

The authors found uncorrected correlations of .15 between EC and the two work performance measures of task effectiveness as well as social integration (both p<.05). In addition, they found support for their hypotheses that the relationship between EC and job performance was mediated by "interpersonal proactive behaviors", measured here by supervisors as the extent to which the employee engaged in feedback seeking behavior and relationship development with the supervisor. So not huge correlations, but useful. The strength of the correlation is in line with what we often see for uncorrected self-report measures such as personality inventories.

To their credit, the authors chose employees who likely would have need of some type of emotional awareness. This of course would be one of the big questions if one were considering this type of selection tool, and the decision as always would rest with the results of a detailed study of the job. What the Uniform Guidelines would have to say about supporting this measure using content validation is another story for another day!

Hybrid tests and the June '09 IJSA

What do you get when you combine a structured interview with a performance assessment? Perhaps some sort of hybrid with pieces from both sides. In the June 2009 issue of the International Journal of Selection and Assessment we find out more.

Morgeson et al. describe the development of a "performance interview" that combines a structured interview with an on-site performance demonstration. Essentially this involved going to the relevant work area (this study was for parts manufacturers) and asking a series of questions to determine promotability, such as "How do you set up this machine?" It's fascinating stuff, and it worked (using concurrent measures), although it might be challenging to use for less observable performance measures. For more details, check out the in press version here; the recipe book starts on page 12, and check out the example on page 41.

What else is in the issue? Take a look:

Predicting managerial readiness in Chinese workers

Is inflation in personality inventories necessarily a bad thing?

Do occupations tend to have their own personality? (yep)

Leadership effectiveness: Self- versus other-ratings (check out who tends to inflate)

CWBs: The organization plays a role

Biodata continues to shine (this time in healthcare organizations)

Is handwriting analysis popular among European employers? Not so much.

Can you predict military performance using personality inventories? Seems so.

Job experience v. personality measures in a small sample

Does going green matter to applicants?

A lot of people have recommended that organizations explicitly communicate an environmental (green) message to applicants in order to make the organization more attractive to applicants, particularly younger ones. But is this advice sound? Some new research suggests it may be.

In a recent article in the Journal of Business and Psychology, Behrend et al. had 183 participants view a printout of web pages that either did or didn't have a message expressing the organization's support of environmental causes.

First significant finding: the green web page was linked to more job pursuit intentions.

Second significant finding: the link was mediated by organizational reputation.

Third (and perhaps most important) finding: the link was not impacted by individual stance on the environment.

So there may just be something to the advice that greening up your career page can help you attract applicants. Assuming you want to attract more applicants.

Free monograph on test validation

IPAC (the International Personnel Assessment Council) is making available, free of charge, a monograph by Dr. Charles Sproule titled Rationale and research evidence supporting the use of content validation in personnel assessment.

Having seen a copy, I can tell you it's chalk full of great content, spanning much of the field of personnel assessment, including updated information on validity coefficients and special sections for several different types of tests (e.g., interviews, training and experience exams).

It's a great primer for anyone who wants to learn more about what it means to "validate" an exam, and it's a worthy addition to the library of any seasoned professional.

The monograph can be accessed here if you are a member, or you can request a copy here if you are not. Check it out!

Exploring the wild west of staffing

Last week I gave a presentation at the WRIPAC meeting in Burlingame, CA. The topic was "Using the Internet to effectively attract and screen the right applicants." Here are a few observations given my conversations with a very engaging group of participants:

1. There is immense interest in using the Internet for more than just posting jobs. But many folks are unsure how to start or what's out there. For individuals new to Web 2.0 concepts (e.g., social networking sites) there is a healthy--not necessarily unwarranted--skepticism.

2. Our worlds can shift relatively rapidly from "how do we get more applicants?" to "how do we get fewer applicants?" and then back again. As recruitment and assessment professionals we need to be able to pivot quickly and make sure our organizations are up to snuff in both realms.

3. In order to use the Internet effectively, most of us need to work with our IT staff. But this can be challenging given built-in resistances to change, security concerns, and assumptions. My advice was to push them as long as you have a good business case.

4. We have a lot of work to do on the most basic use of the Internet for recruitment and self-screening: our career portals. Many are cluttered and most likely have not been reviewed critically. More white space, more links, more pictures and video.

5. Most folks have some type of applicant tracking system, but many are unsure what its capabilities are and are not 100% sure how to use it to screen applicants. Oftentimes we rush to use built-in training and experience questionnaire functionality without first understanding the best way to use them. I just hope we move away from time-in-grade.

6. People seem to be moving away from the big job boards and towards cheaper--and usually more effective--alternatives, like craigslist or more targeted postings. People are also taking a harder look at whether their advertising dollars are really bringing in the candidates they need.

7. The balance of power has shifted in some ways from the employer to the applicant. They're not just responding to our (often uninspired) job postings, they're asking their friends, looking us up on glassdoor, connecting to people through LinkedIn, etc. As partial keepers of the organization's reputation, we need to be aware of all the different ways applicants research us.

8. These are exciting times. At several points in the presentation I mentioned that right now we are in the Wild West of staffing. Technology changes constantly, and along with them so do expectations placed upon us. We learn as we go. And it's challenging, but opportunities abound for those that are willing to take some risks.

Hang on, we're in for a heck of a ride.

April 2009 Issues of Merit

The U.S. Merit Systems Protection Board (MSPB) recently released their April 2009 Issues of Merit, with several interesting articles, including:

- Using Internet technology to recruit (public sector employers, are you on GovLoop?)

- Improving hires through multiple hurdles (e.g., accomplishment records)

- HR practices to increase employee engagement (supervisor selection, orientation, etc.)

- How to prepare a structured interview

Also, a very interesting note on the last page about OPM removing the time-in-grade requirement for promotions in competitive service!

Is your test the 40-yard dash?

A fascinating article came out in the last couple days about how the 40-yard dash is used to evaluate NFL prospects. And while I'm not a huge fan of equating sports teams with other organizations, sometimes the comparison works.

The article starts by describing the story of Brice McCain, a former defensive back from Utah. McCain was considered "too small" by football scouts until they came to town and saw him run the 40-yard dash; he did it in 4.30 and 4.34 seconds. As the article states, "Suddenly, his size (5-foot-9) was less of concern."

Consider these other points from the article:

- McCain's 40-yard dash time wasn't his only strength; he got high marks in other drills that assess quickness rather than overall speed (multiple-method convergence, anyone?)

- Scouts often feel you can teach things like catching, but teaching prospects to run faster is difficult (abilities vs. skills?)

- The 40-yard dash is considered more relevant for evaluating receivers and defensive backs than linemen, where scouts are looking more at foot speed and agility (job analysis informs assessment choice)

- Times for the dash vary with surface (grass v. artificial surface) and runners are never clocked wearing their uniform (beware fidelity of the test instrument)

- The importance of the test is debated given that few players ever run 40 yards during a game (some criterion-related validation might be in order)

- Businesses have sprouted up that provide physical training to prospects to help them perform better in front of scouts (test prep industry expands its reach)

There's quite a lot here that overlaps with assessment in general.

Where doesn't the comparison work? Well, non-sports organizations almost never have the wide variety of statistics available to them to use in judging applicants that sports scouts do. Many organizations also don't have recruiters constantly traveling around the country evaluating groups of applicants. And course there's that whole draft thing.

Still, an example of how assessment can be found in all kinds of situations, and how sticking to best practices pays off in a variety of situations.

The state of UIT

Many recruiters and assessment professionals believe that the future of employment testing lies with screening applicants over the Internet (rather than in person). And while there are substantial benefits associated with this method (e.g., convenience, speed), there are plenty of concerns as well (e.g., cheating, validity). Research in this area is in its infancy, which is why it is refreshing to see a full half of the March 2009 issue of Industrial and Organizational Psychology is devoted to describing the state of unproctored internet testing (UIT).

What struck me most about when reading these articles is the variety and excitement surrounding this field but even more, the tremendous lack of consensus in the professional community has about important issues related to UIT.

The articles start off with an updated summary by Nancy Tippins, who with her colleagues provided a heavily cited summary in 2006. This focal article is followed by twelve response articles and a final summary by Tippins.

Here's a (very) brief summary of some of the important points raised by the authors:

1. Cheating happens. But let's not forget that proctored tests have always been susceptible to some degree of cheating (e.g., via test question memorization).

2. There are many ways to mitigate the risks associated with cheating. This includes retesting, identity verification, and response pattern analysis. But it's not clear how successful these measures are, or even how needed they are.

3. Although there are potential legal risks (e.g., lack of standardized administration), UITs have not been directly evaluated in court.

4. The choice of whether or not to use UIT is influenced by many factors, not the least of which is the organizational reality communicated by upper management.

5. Some applicants may be turned off by an organization that uses assessments so obviously prone to cheating. But this may be balanced by increased convenience, speed, and immediate feedback.

6. Although cheating and response distortion occurs, it's unclear to what extent it impacts validity.

As an interesting note, the most common types of tests delivered via UIT seem to be biodata, personality, situational judgment, and T&E and preference questionnaires. There are also those who are administering cognitive ability tests in this way, sometimes adaptive.

UIT is in many ways the Wild West of employment testing. It's exciting and innovative, but comes with risk and lots of unanswered questions.

Here's hoping our field quickly speeds up the research side because this ship has clearly sailed.

HRE announces best HR ideas for 2009

Recently Human Resource Executive posted their Best HR Ideas for 2009 winners. Among the topics (benefits, HR tech, talent management, and T&D) they highlight some pretty interesting examples related to recruitment and assessment. Here's a sample:

- AlliedBarton Security Services gathered productivity data and discovered many managers were leaving at the peak of their productivity; this allowed them to focus additional resources on targeted retention efforts

- US Cellular uses web-based meeting technology to allow hiring supervisors to communicate about their candidates, ensuring more consistency and enhancing communication

- Proctor and Gamble, working with DDI, moved their cognitive ability testing online and made them adaptive

- SITA developed a leadership development portal where employees can literally "gauge" their competencies and receive targeted instruction

- BPO firm iQor created an online test designed to measure "emotional, intellectual, and conversational skills", reducing the need for interviews, increasing productivity, and reducing turnover

SIOP has a blog!

Well it's been a long time coming, but SIOP finally has a blog, the SIOP Exchange (with feed here).

Right now it's focused on updates from the conference, opening today in New Orleans, but moving forward it will broaden to "news and issues in the I-O community." Definitely a welcome addition to the blogosphere!

And speaking of the blogosphere, IPMA-HR's blog has a new location, along with a feed now. The most recent post is about a topic near and dear to me--social networking and performance management.

March '09 IJSA

The March '09 issue of the International Journal of Selection and Assessment is out, with a ton of great content. Check it out:

What's your organization's personality?

Does your career site motivate people who fit?

How does applicant personality interact with organizational personality?

More on faking non-cognitive assessments

Yes, we're still debating about group differences on cognitive tests

Oh, and if you're interested in statistics, check out this recent study on corrections for range restriction from PARE.

Biodata in the news

I don't write about biodata inventories very often. Not because they don't work--quite the opposite--but because like personality inventories, they require special knowledge and resources to develop. They also don't tend to be in the news nearly as much as other types of tests like interviews and personality inventories. So I was pleased to see a recent article in the mainstream press about how the FBI uses biodata inventories as part of its hiring process for Special Agents.

For those of you unfamiliar with this type of test, they look at first glance like a job knowledge-type multiple choice exam. The example given in the article is as follows:

To what extent have you enjoyed being given a surprise party?
A. Not at all
B. To a slight extent
C. To a moderate extent
D. To a great extent
E. I have never been given a surprise party

Like a personality inventory, a common reaction to this type of question is sometimes "What does this have to do with hiring someone?" But herein lies the beauty--and complexity--of biodata inventories: each item on the exam has been statistically linked to job performance. So unlike most other types of tests, items are shown to predict performance before being given to applicants.

Other benefits? They are resistant to faking so are great candidates for Internet administration and they typically result in smaller group differences than cognitive tests.

So why don't more organizations use them? A few reasons:

1) They require large sample sizes to develop
2) They require a fair amount of statistical expertise
3) They look a little funky ("why are you asking about my surprise party??")

Those are substantial hurdles for many organizations. But given the rush to online assessment and screening, we may see their usage increase.

What tweeting says about you

Cynical or spot on? Either way, Brian Unger's piece on Twitter is darn funny.

March '09 Journal of Applied Psychology

The March 2009 issue of J.A.P. is out with a lot of great content; for example:

To maximize diversity and validity, try recruiting on cognitive ability and selecting on conscientiousness

In a multistage assessment scenario (which exists for practically every hire), there are several options for maximizing diversity and validity

What determines the success of word of mouth recruitment strategies? Turns out, several things...
(read the draft version here and see Van Hoye's IPAC presentation here)

Interested in improving your T&Es? Check out what this study has to say about self and supervisory agreement of performance ratings.

More clues about what GMA is all about

How P-E fit is related to performance dimensions such as OCB

The importance of job offer negotiation: Subjective value predicts long-term employee outcomes

Recruiting using Web 2.0

Just gave a presentation at the PTC-NC conference on recruiting using Web 2.0 technologies. Similar to what I gave at WRIPAC last year but added a few things, including micro blogging. I'll be giving a much longer presentation at the May WRIPAC meeting where I'll talk about recruiting and screening using the Internet.

Great conference, lots of things to think about, including HR scorecards, misperceptions of testing, and the possibility that unstructured interviews may have more validity than we thought! My slides are below.

Feedback can Rypple

Every once in a while I post about something that's not directly related to recruitment and selection. Particularly if it's about a cool new technology. And when that cool new technology is free, well heck...I can't help myself.

Which brings me to Rypple. Rypple is a website that allows anyone to request anonymous feedback from people they work with (or, frankly, anyone). It's similar to Checkster (which also has a free feedback system called "Talent Checkup" with limited functionality) but has a different focus. Whereas Checkster has tools for both employers and individuals, Rypple is focused on the individual. The interface is also a little different, a little more micro-bloggy.

Why do I think this is important? Because we know that positive reinforcement is the best way to encourage the same behavior, yet many employees feel under-appreciated and lack the information they need to improve their performance and gain insight. Supervisors and managers are often unavailable or distracted, forgetting how important feedback is. Tools like Rypple put the power (or responsibility) with the employee.

You can easily see how this could become part of not only performance management but things like leadership development and team success measurement.

One last note: they are in beta, so the site and services may change. You can read about some of the developments on their blog.

Hat tip

New newsletter

There's a new newsletter in town. It's called EEO Insight, it's published by Biddle Consulting Group, and it focuses on EEO/AA issues, including employment testing.

Check out some the topics from the first issue (December '08):


The EEOC, OFCCP, and “Systemic Discrimination”: The Rules Have Changed

Where are the Courts Today? Proving and Defending Against an “Adverse Impact” Claim: OFCCP’S New Approach to Employer Selection Systems

Five Steps to Successful AAP Goal Development Diversifying Your Organization: How to Actually Make it Happen

Claims of Employment Test Validity: Who Can You Trust?


Good stuff. You can subscribe here.

What's in a name?

Personnel...Human Resources...Human Capital...Workforce Management...Talent Management...

If there's one thing we like doing, it's changing our name.

But do we ever consider the unintended consequences? Scott Adams does in this strip.

Does education level matter?

One of the most frequent questions related personnel assessment is does education level predict job performance?

The answer to this question has big implications for setting minimum qualifications for jobs, for screening criteria, leadership development--you name it, across the spectrum of talent management.

It seems like the answer would be fairly obvious--of course more education would result in better performance. People learn good study habits, increase their writing and oral communication skills, etc. etc.

There's just one thing: the research up until now hasn't shown this assumption to be true. For example, in Schmidt & Hunter's well known 1998 study, they found a (corrected) correlation between education level and job performance of .10--the same they found for interests and two and half times less than the predictive validity of reference checks. So not good.

But maybe it was the way they conducted the study. Or maybe education level predicts differently for different types of jobs, or maybe the way you measure performance matters.

That's a lot of maybes. So it was with much anticipation that I read Ng & Feldman's meta-analysis in the most recent issue of Personnel Psychology.

The authors found nearly 300 studies and looked at several measures of performance (both task and OCB) from a variety of sources (e.g., self-ratings, supervisor ratings, objective measures).

Results? Depends on your point of view. According to the authors, "...the results of this study suggest that using education level as a screening device has quite robust validity. In many cases, then, the higher recruitment costs and wage costs that typically accompany hiring highly educated workers are justifiable."

Reading that, you might expect the correlations they found to be quite high. They weren't. The correlation with supervisory ratings was quite similar to Schmidt & Hunter's at .09. Correlations with OCB varied, but the highest was .23. One of the worst correlations was with training program performance, at -.03

So what did education level predict? The highest values were for self-rated on-the-job substance abuse (-.28), creativity ratings (.25), "objective measures" (.24),and general absenteeism (-.22).

What about moderator variables? Nothing for job tenure, organizational tenure, or managerial nature of the job. As far as job complexity, results were mixed and effects not strong.

What's worse, on three out of the six relationships studied, the relationship between education level and job performance was more positive for either men or Caucasians compared to women and minority groups, respectively.

So bottom line? We still don't have good support that education predicts job performance. Certainly it does a much worse job compared to other assessment methods like ability testing or structured interviews. This doesn't bode well for using educational attainment for minimum qualifications nor for creating eligible lists or otherwise screening using education level. It may be that education has a stronger relationship with aspects of task performance that relate to education, such as written communication, presentation, and analytical skills. It's also possible that the type and level of the degree matters (something the authors point out but didn't analyze).

Of course it may be that education level is simply too broad of a measure to accurately predict all of the aspects that go into job performance. When it comes to attracting and screening, there are simply better ways to find the right person.

Strengths-based hiring and job posting customization

I don't usually just point to other articles, but there are two worth special mention.

First: For a while I've been meaning to do a post about the strength-based movement, typified by the StrengthsFinder tool and related books. But a recent post on The Science of Personality does a much better job than I could of discussing the pros and cons of this approach. It's actually an excerpt from an upcoming book on the subject.

And if you're interested in that subject, check out this article that provides a fairly balanced take on the subject.

My 1.5 cents: great idea for managers to focus a little more on strengths; we spend way too much time obsessing over behavioral problems. But not taking weaknesses/growth opportunities/whatever-you-want-to-call-them into account is seeing only half the equation.

Second: Jamie Madigan over at the SelectionMatters blog describes a recent article in Journal of Applied Psychology about the substantial benefits of customizing job postings; it's a great addition to an earlier study I posted about. Jamie has some great ideas on how to implement the results, and the authors provide additional suggestions in the paper, such as providing higher quality information to applicants. I look forward to further studies that investigate other strategies and use a broader subject pool (this study used undergraduates). It would also be interesting to look at using this approach for career development of existing staff. Finally, it will be interesting to see how well the job boards using a similar approach (e.g., JobFox) succeed.

Get in the game

Longtime readers know that I've considered one of the holy grails in our field to be a way of combining the interactivity and engaging content of video game technology (VGT) with recruitment and assessment. Yes, part of this is because I enjoy the occasional Nuka-Cola and killing the occasional troll, but there is so much potential in the marriage of these two fields that we can't ignore it.

Up until now, the best efforts have gone one of two ways. The first is creating an entire first person video game for recruitment purposes--this is what the U.S. Army did. The second is using VGT but in a very basic and limited way--this is what the FAA is doing. But to my knowledge no one has created a web-based tool that showcases the basic functionality of VGT while also serving as an assessment tool. In fact many people may not even know what this might look like.

Well I ran across something the other day (hat tip) that gets us pretty darn close. It's actually an onboarding program designed by Vestas, a Danish energy company. It takes the form of a situational judgment test (SJT) that leads new employees through an orientation of what Vestas does and their approach to their business.

I think once you've watched, you'll agree with me that the potential is vast.


So why this type of technology over, say, existing SJT solutions such as those offered by companies like Ergometrics and Biddle? Those definitely still have a place, and live actors are obviously higher fidelity, but here are some advantages to think about:

1. You can do more, and show more, with VGT. Need to show someone hanging onto the bottom of a helicopter then jumping to a rooftop? Not a problem, no wires required. Need to show someone underwater? Scaling a mountaineous peak? Again, much easier (and cheaper).

2. No screen actors required. No more worrying about makeup or getting the right shot--you create what you want. Of course voice talent is still very important if you decide to use sound.

3. It's just plain more modern. For folks that grew up watching cartoons and playing video games, they will naturally gravitate more toward something that feels familiar. Text job descriptions that link to an ATS? Yawn.

4. It will make you stand out. Yes, I know the unemployment rate is high here in the U.S., but don't think that means the end of competing for the most qualified. Now's the time to plan how you're going to compete when the pendulum swings the other way again.

5. It will stand the test of time. People still watch old cartoons. Very few old shows are on. That video shot of the desktop computer in the background may look outdated sooner than you'd like.

but perhaps most importantly:

6. VGT holds the promise of a truly interactive experience, where candidates explore their future work environment, make decisions, and learn about the organization. This has the potential to be both a realistic job preview that helps candidates decide whether to apply, as well as a measurement tool that gauges how well the candidate meets job requirements. (Yes this sounds a bit like Second Life but need not be so complex)


So what do we need to do moving forward? Here are some things we need to make this work:

1. More education. What do projects like this need to succeed? How much do they cost? What are the challenges and potential roadblocks?

2. Outreach to the VGT industry including the big companies (Activision Blizzard, EA, etc.) as well as the smaller shops, industry groups, schools, etc. No doubt they have much to teach us--but we have a lot to share as well. (As an aside, Activision has a very attractive Careers page that showcases some of their work, but they dump applicants right into their ATS like most companies--failed opportunity to continue the brand experience with a game-like character sheet!)

3. What are the psychometric implications? Is this just another version of unproctored Internet testing, or is there more here? How does this relate to run-of-the-mill adaptive testing? Are there demographic differences in willingness or performance?

Now what may throw a big monkey wrench into this is cost. Video games are not cheap (WOW cost $63M to develop). But we're not talking multi-user, latest video card, and all that stuff. This could be much shorter, more cartoonish, and much simpler.

I think this is the most exciting thing happening in assessment; I hope there are enough developers out there that agree.

Who are your pitchers and catchers?

One of the key pieces of advice I often hear in talent management (TM) circles is to focus on your high performers. But when it comes to selection, we often treat each position the same--each recruitment and hire should be based on job analysis, sound exam development, administration, scoring, etc. We often spend the same amount of time on non-critical roles as we do for critical roles, all in the name of defensibility and merit.

Well, a new study may make you re-think your resource allocation and decide to take that TM advice. The catch? You have to think about your group like a baseball team.

In the study, published in the January '09 issue of the Journal of Applied Psychology, the authors looked at 29 years of data from nearly 800 teams in Major League Baseball. The goal was to determine whether the experience and skill of, and resources allocated to, "core role holders" (CRHs) contributed more to team success than non-core role holders (non-CRHs).

What's a "core role"? The authors define this as one where the incumbent "encounters more of the problems facing the team, handles more of the work than other roles, and is central to the workflow of the team."

Who would this be on a baseball team? The authors chose pitchers and catchers, for several reasons--one big one is they are involved in every play. In many plays, they're the only players involved.

How did they define skill? Using rate statistics and a measure of defensive skill. The rate statistics were on base percentage for (catcher) and on base percentage against (pitcher). Choice of metric is obviously a big sticking point, but consider that the latter was substantially correlated (r=.69) with earned run average, another common way of measuring pitcher performance. The defensive skill chosen was error rate. The authors used a one-year lag so skill from one year was correlated with team performance in the next.

How about resource allocation? This was defined simply as the sum of the salaries of all players who filled the core positions from 1985-2002 (no salary data prior to '85).

Okay, enough about measures. What did they find (using hierarchical linear modeling)? Well, quite a lot of interesting things actually (particularly if you're a baseball fan), but for our purposes:

- Career experience, job-related skill, and team experience average were all positively related to team performance.

- Career experience of CRHs was more strongly correlated with team performance than career experience of non-CRHs.

- Job-related skill of CRHs was more strongly related to team performance than job-related skill of non-CRHs.

- Core resource allocation (salary of pitchers & catchers) added a small, but significant amount to team performance prediction (3%).

So what are the implications?

1) Certain positions are worth more of your time in terms of recruitment and selection (sounds intuitive but do you practice this?). When conducting job analysis, we should consider the overall strategic importance of the position, not just the tasks performed and competencies required.

2) Certain general qualifications are more important for certain positions. This has major implications for things like recruitment messages, minimum qualifications and other screening and assessment methods. It even begs the question of whether we need to revisit how we calculate criterion-related validity statistics.

3) Performance management for these positions, including top-notch onboarding, appraisal, compensation, and career development plans, are critical.

4) The selection of individuals onto teams should pay particular attention to the CRH positions.

For those of us that don't work for a professional baseball organization, what roles might be considered "core"? Obviously the organization in question matters a great deal, but considering the definition the authors use, this might include people such as receptionists, lead and senior positions, and people in QA. The situation is a little tricky since most of us don't work in groups that are directly competing against other groups, so the measurement of success might also impact the results.

One downside of the study is it didn't account for (nor does it claim to) other factors, such as the importance of leadership (in this case, the coach), personality factors, resources, reward and information systems, etc. Using their best model, the authors were able to account only for 30% of variance in team performance. I'd also be curious to see if the offensive and defensive measures differentially predicted success and whether pitchers' performance predicted team performance any better (or worse) than the catchers'.

Want to read more? Check out the in press version here.

Obama nominations illustrate lessons

The recent high profile withdraw of two of President Obama's nominees yields several lessons for employers and HR professionals about the hiring process. Here are a few:

1) Determine the real essential qualifications of the job. This includes more than just competencies; it includes things about the person that will add to their ability to get things done and work with others. For example, Daschle was widely seen as an expert on health care issues, but his tie to lobbyists and his tax problems made him an easy target.

2) Understand the environment the position is in. In Obama's case, the public was fed up with a perceived double standard that while average Americans struggled but paid their taxes, the wealthy--incredibly--failed to fulfill their tax obligations. Nominating three (if you include Geithner) individuals who exemplified this was sure to raise the ire of many Americans.

3) Don't rush the hiring process. A full assessment, which always includes reference checks, are vital. The more important the position, the longer the hiring process should be. It's not for me to judge Obama's vetting process, it's likely it was quite thorough. But for certain positions, no rocks should be left unturned.

4) Don't count your chickens. Planning the onboarding process is fine, but don't get too carried away with your plans for your new hire because they can pull out at any time. The hiring process is a two-way contract, and even in these difficult economic times, we shouldn't forget that.

5) You'll make mistakes--everyone does. Finding the right hire is just plain difficult in some cases. The more specific and rare the competencies, the harder it will be. Don't be too hard on yourself, just learn from your mistakes and try again.

6) People are watching. In Obama's case, every decision is broadcast on the 24-hour news cycle. While none of us live in that fishbowl, your current employees are always watching what you do when you make a hire (did they choose that person because they're friends? because they're attractive?). Oftenttimes your customers will be watching as well, to see what this says about your devotion to making merit-based hires. Have a sound hiring plan, and make sure it's transparent.