Employment testing: Imperfect but still invaluable

Imagine the following scenario. A detailed study is performed on a job. Observations of incumbents are made. Discussions are held with subject matter experts. Survey data are collected. The results all indicate that this job requires a high level of intelligence, extraversion and conscientiousness, customer service skill, and a fairly advanced ability to use computers. These attributes are, by far, the most important in predicting success on the job.

Taking all this information, we construct a rigorous assessment process. Candidates spend an entire day being observed as they take well-constructed ability and personality tests, participate in several scenarios that require them to demonstrate how they handle customer service situations, and are asked to produce several products on the computer using a variety of software packages. Raters are experts in the field, and the job, and use objective, behaviorally-anchored rating scales.

The results are then combined for an overall score, based on a formula generated from the initial study of the job. Those with the highest score are hired.

Given all the test scores, what is the maximum percentage of subsequent job performance that we can expect to predict?

a) 10%
b) 25%
c) 50%
d) 75%

Those of you in the field of personnel assessment can see this coming. Those of you that aren't, which option did you select? If you chose (b), move to the head of the class.

Why only 25%? Because tests aren't perfect, and because so much more than individual competencies go into predicting job performance. In fact, 10% is a much more commonly observed statistic.

25% doesn't sound like a lot. Does this mean we shouldn't use tests when hiring? Absolutely not. Because without well-developed assessments, you can expect to predict nothing--zero percent, nada.

In an article in the September 2008 issue of Industrial and Organizational Psychology, Scott Highhouse addresses one result of this low percentage: many hiring managers and HR professionals persist on using their own intuition and judgment using things like informal interviews to supplement (or replace) tests because of a stubborn belief that this will dramatically increase the likelihood of hiring the right candidate. The problem? They're wrong.

The available research makes clear that unstructured, overly subjective techniques such as resume review and informal interviews are simply not very predictive of job performance. So why do people keep using them? Highhouse suggests a key problem is our addiction to our own judgment. Commentators on the article point out other factors, such as lack of feedback, evolution, and different criteria of interest (e.g., hiring managers often care more about getting someone quickly rather than getting the "best" candidate).

In my experience the primary reason why decision makers insist on using less valid assessment methods comes down to ego, in two ways. One, people simply have a hard time admitting that tests do a better job of predicting things than they do. Second, we have an innate need to be involved in decisions impacting our lives. Who amongst us is willing to hire someone, unseen and unheard, based solely on test results? Even the most die-hard assessment fanatics among us have a difficult time with it, even though we know it would probably be for the best!

All in all, a highly recommended and provocative article that gets to one of the biggest challenges in personnel assessment: how assessment professionals and hiring managers can work together to find the right person for the job.

The best recruitment and retention tool: That touchy-feely stuff

Everyone who's anyone in the HR circles these days knows that to have a "seat at the table", be taken seriously, and avoid criticism, we need to ditch our reputation as touchy feely "people" people and focus on ROI, adding value, walking the walk, metrics, and all those other magical things that will somehow convince executives that we're worth our salaries. Right?

Here's the funny thing. Probably the biggest single factor in attracting and keeping the right people is your reputation--what people say about your organization. Referrals are time and time again cited as the most valuable source of high quality candidates (although we desperately need more non-survey research into this issue). And whose word carries the most weight? Current employees. So where should our focus be? Yep, you guessed it, our current employees.

So what does that mean? All that touchy feely stuff like job satisfaction surveys and wellness programs help ensure that your employees are happy (or as happy as you can make them). Happy employees are not just more likely to leave your customers with a positive impression, they're more likely to sing your praises to potential applicants. This positive reputation, combined with salaries that are at least in the ballpark, and clear communication about expectations and rewards, will go a long way toward your future success in attracting talent.

Using video games to recruit and select candidates

A new study by the Pew Internet and American Life Project found that:

"virtually all American teens play computer, console, or cell phone games and...the gaming experience is rich and varied, with a significant amount of social interaction and potential for civic engagement."

and

"Game playing is universal, with almost all teens playing games and at least half playing games on a given day"

This raises a question:

Is there a benefit, or even a mandate, to make recruitment and assessment more like a video game?

We've already seen a massive amount of interest in using virtual worlds like Second Life for recruiting (which has met with mixed success). And the U.S. Army is always on the cutting edge with things like America's Army (which has enjoyed quite a bit of success).

When it comes to assessment, we've seen some valiant efforts, such as the virtual job tryout. And video-based testing has been around for a long time.

But with everything that's out there, would you describe your candidate experience as "rich and varied" with a "significant amount of social interaction"?

Laying aside for the moment the fact that many organizations lack even realistic job preview videos, what competitive advantage is to be gained by the employer that figures out how to make its recruitment and selection process interactive? What if instead of the process being a one-way street (candidates search for information about employers, employers try to figure candidates out), it was a two-way simultaneous sharing of information?

Doom came out 15 years ago. The Sims, 8 years ago. Isn't it time we developed realistic 3-dimensional worlds that allow candidates to make real-time branching decisions and learn about a potential employer, while we measure things like attention to detail and judgment?

Is it just me or are we missing an enormous opportunity to attract a new generation of workers and gather valuable competency information at the same time?

What to look for in a leader

With the upcoming U.S. Presidential election, my co-workers and I have been spending a lot of time talking politics. We're of different political parties, so things can get pretty interesting.

Debates aside, we're all assessment professionals, so we keep coming to the same conclusion: it's difficult to figure out who would be the better president without an accurate job analysis. Without knowing what it takes to be good at a job, it's almost impossible to predict who would best fill it.

This topic isn't a new one for me; I wrote previously about whether "experience matters" in a leader, and how you would actually go about hiring one. But in a way that's putting the cart before the horse. Before we look at actual ways of finding a leader, let's look at what it takes to be one.

Of course many, many people have written about leadership and the KSAs or competencies it takes to be good at it, including respected voices in the assessment world. But new data is always welcome, which is why I was pleased to see a recent analysis by Doris Kearns Goodwin, author of Team of Rivals: The Political Genius of Abraham Lincoln, and one of the most thoughtful authors about the subject today.

In her recent article she outlines 10 attributes that she believes distinguishes "truly great Presidents." Using Abraham Lincoln and Franklin Roosevelt as exemplars (identifying and agreeing on high performers is a particular challenge in this case), she listed the following. Think about how these relate to leadership in your organization:

1. The courage to stay strong--to survive in the face of adversity and motivate oneself even when frustrated.

2. Self-confidence--the ability to surround oneself with experts, regardless of whether they agree with you or not.

3. An ability to learn from errors--to acknowledge and grow from mistakes rather than continue down a road of failure.

4. A willingness to change--not just by getting elected (or hired) but by going against previous tendences and preferences when situations require it.

5. Emotional intelligence--the ability to encourage others, take blame, and help others play to their strengths.

6. Self-control--either not letting events upset you, or pausing before responding when they do.

7. A popular touch--an awareness of where the citizens (employees) are and what they are (and aren't) ready for.

8. A moral compass--doing what's right, not just what's politically expedient or popular.

9. A capacity to relax--this includes the ability to diffuse your own emotions as well as relax those around you through tools like humor.

10. A gift for inspiring others--the ability to communicate broad goals shaped by history and context, shift opinions, and motivate.

So given these qualities, here are some things to think about:

- Are these attributes generalizable to leaders in your organization?
- Are these the types of things you're looking for when recruiting/hiring leaders?
- If so, are you assessing them in the right way?
- If you're a leader, do you embody these qualities?

HR goes to jail

It's extremely unusual for human resources employees to face personal liability for violating laws relating to recruitment and selection (constitional claims being one of the rare exceptions). It's even more rare for there to be criminal penalties associated with hiring; no one's going to jail for not showing a .30 criterion-related validity coefficient or for surfing Facebook for people under 40. But when it comes to immigration violations and child labor, the law doesn't mess around.

Case in point: on Tuesday, September 9th, two HR employees at Agriprocessors, a meatpacking facility in Postille, Iowa, were named in court documents and face both misdemeanor and felony charges for violating child labor and immigration laws.

Agriprocessors has been in the news for months ever since a May immigration raid found nearly 400 workers were in the country illegally. But up until now none of the management or HR employees had been in trouble.

The HR employees face charges for helping to hire children under 18 into dangerous jobs and helping applicants obtain identification using false documents. The misdemeanor charges associated with state child labor law violations carry a penalty of 30 days in jail and a fine up to $625.

The felony charges associated with immigration law violations, on the other hand, carry a prison term of 2-22 years and a $750,000 fine for one of the HR employees, and up to five years and $250,00 for the other if they are convicted. One of the more egregious charges was helping employees complete new paperwork using new names and false documents one day before the raid.

Sort of puts those large discrimination settlements into context, doesn't it?

One man's journey into Talent Management software

Several months ago I began looking into "talent management" software. To be honest, I wasn't sure exactly what I was looking for--or at--but I learned quite a bit along the way. In this post I'll give you a glimpse into my learning experience.

I began my journey looking simply for something that would allow my organization to inventory our existing talent--for example, by allowing employees to describe their own competencies and work preferences. My interest was piqued by this article in Workforce Management, which does a very good job of describing some of the major players and their products (they had me at "baseball card-style interface").

The first thing I learned was that there isn't a clear definition of what talent management (or "TM") is in the first place. For argument's sake, let's define it as documentation, analysis, planning, and decision making regarding how competencies are brought into and used within an organization.

Now I know many of you out there are thinking, "you're JUST NOW looking at TM products?" Well, yes, and I'll blame the fact that I work in the public sector and we tend to be a bit tardy to the game. But I actually think that's a good thing in this case, because of where the product development is right now.

Turns out the "talent inventory" function is only a small part of TM product offerings and is usually included in the career management portion. But what I have come to believe is this is one of the less important features of this type of software. What may be much more important is the performance management component.

Now, I know this is a blog on recruitment and assessment, so you may raise an eyebrow when I mention performance management, but bear with me. If we think broadly about talent management (i.e., it includes everything from branding to employee exit), how we place individuals and assess their performance is not only key to organizational success, but is the ultimate indicator of how successful our recruitment and assessment methods are.

So what's out there? Quite a few offerings, actually (it's a rapidly growing field). I looked at several products with my primary criteria being usability--if supervisors aren't going to use it, it's worthless. And, it turns out many of them have similar functionality--even similar cost--so it really came down to picking something that looked attractive and useful.

So, after watching many demos, and talking to many sales reps, here in roughly descending order of usability, are the companies whose products I looked at:

Sonar 6
SuccessFactors
CornerstoneOnDemand
Authoria
Taleo
Kenexa

This list is somewhat deceiving because all of these products tend to look the same and have similar abilities--except one. For me, only one of these products stands out in terms of design, and that's Sonar6. Why? White space, white space, white space. Simple but attractive graphics and GUI. Lack of 400 stemming text menus. I encourage you to check out the demo videos on all of their websites, but I think you will agree with me that their interface is simpler, more graphical, and more engaging.

Am I going to be purchasing one of these products? Hard to say at this point. But I will certainly keep you posted. So if you're interested in these products, take your time, ask a lot of questions, and think about how this will fit with your culture. And by the way, if you're as new to this field as I was, I strongly recommend the resources over at Bersin and Associates.

Free webinar on adverse impact

My colleague Dr. Jim Higgins shares my passion for providing education on topics relating to recruitment, testing, and selection.

So I'm pleased to draw your attention to an upcoming free webinar he's offering on Understanding Adverse Impact in Testing, Selection, Promotion, and Staff Reductions.

From the registration page:

"This free webinar will help you better understand the history, present and future of adverse impact analysis and will aid in your efforts to ensure that your organization takes the steps necessary to protect itself from claims of discrimination. It will also help you ensure that your organization’s hiring and promotional practices are maximally compliant with the letter and spirit of EEO laws and regulations."

The webinar takes place on September 16th at 11am and it's only 45 minutes. If you like what you see/hear, Jim offers a free (yes, free) 9-session course on basic applied statistics.

Power v. Group Differences

In a recent post I wrote about a chart my co-workers and I created to help us communicate with hiring supervisors about the pros and cons of various testing instruments. That graph mapped power (validity) on one axis, and speed of administration on the other.

One of the comments on that post mentioned it would be nice to see power vs. group differences. I agreed. So here it is!

The bottom line on this graph (no pun intended), if you're looking for the best combination of both, will be in the upper left quadrant.

A few notes of notes of caution before interpreting the graph:

- this graph charts only Black-White differences, which is the largest data set we have. It's important to remember that combinations of other groups (including gender) will yield slightly different results.

- the evidence on group differences for T&Es is rather scant. Not much has been found, but that doesn't mean it couldn't in the future, depending on what specific training or experience is being measured.

- finally, as the excellent recent article by Roth, et al. reminds us, adverse impact in your selection process depends on several factors, including the specific test or construct, the selection ratio, your applicant pool, and the order you place your assessments in.