What's hot in I/O?

You can often get a sense of what's hot in a field by looking at conference presentations or workshops.

Recently I got the registration materials for the 2009 SIOP conference, and it includes a description of the seminars and workshops. Here's a sample:

Frontiers of personality research and practice

Making HR measurement strategic (by the authors of this book)

Reliability, ratings, and reality: Oh my!

Development in place: Leveraging the other 90% of your organization's talent

The future of HR metrics: It's a brave new world

O*NET products and tools: What's new and what's useful for your research and practice (O*NET is here)

Selection of first-line supervisors: What we know

Exploring new frontiers in test security: Approaches for protecting your testing program

Hmmm....sounds like metrics are here! As always, I'm sure the conference will be a great learning and networking opportunity. Feel like going to New Orleans in April?

We're missing half the equation

Matching the right person with the right job has two main components, one focused on the employer, the other on the applicant:

#1) A careful analysis of the job by the employer to identify the critical tasks and essential KSAs/competencies. This allows us to develop selection mechanisms that filter applicants to identify the most qualified.

#2) An accurate and thorough description of the job so the applicant can decide if there is a good match between the job and their qualifications.

We tend to do a pretty good job on #1. But many of us completely fall down when it comes to #2. We assume that the job and the organization have been described sufficiently so that applicants--qualified and not--can properly select in and out. But I think we're deluding ourselves.

Ask yourself this: What do you do to make sure applicants have a feel for the job? Let's think about this like we might a selection mechanism, from least to most fidelity:

(a) Describe the job/organization using text description of duties and qualifications
(b) Describe the job as above, also include benefits and "what's in it for you"
(c) Same as above, plus pictures of the work environment
(d) Same as above, plus other multi-media like videos and podcasts
(e) Same as above, plus Web 2.0 engagement strategies, like employee blogs
(f) Allow the applicants to perform simulated job duties online
(g) Alllow applicants to perform simulated job duties in person offsite
(h) Allow the applicants to perform actual job duties on the job site

I'll bet most of us would say (a) and (b) are standard, we have (c) on a good day, and we're moving slowly toward (d) and maybe (e). But this is, frankly, insufficient. The modern job seeker has access to a mountain of information--but desperately lacks information from the horse's (employer's) mouth. The reality is if you don't give it to them, someone else will.

What other benefits are there? Self-screening for one. The fewer unqualified people you have "in the pipe", the less time you waste screening out (not to mention the less time candidates waste, which they greatly appreciate).

By this point you might be wondering what these pictures are. They're from a children's museum I visited recently. They had all these leftover pieces of equipment for children to play with. Thing is, it was fun for the adults too. It gives you a real sense of what it would be like to sit in a helicopter, ride a police motorcycle, and drive a bus.

Why don't we have something like this geared toward adults? This goes way beyond an open house. It could be part of a career fair, or even better, employers would maintain a standalone site where applicants could get a clearer picture of what they'd be doing.

Now not everyone is going to be excited about a day in the life of a cubicle dweller, but everyone works somewhere--why aren't we doing a better job of explaining what it's like? Until we recognize the applicant's perspective, and the impact this has on getting the right person-job match, we're missing half of the equation.

Finally, for those of you that follow research, here's a listing of journals; half focus on the employer, half more on the job seeker. Which ones do you typically read?

Research round up

There has been so much good research coming out that rather than attempting to write a summary for each of them, allow me to simply present titles and links for you. I'll bet you see something you'll want to follow up on:

Cognitive and emotional processes in individuals and commercial web sites (how people respond to banner ads)

Occupational embeddedness and job performance (interesting results for people that are strongly linked to their career field)

Age stereotypes in the workplace: Common stereotypes, moderators, and future research directions (qualitative summary)

The hidden prejudice in selection: A research investigation on skin color bias

Does stereotype threat affect test performance of minorities and women? A meta-analysis of experimental evidence (really good stuff about moderators)

Stereotype threat reinterpreted as a regulatory mismatch.

Does socioeconomic status explain the relationship between admissions tests and post-secondary academic performance? (short answer: no)

Similarity and assumed similarity in personality reports of well-acquainted persons. (depends on which HEXACO factor you look at)

Not all conscientiousness scales change alike: A multimethod, multisample study of age differences in the facets of conscientiousness. (hint: there's something special about orderliness)

Enjoy!

Living with T&Es

There are a number of ways to use the Internet to perform personnel assessment. Examples include timed cognitive and job knowledge tests, biodata instruments, and personality inventories. But one of the easiest--and thus most tempting to use--types of tests is known as training and experience questionnaires, or T&Es (also sometimes called E&E for education and experience).

A typical (poor) example of a T&E item might be something like this:

How much experience do you have conducing job analysis?

a) None
b) Less than 2 years
c) 2-4 years
d) 4 or more years

I could go on at length about the challenges inherent with using this type of assessment, but I'll spare you. Instead I'll point you to Jim Higgins' December 2008 newsletter, HR Rampage, in which he addresses this topic, among others (see page 2).

Jim points out several problems with this type of assessment, including the overwhelming urge to self-inflate in high-stakes scenarios (we already have problems with outright cheating), the inability of highly qualified individuals to give themselves sufficient credit, and the work required to validate responses.

So given all these challenges, what can we do to mitigate them? The solutions investigated so far (e.g., elaboration, warnings) have met with very limited success. But there are a number of tactics we can take in this situation. Here are some other methods to consider as we wait for more research in this area:

1) Accurately describe the job and requirements to prevent an unqualified individual from applying in the first place.

2) Clearly word stems and responses to avoid legitimate mis-reads.

3) Include lie items (e.g., "I have experience using the HR Tests Job Analysis Technique") and deduct points when candidates endorse them.

4) Use false bottoms (e.g., both (a) and (b) are worth zero points) and false tops (e.g., both (c) and (d) are worth the same).

5) Use scales appropriate to the item. For example, amount of experience is often the incorrect scale; type of experience is better.

6) Encourage hiring supervisors to follow up on specific items in their interview.

7) Use a friendly zero point, such as "I do not have any experience but I would be willing to learn" instead of "none." Remember there are egos involved here.

8) Ask questions that are appropriate for a T&E. Don't ask candidates to rate their oral communication skills.

9) Don't ask people to compare themselves to others (e.g., Average, Above Average). Instead use objective measures such as number of times.

10) Seriously consider weighting the items. This is of course dependent upon subject matter expert input, but it's highly likely that your SMEs consider certain training or experience areas more important than others.

11) Before they even get to the T&E, use willingness/pre-screening questionnaires that ask candidates to acknowledge they understand the less-than-perfect conditions related to the job (e.g., mandatory overtime, working outside in the heat).

12) Consider using them as a feedback tool for candidates rather than a scored instrument (e.g., "Your responses indicate you have very little experience and education related to this job. Would you like to continue to apply?").

13) Base everything on SME input. Yes, I realize this probably doesn't need to be stated, but one of the worst temptations is for HR folk to draft T&Es themselves thinking they're easy to write. This is a myth, and helps contribute to poor quality eligible lists.

This is by no means an exhaustive list, and I'm sure there are other methods out there for helping us live with T&Es. Feel free to add your suggestions!

ATP Conference Feb 22-25

The Association of Test Publishers (ATP)'s annual conference is February 22-25 in Palm Springs, CA and if you have even a passing interest in personnel assessment, you owe it to yourself to check it out.

There's a lot more there besides I/O--they have tracks for education, clinical, and certification/licensure as well--but take a gander at some of these sessions:

Test development in a global economy: A practical guide to best practice in test construction, localization and adaptation.

Innovation and going green: Use of web-enabled technologies in applicant screening.

Developments in innovative technology-based assessments.

This is just the tip of the iceberg, there's tons of great content planned. It even looks like they know how to have a little fun. Cost is $800 for non-members ($725 for members).

Old school competencies part 2

I seem to have a thing for noticing signs in museums that remind me of a job analysis. Either I'm developing an appreciation for our nation's history or I'm obsessed with my subject matter. Maybe a bit of both.

Anyway, this weekend I visited the California State Railroad Museum and happened to notice this sign:


It reminded me of several things:

(1) Analyzing job requirements, whether you want to call them KSAs, KSAOs, KSAOPs, competencies, or whatever, is not new.

(2) Despite this, and despite efforts of the feds and well-known vendors, we still don't have a common language when we talk about job requirements. Non-HR folk tend to be more attracted to the words used in this poster--words like "people skills" and "perseverance." We need to seriously de-jargon ourselves.

(3) Many HR professionals are still stuck in the paradigm of (1) analyze job, (2) test for few relevant competencies, (3) choose person with best score. As the last paragraph of this picture points out, we need to remember that many different competencies go into job performance and very few jobs are identical. If a highly competent person drops in your lap, are you able to plug them in where you need them, or do you shunt them to a website where they spend 30 days going through your application process? Are you looking for competencies or people?

(4) I really need to stop using my phone to take pictures.

LAPD uses economy to recruit

Looking for an example of how to take advantage of tight economic times to increase recruitment efforts? Look no further than the Los Angeles Police Department (LAPD), who--accordingly to a recent article--have increased their applicant numbers over 100% compared to last year.

How did they do this? A good career site helps, and high unemployment is certainly a factor, but it's one that LAPD is tapping directly through advertising language such as this that has been posted in newspapers and on the web (bold added):

"During these times of economic uncertainty, the Los Angeles Police Department is always looking for a few good women and men to protect and to serve our communities. Never have to face a layoff again! Start your new career today!"

Subtle, huh? But apparently pretty darn effective.

IPMAAC is now IPAC

As of January 1st, 2009, the professional organization formerly known as the IPMA-HR Assessment Council (IPMAAC) has changed its name to IPAC--the International Personnel Assessment Council.

Who?

IPAC is a group of professionals devoted to the science and practice of employee assessment. That pretty much covers it. Smaller, more accessible, and more focused than SIOP. Larger than say a regional PAC or PTC.

Why the change?

IPMA-HR restructured and decided having IPMAAC a Section no longer fit with their mission. IPMA-HR targets HR heads, while IPMAAC tends to focus on practitioners.

Why join?

1) Networking. IMHO the biggest benefit. Great group of people dedicated to their field, extremely knowledgeable, and willing to lend a hand. IPAC's forum is one of the best.

2) Top notch conference. If you're looking for the latest research and best practices on employee selection, this is where it's at, and members get a discount. Check out some of last year's presentations.

3) Leadership opportunities. Looking to distinguish yourself in the field through a leadership post? IPAC's always looking for people to step up.

4) Training. Staying on top of this field can be challenging. Hooking yourself into an organization dedicated to being current is a great idea.

5) Speaking of staying up-to-date, did I mention the monthly newsletter?

Interested in joining? Head on over to this page.

Most popular posts of 2008

As we say goodbye to 2008, I thought it time to review which topics people seem to be the most interested in--on this blog at least.

So without further ado, here, in descending order, were my top 10 most popular posts of 2008:

1. Adverse impact on personality and work sample tests

2. Can the Dewey Color System be used for hiring?

3. What the new ADA law means for recruitment and assessment

4. What does your selection process say about your culture?

5. A review of situational judgment tests

6. B = f (P, E)

7. One man's journey into Talent Management software

8. Real world value of strategic HRM practices

9. Grading topgrading

10. Staffing.org's 2008 benchmark report

So what do I take from all this? Reader interest varies, but combined with the sidebar poll on the homepage, it remains obvious that personality testing is far and away the most popular topic. Hiring managers continue to be very interested in selecting candidates for more than just their ability, and HR professionals continue their quest for the best way to do this. There are very good personality tests out there, but I get the feeling no one's discovered the holy grail here yet.

What I'd like to see in 2009 is much more emphasis on candidate self-selection. Realistic job previews (descriptions and multi-media), assessments used for feedback rather than selection, and much much more. It's a very exciting direction, so let's get to it!

So Happy New Year, keep up the good work, and thanks for reading!

Oh, and please complete Dr. Charles Handler's survey. Thanks!

Holiday HR Humor

One of my favorite games at work is HRspeak. Using a few words and phrases that seem to be overly popular in some HR circles, we create sentences that sound good but are practically meaningless.

For example:

"Hey, watcha up to?"

"Not much, just transforming our operational cluster into customer-centric business practices that strategically meet our mission-critical goals using knowledge-based ROI-driven solutions."

Why this is funny (at least to a very small segment of the population) is because an even smaller segment of the population actually finds language like this to be practically useful. Typically in my experience it's HR consultants who are trying to convince HR to reinvent itself. The concept may be sound, but the words get in the way.

As usual, Scott Adams picks up on this phenomenon (and makes it much funnier than I can) in a recent Dilbert strip.

This time of year, often when people reflect and plan, let's all try to communicate a little more clearly.

And here's hoping that you and yours enjoy a joy-filled, family and friend-centric season focused around cuisine-based activities that meet your critical soul-based needs.

The future of selection

Last week I presented at the Personnel Testing Council of Northern California's monthly luncheon. The title was "Selection in a changing world: What will we be doing, and who will be doing it?"

The topic was motivated mostly by my own experiences related to what's going on in our field lately. With many of us facing severe budget challenges, I got to thinking about things like:

- What do we need to be doing, what should we be doing, and how can we add value in lean times?

- What impact does automation (e.g., applicant tracking systems) have on the work we do?

- What impact does automation have on the competencies we need?

The field of HR has been talking about transformation for years now, with IMHO only partial success. It's time for assessment professionals to take a look at ourselves and determine if we're where we need to be.

You can see the slides here:



They should also be posted soon on PTC-NC's website.

Resume <> Personality

I don't know about you, but one of my least favorite forms of assessment is pouring through resumes. They're not standardized, they leave out important details, and often provide way too many details about things we don't care about. But most importantly, it just doesn't feel like a very valid way of making inferences about candidates.

There are good reasons to dislike this activity. Not only are there rampant self-inflation problems, the inferences recruiters tend to make about applicant personality are erroneous, according to a recent study. After looking at responses from 244 recruiters, the authors found several important results:

1) Low interrater reliability -- in other words, the recruiters didn't agree with each other very often about what the resume said about an applicant's personality.

2) When correlations were made between recruiters' inferences of personality with actual Big 5 scores from the applicants, low levels of validity were found (slightly better for conscientiousness and openness to experience).

3) Despite the two findings above, rater perception of extraversion, oppenness to experience, and conscientiousness predicted their assessment of employability of the applicants.

Lesson? Be very careful what you imply from a resume. Think carefully about the facts you're using to infer personality. If you must use them, screen out only those who lack the basic qualifications to do the job. Follow up the resume screen with a number of much more valid assessments--work sample tests, structured interviews, in-depth reference checks, etc.

New evidence of the power of GMA

One of the biggest areas of focus for personnel psychologists is uncovering which selection mechanisms do the best job of predicting job performance.

Different researchers have focused on various tests, but perhaps no tests have received as much attention as those that measure general mental ability (GMA). GMA has consistently been shown to produce the highest criterion-related validity (CRV) values and has some very strong proponents. (For those of you not up on your statistics, CRV refers to the statistical relationship between test scores and subsequent job or training performance; with a maximum value of 1.0, the bigger, the better)

One of the most strident advocates of ability testing is Frank Schmidt, who has studied and written extensively on the topic. You may have heard of the widely cited article he co-authored with John Hunter in 1998. In that article, they present a CRV value of .51 for cognitive ability tests, which is considered excellent. Only work samples received a higher score, but this value has been subsequently questioned.

In the latest issue of Personnel Psychology (v61, #4), Schmidt and his colleagues present an updated CRV value, and it's even higher. Using what they claim is a more accurate way of correcting for range restriction, the authors present an overall value of .734 for job performance and .760 for training performance. This value is the highest I've seen reported in a major study such as this and further solidifies GMA as "the construct to beat" when predicting performance.

The article also uses this same updated statistical approach to looking at the CRV of two personality variables that have been generally supported--Conscientiousness (Con) and Emotional Stability (ES). The values presented for these unfortunately were not that much larger than previously reported: for Con: .332 (.367) and for ES: -.100 (-.106) for job (training) performance.

That all being said, there are some things to note:

1) Use of GMA tests for selection are likely to produce substantial adverse impact with most applicant samples of any substantial size, potentially limiting their usage in many cases.

2) CRV coefficients are just one "type" of validity evidence. The calculation is far from perfect and depends greatly on the criterion being used. The authors admit that they were unable to measure the prediction of contextual performance, which could have resulted in substantially higher values for the personality variables.

3) On a related note, some of the largest CRV values for personality tests I've seen were reported in Hogan & Holland (2003), where they aligned predictor and criterion constructs. This study was excluded from the current study because "the performance criteria they employed were specific dimensions of job performance rather than overall job performance."

4) The lower values reported in this study for personality measures may also reflect the way personality is measured, which the authors acknowledge. They suggest using outside raters as well as multiple scales for the same constructs may yield higher CRV values. Interestingly, they also suggest that personality may not be as important because with sufficient GMA, individuals can make up for any weaknesses--such as forcing yourself to frequently speak with others even if you're an introvert.

5) CRV values for GMA continued to vary substantially depending on the complexity of the job, yielding values that ranged .20-.30 apart from one another. This is a key point and is related to the fact that the type of job--and job performance--matters when generating these numbers.

Last but not least, there's another great article in this issue, devoted to (coincidentally) conducting CRV studies by Van Iddekinge and Ployhart--check it out. They go into detail about many issues directly relevant to the study above.

Giving thanks for research

It's almost Thanksgiving here in the U.S., a time to give thanks, and I'd like to thank a largely unsung group of people. Thank you to all the researchers out there who try to help us put some science around the art we call personnel recruitment and selection. Thank you for all your work and insights.

What better way to celebrate this wish of thanks than by talking about a new issue of the International Journal of Selection and Assessment (v16, #4)! As usual it's chalk full of good articles, so let's take a look at some of them.

First, a study of applicant perceptions of credit checks, something many of us do for sensitive positions. Using samples of undergraduates, Kuhn and Nielsen found mostly negative reactions, especially for older participants, but they varied with the explanation given as well as privacy expectations. Worth a look for any of you that conduct large numbers of background checks (and if you do, don't miss the Oppler et al. study below).

Next up, a fascinating study of police officer selection in the Netherlands. Using data from over 3,000 applicants, De Meijer et al. found evidence for differential validity between ethnic majority and minority participants. Specifically, cognitive ability tests predicted training performance for minorities but not for those in the majority. Performance prediction for the latter group was low for cognitive ability tests and somewhat better using non-cognitive ability variables. By the way, the dissertation of the primary author, a fascinating look at similar issues, can be found here.

The third article is one of those articles that almost (...almost) makes me want to pay for it, and anybody interested in electronic applicant issues take note. In this study, Dunleavy et al. used simulations to show the tremendous impact that small numbers of applicants can have on adverse impact (AI) analysis. In fact, the authors reveal situations where AI can be caused or masked by a single applicant applying multiple times! The authors present ways of identifying and handling these cases. Scary stuff. Hope the OFCCP is reading.

Fourth, Lievens and Peeters present results of a study of elaboration and its impact on faking situational judgment tests. Using master students, the researchers found that requiring elaboration on items (i.e., the reason they chose the response) had several positive results. It reduced faking on items with high familiarity. It also reduced the percentage of "fakers" in the top of the distribution. Lastly, candidates reported that the elaboration allowed them to better demonstrate their KSAs. This could be a great strategy for those of you worried about the inflation effects of administering SJTs online.

Next, Furnham et al. with a study of assessment center ratings. The authors found that expert ratings of "personal assertiveness", "toughness and determination", and "curiosity" were significantly correlated with participant personality scores, particularly Extraversion. Correlations with intelligence test scores were low.

Last but definitely not least, Oppler et al. discuss results of a rare empirical study of financial history and its relationship to counterproductive work behaviors (CWBs). Using a "random sample of 2519 employees" the authors found that those with financial history "concerns" were significantly more likely to demonstrate CWBs after hire. Great support for conducting these types of checks.

There are other articles in here, so I encourage you to check them all out. Thank goodness for research!

Does vendor size matter?

Yesterday I attended a demonstration by a smallish firm whose product automates the job application, exam administration, and applicant tracking process. We talked about a lot of different things, including how easy it would be for end users to understand.

After the vendor left, I had a discussion with another of the attendees about the meeting. But our conversation wasn't so much about the product as it was about the company. We talked about the small size of the firm and how much of an issue that is when selecting the vendor.

On the one hand, it would seem small firms are more susceptible to succession planning issues. They're typically run by a charismatic, passionate, and extremely talented individual whose energy continually sustains the business. What happens when they're gone? They may also not have the built in redundancies that larger firms have, as well as the capacity to handle larger projects.

On the other hand, in my experience it's the quality of the product and the support that matters most for an IT implementation, not sheer size. Does the vendor "get" the customer? Do they have experience with the relevant issues? Are they honest about the product's capabilities and time frames? These are the factors I've found to be most important.

What do YOU think? Is vendor size relevant? Is it a make or break issue? I've temporarily turned off the registration requirement for the blog, so anyone can comment. I'm interested in hearing from users as well as vendors.

New blog--and newsletter

Over at U.S. Research Associates, my friend and colleague Dr. Jim Higgins has started a blog, and a newsletter, devoted to topics such as personnel selection and employment discrimination. They're both worth checking out.

Jim starts his blog with a post about legally defensible competency modeling for selection. He offers some great words of advice to those of you thinking about (or already) using competencies for selection. For example: what validation strategy is appropriate? (hint: it's not content validation)

The first issue of the newsletter (November 2008) starts with the same article but includes several others you might be interested in. Topics include:

- Why employers lose legal challenges to their testing process

- Adverse impact analysis using binary logistic regression

- A review of several statistical analysis software packages

- An overview of criterion-related validation

- Multiple regression and the OFCCP

Good stuff! Jim also offers several free on-line training classes on topics such as criterion validation, adverse impact, and Microsoft Excel.

SkillSurvey adds passive candidate search

About a year ago I posted about two new websites that automate the reference checking process--Checkster and SkillSurvey. Turns out it was one of the most popular posts I've ever done, so there must be an interest out there in making this headache-inducing process easier. I'm still a big fan of both services, and had the pleasure of meeting Yves Lermusi, CEO of Checkster, at this year's IPMAAC conference.

Anyway, I got an email the other day from SkillSurvey about some cool new functionality they've implemented. Back when I wrote the original post, it occurred to me that one eventual outcomes of having all this reference check data is the database that's being generated. Imagine the sourcing possibilities if each reference gives contact information on 5 other individuals in their field!

Well, SkillSurvey apparently had the same thought and this new function (with the somewhat cumbersome name of Passive Candidate Compiler) allows users to search their own database. A lot of recruiters already maintain a database of reference-givers, but this automates the whole process, saving tons of time.

Pretty slick if you ask me. Next step? How about access to the entire database?

The red clothing effect

A little fun for our Friday...

I'm not a big fan of interviews. Particularly ones that are unstructured (e.g., different questions for different candidates, no rating scales, etc.).

Why? Aside from the fact that research has shown them to be much less predictive of job success, they're problematic because most people think they are above average interviewers (they're obviously not), that they're particularly good at picking up things like deception and lies (once again...they're not), and that interviews are done easily and quickly (they shouldn't be).

Another reason interviews are tricky is they're susceptible to all kinds of perceptual errors. Some of the more common ones include:

- The "Halo" effect: something about the candidate biases the way you see other things about them. My favorite example is having positive feelings toward someone because they went to your alma mater.

- The contrast effect: your opinion of a candidate is biased because they followed a particularly good or bad candidate.

- The fatigue effect: the way you evaluate candidates changes over the course of a day or week because you get tired of interviewing.

This is just a sample of cognitive biases that enter into the interview process. Other typically non-job related factors come also into play, such as someone's height.

Now we may have to add the red clothing effect. A recent study of undergraduates at the University of Rochester found that the color red, compared to other colors, led heterosexual men to find women more attractive (it had no effect on female participants' perception of other females).

The researchers validated this (albeit with small samples) using a variety of experiments, including digitally altering the shirt color of the same image. They even looked at other factors such as willingness to ask out on a date (hint for straight women: chose red over blue).

The silver lining for us is that color had no impact on perceptions of likability, kindness, or intelligence. Still, it's something to be aware of that could potentially have important consequences. After all, remember what happened to Neo.

Webinar on using blogs for recruitment

We all know blogs can be a great way to share information. But what about using them for recruiting purposes?

I wrote about this a while back--how blogs can be used as information "push" and "pull" mechanisms or even as a retention tool--but if you're interested in learning more, check out this webinar on Thursday November 6th put on by Bernard Hodes and the American Hospital Association.

Speaking of webinars, in this time of budgetary challenge, make sure to check out all the free webinars offered by places like HR.com and HCI. Could be a good time to focus on internal competencies. Just be wary of giving out your email address too freely--unless you like lots of email.

Upcoming webinar on defending your tests

Tests aren't valid or invalid per se--it depends on what you use them for.

But if your tests are challenged legally (say, because they have a discriminatory impact against a protected group), one of the things you'll want to defend yourself with is a test validation report--a documentation of why the test was developed, how it was developed, and the purposes for which the test should be used.

This is one of the topics that will be covered in an upcoming webinar sponsored by Talent Management and presented by some well known folks over at APT. Taking place on November 11th at 11am PST, the webinar is titled "Testing the Test: What You Need to Know about Test Validation, Litigation, and Risk Management."

Should be worth a watch/listen.

On a related note, the latest issue of Talent Management had some good articles in it, including ones on "role based" assessment (which just sounds like good 'ol fashioned position-based assessment) and employee surveys. It's actually not a bad little magazine, and it's free. You can subscribe here.

Leadership podcast

Seems to be the season to discuss leadership. Wonder why?

A recent guest on the Doug "lawyer to peacemaker" Noll show was Dr. Robert Hogan, renowned expert on leadership and personality and President of Hogan Assessment Systems. Dr. Hogan discussed several things, including:

- how evolution relates to leadership and followership

- why the current trend of focusing on "strengths" may be misguided

- the base rate of bad leaders (hint: it's not 5%)

- what followers want in a leader (hint: it's not a big ego)

Good stuff. Reminds me that I need to join 2002 and start doing more audio/video.

Discrimination, assessment centers, and handshakes

The title sounds like a strange combination, no? That's because it refers to three separate pieces of research published in the September 2008 Journal of Applied Psychology.

First, Umphress et al. describe a study that demonstrates how important leaders can be in setting the tone for selection. Specifically, the authors found that when authority figures focused team selection decisions on job performance factors, individuals that had a tendency to discriminate based on social dominance orientation were less likely to do so. Implication? To help avoid discriminatory hiring and promotion decisions, focus decision makers on job-related performance factors. That way, they're less likely to rely on their own biases.

Next, Meriac et al. with a meta-analysis of the incremental validity of assessment center (AC) ratings over other assessment tools. Specifically, the authors found that AC ratings explained a "sizable proportion of variance in job performance" beyond cognitive ability and personality tests. Good news for fans of assessment centers out there.

Last but not least, Stewart, et al. describe the results of a study of 98 undergraduate students that participated in mock interviews. In the words of the authors, "quality of handshake was related to hiring recommendations." How exactly does that work? Apparently how you shake hands sends messages about your degree of extraversion (above and beyond your appearance). The authors also found that the effect seemed to be stronger for women than men. Implication? For those of you interviewing for sales jobs, pay attention to your handshake!

Honorable mentions:

- Judge & Livingston on how traditional gender role orientation impacts the wage gap

- Cascio & Aguinis on trends in I/O psychology from 1963 - 2007 (good stuff; read here)

- Chiaburu & Harrison's meta-analysis on how co-workers impact job performance

- Levi & Fried on differences between African Americans and Whites on attitudes toward affirmative action programs

FAA uses games to hire and train

Turns out all I have to do is post about how we should be using video games for recruitment and assessment, and an example appears!

In this recent article in the New York Times (should be first link), the author describes how the Federal Aviation Administration (FAA) is using a sophisticated simulator to train air traffic controllers.

Motivated primarily by the impending retirement wave and massive need for new controllers (1,700 a year for the next 10 years) , the FAA has developed a multi-screen simulator that allows trainees to hone their skills in a safe but semi-realistic environment. From the article:

"The tower simulation is realistic. Aircraft first appear as tiny dots against blue sky, clouds or stars. On the ground, drivers of maintenance trucks ask permission to cross a runway so they can fix a lighted sign. A click of the instructor’s mouse can shift the time of day, and change the weather — from rain to hail or cloudy to clear. To make the simulations as unpredictable as in the real world, some pilots ignore instructions."

But it's not only the training that's innovative. The FAA's screening process puts most of ours to shame. Specifically, the candidates complete a six hour (which might be overkill) computerized aptitude test that measures geometry and math ability.

This is followed by "game-like tests" designed to measure things like ability to work under pressure, maintain "situational awareness", short- and long-term memory, multitasking, and flexibility. The tests vary from air traffic simulations to ones that look like Frogger or Tetris.

So someone out there gets it! The system is even decribed as "a big Xbox."

But they could do even better. Here are some ideas how:

1. Use the multi-screen simulator for recruitment and selection, not just training. I really hope they show the system off during recruitment open houses--I know I would. And if it isn't cost prohibitive, and makes sense given entry level requirements, why not use the simulator as part of the screening process?

2. They've got a pretty good recruitment site and make good use of video. Why not add a java-powered mini-game that simulates the job? Maybe have a leader board and allow people to put in their e-mail and opt-in to getting more information about becoming an air traffic controller?

3. On a related note, why not go the route of America's Army and mix in a little SimCity and Flight Simulator and produce a more full-featured game that simulates the job? Again, players could have the option of uploading their scores to a public website, and allow them to enter their email (securely) to get more info?

Not all of our jobs lends themselves so well to simulations and video (I'm not sure SimHRManager would be very popular). But whenever possible, let's take advantage of the technology around us!

Googling applicants, RJPs, and engagement

The September 2008 Issues of Merit was just released by the U.S. Merit Systems Protection Board and has some articles worth a quick read:

- The drawbacks inherent in doing internet searches of potential applicants (e.g., finding inaccurate or misleading information, gathering information that could be potentially discriminatory)

- Realistic job previews (of which I'm a huge fan)--these can take the form of videos, classes (on-line or otherwise), or simply a better description of the job. MSPB gives the example that they provide applicants with a list of "This job might be for you if..." factors along side "This job might not be for you if..."

I've found "willingness" or "pre-screening" questionnaires to be helpful as well, which simply have candidates answer a series of questions related to the screening process (e.g, "Are you willing to have your credit reports checked?") or the job itself (e.g., "Are you willing to come in contact with toxic chemicals on a daily basis?")

- Performance management and employee engagement. MSPB studies have shown that the PM process itself is more important than the formal structure--so what you say to your employees is more critical to engaging them than whether they receive a report each year. Makes intuitive sense, but many organizations assume that because everyone receives an annual appraisal, their performance management system must be working!

What the new ADA law means for recruitment and assessment

On September 25, 2008, President Bush signed the Americans with Disabilities Act Amendments Act (ADAAA) or Senate Bill 3406 (actual text of which can be found here).

The changes the ADAAA entails have been well covered, especially over at George's Employment Blawg. The purpose of this post isn't to give a detailed analysis of the changes, but rather highlight what it means for recruiting and hiring.

First, a brief reminder of who is covered by the ADA. It covers not only those who have a substantially disability that limits a major life activity, but those who have a record of such a disability, and those who are REGARDED as having a disability (more about this in a second). Check out this information from the EEOC for a brief overview.

Given that introduction, there are two big things we all need to be aware of when it comes to the new bill:

1) Who is considered "disabled" under the ADA has just been increased dramatically. The ADAAA explicitly rejects several U.S. Supreme Court cases made in the last 10 years that narrowed who was considered disabled, and restores the broad scope. One big way they did this was by including those whose disability can be mitigated (e.g., controlling diabetes).

2) Individuals who claim injury because they were "regarded" as having a disability have a much easier time qualifying under that category.

So what does this mean? It doesn't fundamentally change anything we do when recruiting or assessing candidates. But it does mean this: recruiters and hiring managers have to be even more careful about making assumptions when sizing up candidates. Because more people are now covered by the ADA, our potential risk has expanded.

And because of the expansion of who is covered, particularly in the "regarded" category, recruiters and hiring managers should receive explicit training not to assume someone is disabled and therefore can't do the job. Their inquiry (if they must make one) should be limited to this question: "Can you perform the essential functions of the job, with or without reasonable accommodation?"

That's the only question they should ask on this topic, and they should ask it of all candidates. Of course they should also be prepared to answer this question: "Well I'm not sure. What are the essential functions?" (another great use for job analysis)

So bottom line: big change to the law, relatively small change to the way we do business. But a good reminder of our responsibilities.

p.s. other good reads regarding ADAAA include this article and this essay

Employment testing: Imperfect but still invaluable

Imagine the following scenario. A detailed study is performed on a job. Observations of incumbents are made. Discussions are held with subject matter experts. Survey data are collected. The results all indicate that this job requires a high level of intelligence, extraversion and conscientiousness, customer service skill, and a fairly advanced ability to use computers. These attributes are, by far, the most important in predicting success on the job.

Taking all this information, we construct a rigorous assessment process. Candidates spend an entire day being observed as they take well-constructed ability and personality tests, participate in several scenarios that require them to demonstrate how they handle customer service situations, and are asked to produce several products on the computer using a variety of software packages. Raters are experts in the field, and the job, and use objective, behaviorally-anchored rating scales.

The results are then combined for an overall score, based on a formula generated from the initial study of the job. Those with the highest score are hired.

Given all the test scores, what is the maximum percentage of subsequent job performance that we can expect to predict?

a) 10%
b) 25%
c) 50%
d) 75%

Those of you in the field of personnel assessment can see this coming. Those of you that aren't, which option did you select? If you chose (b), move to the head of the class.

Why only 25%? Because tests aren't perfect, and because so much more than individual competencies go into predicting job performance. In fact, 10% is a much more commonly observed statistic.

25% doesn't sound like a lot. Does this mean we shouldn't use tests when hiring? Absolutely not. Because without well-developed assessments, you can expect to predict nothing--zero percent, nada.

In an article in the September 2008 issue of Industrial and Organizational Psychology, Scott Highhouse addresses one result of this low percentage: many hiring managers and HR professionals persist on using their own intuition and judgment using things like informal interviews to supplement (or replace) tests because of a stubborn belief that this will dramatically increase the likelihood of hiring the right candidate. The problem? They're wrong.

The available research makes clear that unstructured, overly subjective techniques such as resume review and informal interviews are simply not very predictive of job performance. So why do people keep using them? Highhouse suggests a key problem is our addiction to our own judgment. Commentators on the article point out other factors, such as lack of feedback, evolution, and different criteria of interest (e.g., hiring managers often care more about getting someone quickly rather than getting the "best" candidate).

In my experience the primary reason why decision makers insist on using less valid assessment methods comes down to ego, in two ways. One, people simply have a hard time admitting that tests do a better job of predicting things than they do. Second, we have an innate need to be involved in decisions impacting our lives. Who amongst us is willing to hire someone, unseen and unheard, based solely on test results? Even the most die-hard assessment fanatics among us have a difficult time with it, even though we know it would probably be for the best!

All in all, a highly recommended and provocative article that gets to one of the biggest challenges in personnel assessment: how assessment professionals and hiring managers can work together to find the right person for the job.

The best recruitment and retention tool: That touchy-feely stuff

Everyone who's anyone in the HR circles these days knows that to have a "seat at the table", be taken seriously, and avoid criticism, we need to ditch our reputation as touchy feely "people" people and focus on ROI, adding value, walking the walk, metrics, and all those other magical things that will somehow convince executives that we're worth our salaries. Right?

Here's the funny thing. Probably the biggest single factor in attracting and keeping the right people is your reputation--what people say about your organization. Referrals are time and time again cited as the most valuable source of high quality candidates (although we desperately need more non-survey research into this issue). And whose word carries the most weight? Current employees. So where should our focus be? Yep, you guessed it, our current employees.

So what does that mean? All that touchy feely stuff like job satisfaction surveys and wellness programs help ensure that your employees are happy (or as happy as you can make them). Happy employees are not just more likely to leave your customers with a positive impression, they're more likely to sing your praises to potential applicants. This positive reputation, combined with salaries that are at least in the ballpark, and clear communication about expectations and rewards, will go a long way toward your future success in attracting talent.

Using video games to recruit and select candidates

A new study by the Pew Internet and American Life Project found that:

"virtually all American teens play computer, console, or cell phone games and...the gaming experience is rich and varied, with a significant amount of social interaction and potential for civic engagement."

and

"Game playing is universal, with almost all teens playing games and at least half playing games on a given day"

This raises a question:

Is there a benefit, or even a mandate, to make recruitment and assessment more like a video game?

We've already seen a massive amount of interest in using virtual worlds like Second Life for recruiting (which has met with mixed success). And the U.S. Army is always on the cutting edge with things like America's Army (which has enjoyed quite a bit of success).

When it comes to assessment, we've seen some valiant efforts, such as the virtual job tryout. And video-based testing has been around for a long time.

But with everything that's out there, would you describe your candidate experience as "rich and varied" with a "significant amount of social interaction"?

Laying aside for the moment the fact that many organizations lack even realistic job preview videos, what competitive advantage is to be gained by the employer that figures out how to make its recruitment and selection process interactive? What if instead of the process being a one-way street (candidates search for information about employers, employers try to figure candidates out), it was a two-way simultaneous sharing of information?

Doom came out 15 years ago. The Sims, 8 years ago. Isn't it time we developed realistic 3-dimensional worlds that allow candidates to make real-time branching decisions and learn about a potential employer, while we measure things like attention to detail and judgment?

Is it just me or are we missing an enormous opportunity to attract a new generation of workers and gather valuable competency information at the same time?

What to look for in a leader

With the upcoming U.S. Presidential election, my co-workers and I have been spending a lot of time talking politics. We're of different political parties, so things can get pretty interesting.

Debates aside, we're all assessment professionals, so we keep coming to the same conclusion: it's difficult to figure out who would be the better president without an accurate job analysis. Without knowing what it takes to be good at a job, it's almost impossible to predict who would best fill it.

This topic isn't a new one for me; I wrote previously about whether "experience matters" in a leader, and how you would actually go about hiring one. But in a way that's putting the cart before the horse. Before we look at actual ways of finding a leader, let's look at what it takes to be one.

Of course many, many people have written about leadership and the KSAs or competencies it takes to be good at it, including respected voices in the assessment world. But new data is always welcome, which is why I was pleased to see a recent analysis by Doris Kearns Goodwin, author of Team of Rivals: The Political Genius of Abraham Lincoln, and one of the most thoughtful authors about the subject today.

In her recent article she outlines 10 attributes that she believes distinguishes "truly great Presidents." Using Abraham Lincoln and Franklin Roosevelt as exemplars (identifying and agreeing on high performers is a particular challenge in this case), she listed the following. Think about how these relate to leadership in your organization:

1. The courage to stay strong--to survive in the face of adversity and motivate oneself even when frustrated.

2. Self-confidence--the ability to surround oneself with experts, regardless of whether they agree with you or not.

3. An ability to learn from errors--to acknowledge and grow from mistakes rather than continue down a road of failure.

4. A willingness to change--not just by getting elected (or hired) but by going against previous tendences and preferences when situations require it.

5. Emotional intelligence--the ability to encourage others, take blame, and help others play to their strengths.

6. Self-control--either not letting events upset you, or pausing before responding when they do.

7. A popular touch--an awareness of where the citizens (employees) are and what they are (and aren't) ready for.

8. A moral compass--doing what's right, not just what's politically expedient or popular.

9. A capacity to relax--this includes the ability to diffuse your own emotions as well as relax those around you through tools like humor.

10. A gift for inspiring others--the ability to communicate broad goals shaped by history and context, shift opinions, and motivate.

So given these qualities, here are some things to think about:

- Are these attributes generalizable to leaders in your organization?
- Are these the types of things you're looking for when recruiting/hiring leaders?
- If so, are you assessing them in the right way?
- If you're a leader, do you embody these qualities?

HR goes to jail

It's extremely unusual for human resources employees to face personal liability for violating laws relating to recruitment and selection (constitional claims being one of the rare exceptions). It's even more rare for there to be criminal penalties associated with hiring; no one's going to jail for not showing a .30 criterion-related validity coefficient or for surfing Facebook for people under 40. But when it comes to immigration violations and child labor, the law doesn't mess around.

Case in point: on Tuesday, September 9th, two HR employees at Agriprocessors, a meatpacking facility in Postille, Iowa, were named in court documents and face both misdemeanor and felony charges for violating child labor and immigration laws.

Agriprocessors has been in the news for months ever since a May immigration raid found nearly 400 workers were in the country illegally. But up until now none of the management or HR employees had been in trouble.

The HR employees face charges for helping to hire children under 18 into dangerous jobs and helping applicants obtain identification using false documents. The misdemeanor charges associated with state child labor law violations carry a penalty of 30 days in jail and a fine up to $625.

The felony charges associated with immigration law violations, on the other hand, carry a prison term of 2-22 years and a $750,000 fine for one of the HR employees, and up to five years and $250,00 for the other if they are convicted. One of the more egregious charges was helping employees complete new paperwork using new names and false documents one day before the raid.

Sort of puts those large discrimination settlements into context, doesn't it?

One man's journey into Talent Management software

Several months ago I began looking into "talent management" software. To be honest, I wasn't sure exactly what I was looking for--or at--but I learned quite a bit along the way. In this post I'll give you a glimpse into my learning experience.

I began my journey looking simply for something that would allow my organization to inventory our existing talent--for example, by allowing employees to describe their own competencies and work preferences. My interest was piqued by this article in Workforce Management, which does a very good job of describing some of the major players and their products (they had me at "baseball card-style interface").

The first thing I learned was that there isn't a clear definition of what talent management (or "TM") is in the first place. For argument's sake, let's define it as documentation, analysis, planning, and decision making regarding how competencies are brought into and used within an organization.

Now I know many of you out there are thinking, "you're JUST NOW looking at TM products?" Well, yes, and I'll blame the fact that I work in the public sector and we tend to be a bit tardy to the game. But I actually think that's a good thing in this case, because of where the product development is right now.

Turns out the "talent inventory" function is only a small part of TM product offerings and is usually included in the career management portion. But what I have come to believe is this is one of the less important features of this type of software. What may be much more important is the performance management component.

Now, I know this is a blog on recruitment and assessment, so you may raise an eyebrow when I mention performance management, but bear with me. If we think broadly about talent management (i.e., it includes everything from branding to employee exit), how we place individuals and assess their performance is not only key to organizational success, but is the ultimate indicator of how successful our recruitment and assessment methods are.

So what's out there? Quite a few offerings, actually (it's a rapidly growing field). I looked at several products with my primary criteria being usability--if supervisors aren't going to use it, it's worthless. And, it turns out many of them have similar functionality--even similar cost--so it really came down to picking something that looked attractive and useful.

So, after watching many demos, and talking to many sales reps, here in roughly descending order of usability, are the companies whose products I looked at:

Sonar 6
SuccessFactors
CornerstoneOnDemand
Authoria
Taleo
Kenexa

This list is somewhat deceiving because all of these products tend to look the same and have similar abilities--except one. For me, only one of these products stands out in terms of design, and that's Sonar6. Why? White space, white space, white space. Simple but attractive graphics and GUI. Lack of 400 stemming text menus. I encourage you to check out the demo videos on all of their websites, but I think you will agree with me that their interface is simpler, more graphical, and more engaging.

Am I going to be purchasing one of these products? Hard to say at this point. But I will certainly keep you posted. So if you're interested in these products, take your time, ask a lot of questions, and think about how this will fit with your culture. And by the way, if you're as new to this field as I was, I strongly recommend the resources over at Bersin and Associates.

Free webinar on adverse impact

My colleague Dr. Jim Higgins shares my passion for providing education on topics relating to recruitment, testing, and selection.

So I'm pleased to draw your attention to an upcoming free webinar he's offering on Understanding Adverse Impact in Testing, Selection, Promotion, and Staff Reductions.

From the registration page:

"This free webinar will help you better understand the history, present and future of adverse impact analysis and will aid in your efforts to ensure that your organization takes the steps necessary to protect itself from claims of discrimination. It will also help you ensure that your organization’s hiring and promotional practices are maximally compliant with the letter and spirit of EEO laws and regulations."

The webinar takes place on September 16th at 11am and it's only 45 minutes. If you like what you see/hear, Jim offers a free (yes, free) 9-session course on basic applied statistics.

Power v. Group Differences

In a recent post I wrote about a chart my co-workers and I created to help us communicate with hiring supervisors about the pros and cons of various testing instruments. That graph mapped power (validity) on one axis, and speed of administration on the other.

One of the comments on that post mentioned it would be nice to see power vs. group differences. I agreed. So here it is!

The bottom line on this graph (no pun intended), if you're looking for the best combination of both, will be in the upper left quadrant.

A few notes of notes of caution before interpreting the graph:

- this graph charts only Black-White differences, which is the largest data set we have. It's important to remember that combinations of other groups (including gender) will yield slightly different results.

- the evidence on group differences for T&Es is rather scant. Not much has been found, but that doesn't mean it couldn't in the future, depending on what specific training or experience is being measured.

- finally, as the excellent recent article by Roth, et al. reminds us, adverse impact in your selection process depends on several factors, including the specific test or construct, the selection ratio, your applicant pool, and the order you place your assessments in.