Does education level matter?

One of the most frequent questions related personnel assessment is does education level predict job performance?

The answer to this question has big implications for setting minimum qualifications for jobs, for screening criteria, leadership development--you name it, across the spectrum of talent management.

It seems like the answer would be fairly obvious--of course more education would result in better performance. People learn good study habits, increase their writing and oral communication skills, etc. etc.

There's just one thing: the research up until now hasn't shown this assumption to be true. For example, in Schmidt & Hunter's well known 1998 study, they found a (corrected) correlation between education level and job performance of .10--the same they found for interests and two and half times less than the predictive validity of reference checks. So not good.

But maybe it was the way they conducted the study. Or maybe education level predicts differently for different types of jobs, or maybe the way you measure performance matters.

That's a lot of maybes. So it was with much anticipation that I read Ng & Feldman's meta-analysis in the most recent issue of Personnel Psychology.

The authors found nearly 300 studies and looked at several measures of performance (both task and OCB) from a variety of sources (e.g., self-ratings, supervisor ratings, objective measures).

Results? Depends on your point of view. According to the authors, "...the results of this study suggest that using education level as a screening device has quite robust validity. In many cases, then, the higher recruitment costs and wage costs that typically accompany hiring highly educated workers are justifiable."

Reading that, you might expect the correlations they found to be quite high. They weren't. The correlation with supervisory ratings was quite similar to Schmidt & Hunter's at .09. Correlations with OCB varied, but the highest was .23. One of the worst correlations was with training program performance, at -.03

So what did education level predict? The highest values were for self-rated on-the-job substance abuse (-.28), creativity ratings (.25), "objective measures" (.24),and general absenteeism (-.22).

What about moderator variables? Nothing for job tenure, organizational tenure, or managerial nature of the job. As far as job complexity, results were mixed and effects not strong.

What's worse, on three out of the six relationships studied, the relationship between education level and job performance was more positive for either men or Caucasians compared to women and minority groups, respectively.

So bottom line? We still don't have good support that education predicts job performance. Certainly it does a much worse job compared to other assessment methods like ability testing or structured interviews. This doesn't bode well for using educational attainment for minimum qualifications nor for creating eligible lists or otherwise screening using education level. It may be that education has a stronger relationship with aspects of task performance that relate to education, such as written communication, presentation, and analytical skills. It's also possible that the type and level of the degree matters (something the authors point out but didn't analyze).

Of course it may be that education level is simply too broad of a measure to accurately predict all of the aspects that go into job performance. When it comes to attracting and screening, there are simply better ways to find the right person.

Strengths-based hiring and job posting customization

I don't usually just point to other articles, but there are two worth special mention.

First: For a while I've been meaning to do a post about the strength-based movement, typified by the StrengthsFinder tool and related books. But a recent post on The Science of Personality does a much better job than I could of discussing the pros and cons of this approach. It's actually an excerpt from an upcoming book on the subject.

And if you're interested in that subject, check out this article that provides a fairly balanced take on the subject.

My 1.5 cents: great idea for managers to focus a little more on strengths; we spend way too much time obsessing over behavioral problems. But not taking weaknesses/growth opportunities/whatever-you-want-to-call-them into account is seeing only half the equation.

Second: Jamie Madigan over at the SelectionMatters blog describes a recent article in Journal of Applied Psychology about the substantial benefits of customizing job postings; it's a great addition to an earlier study I posted about. Jamie has some great ideas on how to implement the results, and the authors provide additional suggestions in the paper, such as providing higher quality information to applicants. I look forward to further studies that investigate other strategies and use a broader subject pool (this study used undergraduates). It would also be interesting to look at using this approach for career development of existing staff. Finally, it will be interesting to see how well the job boards using a similar approach (e.g., JobFox) succeed.

Get in the game

Longtime readers know that I've considered one of the holy grails in our field to be a way of combining the interactivity and engaging content of video game technology (VGT) with recruitment and assessment. Yes, part of this is because I enjoy the occasional Nuka-Cola and killing the occasional troll, but there is so much potential in the marriage of these two fields that we can't ignore it.

Up until now, the best efforts have gone one of two ways. The first is creating an entire first person video game for recruitment purposes--this is what the U.S. Army did. The second is using VGT but in a very basic and limited way--this is what the FAA is doing. But to my knowledge no one has created a web-based tool that showcases the basic functionality of VGT while also serving as an assessment tool. In fact many people may not even know what this might look like.

Well I ran across something the other day (hat tip) that gets us pretty darn close. It's actually an onboarding program designed by Vestas, a Danish energy company. It takes the form of a situational judgment test (SJT) that leads new employees through an orientation of what Vestas does and their approach to their business.

I think once you've watched, you'll agree with me that the potential is vast.


So why this type of technology over, say, existing SJT solutions such as those offered by companies like Ergometrics and Biddle? Those definitely still have a place, and live actors are obviously higher fidelity, but here are some advantages to think about:

1. You can do more, and show more, with VGT. Need to show someone hanging onto the bottom of a helicopter then jumping to a rooftop? Not a problem, no wires required. Need to show someone underwater? Scaling a mountaineous peak? Again, much easier (and cheaper).

2. No screen actors required. No more worrying about makeup or getting the right shot--you create what you want. Of course voice talent is still very important if you decide to use sound.

3. It's just plain more modern. For folks that grew up watching cartoons and playing video games, they will naturally gravitate more toward something that feels familiar. Text job descriptions that link to an ATS? Yawn.

4. It will make you stand out. Yes, I know the unemployment rate is high here in the U.S., but don't think that means the end of competing for the most qualified. Now's the time to plan how you're going to compete when the pendulum swings the other way again.

5. It will stand the test of time. People still watch old cartoons. Very few old shows are on. That video shot of the desktop computer in the background may look outdated sooner than you'd like.

but perhaps most importantly:

6. VGT holds the promise of a truly interactive experience, where candidates explore their future work environment, make decisions, and learn about the organization. This has the potential to be both a realistic job preview that helps candidates decide whether to apply, as well as a measurement tool that gauges how well the candidate meets job requirements. (Yes this sounds a bit like Second Life but need not be so complex)


So what do we need to do moving forward? Here are some things we need to make this work:

1. More education. What do projects like this need to succeed? How much do they cost? What are the challenges and potential roadblocks?

2. Outreach to the VGT industry including the big companies (Activision Blizzard, EA, etc.) as well as the smaller shops, industry groups, schools, etc. No doubt they have much to teach us--but we have a lot to share as well. (As an aside, Activision has a very attractive Careers page that showcases some of their work, but they dump applicants right into their ATS like most companies--failed opportunity to continue the brand experience with a game-like character sheet!)

3. What are the psychometric implications? Is this just another version of unproctored Internet testing, or is there more here? How does this relate to run-of-the-mill adaptive testing? Are there demographic differences in willingness or performance?

Now what may throw a big monkey wrench into this is cost. Video games are not cheap (WOW cost $63M to develop). But we're not talking multi-user, latest video card, and all that stuff. This could be much shorter, more cartoonish, and much simpler.

I think this is the most exciting thing happening in assessment; I hope there are enough developers out there that agree.

Who are your pitchers and catchers?

One of the key pieces of advice I often hear in talent management (TM) circles is to focus on your high performers. But when it comes to selection, we often treat each position the same--each recruitment and hire should be based on job analysis, sound exam development, administration, scoring, etc. We often spend the same amount of time on non-critical roles as we do for critical roles, all in the name of defensibility and merit.

Well, a new study may make you re-think your resource allocation and decide to take that TM advice. The catch? You have to think about your group like a baseball team.

In the study, published in the January '09 issue of the Journal of Applied Psychology, the authors looked at 29 years of data from nearly 800 teams in Major League Baseball. The goal was to determine whether the experience and skill of, and resources allocated to, "core role holders" (CRHs) contributed more to team success than non-core role holders (non-CRHs).

What's a "core role"? The authors define this as one where the incumbent "encounters more of the problems facing the team, handles more of the work than other roles, and is central to the workflow of the team."

Who would this be on a baseball team? The authors chose pitchers and catchers, for several reasons--one big one is they are involved in every play. In many plays, they're the only players involved.

How did they define skill? Using rate statistics and a measure of defensive skill. The rate statistics were on base percentage for (catcher) and on base percentage against (pitcher). Choice of metric is obviously a big sticking point, but consider that the latter was substantially correlated (r=.69) with earned run average, another common way of measuring pitcher performance. The defensive skill chosen was error rate. The authors used a one-year lag so skill from one year was correlated with team performance in the next.

How about resource allocation? This was defined simply as the sum of the salaries of all players who filled the core positions from 1985-2002 (no salary data prior to '85).

Okay, enough about measures. What did they find (using hierarchical linear modeling)? Well, quite a lot of interesting things actually (particularly if you're a baseball fan), but for our purposes:

- Career experience, job-related skill, and team experience average were all positively related to team performance.

- Career experience of CRHs was more strongly correlated with team performance than career experience of non-CRHs.

- Job-related skill of CRHs was more strongly related to team performance than job-related skill of non-CRHs.

- Core resource allocation (salary of pitchers & catchers) added a small, but significant amount to team performance prediction (3%).

So what are the implications?

1) Certain positions are worth more of your time in terms of recruitment and selection (sounds intuitive but do you practice this?). When conducting job analysis, we should consider the overall strategic importance of the position, not just the tasks performed and competencies required.

2) Certain general qualifications are more important for certain positions. This has major implications for things like recruitment messages, minimum qualifications and other screening and assessment methods. It even begs the question of whether we need to revisit how we calculate criterion-related validity statistics.

3) Performance management for these positions, including top-notch onboarding, appraisal, compensation, and career development plans, are critical.

4) The selection of individuals onto teams should pay particular attention to the CRH positions.

For those of us that don't work for a professional baseball organization, what roles might be considered "core"? Obviously the organization in question matters a great deal, but considering the definition the authors use, this might include people such as receptionists, lead and senior positions, and people in QA. The situation is a little tricky since most of us don't work in groups that are directly competing against other groups, so the measurement of success might also impact the results.

One downside of the study is it didn't account for (nor does it claim to) other factors, such as the importance of leadership (in this case, the coach), personality factors, resources, reward and information systems, etc. Using their best model, the authors were able to account only for 30% of variance in team performance. I'd also be curious to see if the offensive and defensive measures differentially predicted success and whether pitchers' performance predicted team performance any better (or worse) than the catchers'.

Want to read more? Check out the in press version here.

Obama nominations illustrate lessons

The recent high profile withdraw of two of President Obama's nominees yields several lessons for employers and HR professionals about the hiring process. Here are a few:

1) Determine the real essential qualifications of the job. This includes more than just competencies; it includes things about the person that will add to their ability to get things done and work with others. For example, Daschle was widely seen as an expert on health care issues, but his tie to lobbyists and his tax problems made him an easy target.

2) Understand the environment the position is in. In Obama's case, the public was fed up with a perceived double standard that while average Americans struggled but paid their taxes, the wealthy--incredibly--failed to fulfill their tax obligations. Nominating three (if you include Geithner) individuals who exemplified this was sure to raise the ire of many Americans.

3) Don't rush the hiring process. A full assessment, which always includes reference checks, are vital. The more important the position, the longer the hiring process should be. It's not for me to judge Obama's vetting process, it's likely it was quite thorough. But for certain positions, no rocks should be left unturned.

4) Don't count your chickens. Planning the onboarding process is fine, but don't get too carried away with your plans for your new hire because they can pull out at any time. The hiring process is a two-way contract, and even in these difficult economic times, we shouldn't forget that.

5) You'll make mistakes--everyone does. Finding the right hire is just plain difficult in some cases. The more specific and rare the competencies, the harder it will be. Don't be too hard on yourself, just learn from your mistakes and try again.

6) People are watching. In Obama's case, every decision is broadcast on the 24-hour news cycle. While none of us live in that fishbowl, your current employees are always watching what you do when you make a hire (did they choose that person because they're friends? because they're attractive?). Oftenttimes your customers will be watching as well, to see what this says about your devotion to making merit-based hires. Have a sound hiring plan, and make sure it's transparent.

What's hot in I/O?

You can often get a sense of what's hot in a field by looking at conference presentations or workshops.

Recently I got the registration materials for the 2009 SIOP conference, and it includes a description of the seminars and workshops. Here's a sample:

Frontiers of personality research and practice

Making HR measurement strategic (by the authors of this book)

Reliability, ratings, and reality: Oh my!

Development in place: Leveraging the other 90% of your organization's talent

The future of HR metrics: It's a brave new world

O*NET products and tools: What's new and what's useful for your research and practice (O*NET is here)

Selection of first-line supervisors: What we know

Exploring new frontiers in test security: Approaches for protecting your testing program

Hmmm....sounds like metrics are here! As always, I'm sure the conference will be a great learning and networking opportunity. Feel like going to New Orleans in April?

We're missing half the equation

Matching the right person with the right job has two main components, one focused on the employer, the other on the applicant:

#1) A careful analysis of the job by the employer to identify the critical tasks and essential KSAs/competencies. This allows us to develop selection mechanisms that filter applicants to identify the most qualified.

#2) An accurate and thorough description of the job so the applicant can decide if there is a good match between the job and their qualifications.

We tend to do a pretty good job on #1. But many of us completely fall down when it comes to #2. We assume that the job and the organization have been described sufficiently so that applicants--qualified and not--can properly select in and out. But I think we're deluding ourselves.

Ask yourself this: What do you do to make sure applicants have a feel for the job? Let's think about this like we might a selection mechanism, from least to most fidelity:

(a) Describe the job/organization using text description of duties and qualifications
(b) Describe the job as above, also include benefits and "what's in it for you"
(c) Same as above, plus pictures of the work environment
(d) Same as above, plus other multi-media like videos and podcasts
(e) Same as above, plus Web 2.0 engagement strategies, like employee blogs
(f) Allow the applicants to perform simulated job duties online
(g) Alllow applicants to perform simulated job duties in person offsite
(h) Allow the applicants to perform actual job duties on the job site

I'll bet most of us would say (a) and (b) are standard, we have (c) on a good day, and we're moving slowly toward (d) and maybe (e). But this is, frankly, insufficient. The modern job seeker has access to a mountain of information--but desperately lacks information from the horse's (employer's) mouth. The reality is if you don't give it to them, someone else will.

What other benefits are there? Self-screening for one. The fewer unqualified people you have "in the pipe", the less time you waste screening out (not to mention the less time candidates waste, which they greatly appreciate).

By this point you might be wondering what these pictures are. They're from a children's museum I visited recently. They had all these leftover pieces of equipment for children to play with. Thing is, it was fun for the adults too. It gives you a real sense of what it would be like to sit in a helicopter, ride a police motorcycle, and drive a bus.

Why don't we have something like this geared toward adults? This goes way beyond an open house. It could be part of a career fair, or even better, employers would maintain a standalone site where applicants could get a clearer picture of what they'd be doing.

Now not everyone is going to be excited about a day in the life of a cubicle dweller, but everyone works somewhere--why aren't we doing a better job of explaining what it's like? Until we recognize the applicant's perspective, and the impact this has on getting the right person-job match, we're missing half of the equation.

Finally, for those of you that follow research, here's a listing of journals; half focus on the employer, half more on the job seeker. Which ones do you typically read?

Research round up

There has been so much good research coming out that rather than attempting to write a summary for each of them, allow me to simply present titles and links for you. I'll bet you see something you'll want to follow up on:

Cognitive and emotional processes in individuals and commercial web sites (how people respond to banner ads)

Occupational embeddedness and job performance (interesting results for people that are strongly linked to their career field)

Age stereotypes in the workplace: Common stereotypes, moderators, and future research directions (qualitative summary)

The hidden prejudice in selection: A research investigation on skin color bias

Does stereotype threat affect test performance of minorities and women? A meta-analysis of experimental evidence (really good stuff about moderators)

Stereotype threat reinterpreted as a regulatory mismatch.

Does socioeconomic status explain the relationship between admissions tests and post-secondary academic performance? (short answer: no)

Similarity and assumed similarity in personality reports of well-acquainted persons. (depends on which HEXACO factor you look at)

Not all conscientiousness scales change alike: A multimethod, multisample study of age differences in the facets of conscientiousness. (hint: there's something special about orderliness)

Enjoy!

Living with T&Es

There are a number of ways to use the Internet to perform personnel assessment. Examples include timed cognitive and job knowledge tests, biodata instruments, and personality inventories. But one of the easiest--and thus most tempting to use--types of tests is known as training and experience questionnaires, or T&Es (also sometimes called E&E for education and experience).

A typical (poor) example of a T&E item might be something like this:

How much experience do you have conducing job analysis?

a) None
b) Less than 2 years
c) 2-4 years
d) 4 or more years

I could go on at length about the challenges inherent with using this type of assessment, but I'll spare you. Instead I'll point you to Jim Higgins' December 2008 newsletter, HR Rampage, in which he addresses this topic, among others (see page 2).

Jim points out several problems with this type of assessment, including the overwhelming urge to self-inflate in high-stakes scenarios (we already have problems with outright cheating), the inability of highly qualified individuals to give themselves sufficient credit, and the work required to validate responses.

So given all these challenges, what can we do to mitigate them? The solutions investigated so far (e.g., elaboration, warnings) have met with very limited success. But there are a number of tactics we can take in this situation. Here are some other methods to consider as we wait for more research in this area:

1) Accurately describe the job and requirements to prevent an unqualified individual from applying in the first place.

2) Clearly word stems and responses to avoid legitimate mis-reads.

3) Include lie items (e.g., "I have experience using the HR Tests Job Analysis Technique") and deduct points when candidates endorse them.

4) Use false bottoms (e.g., both (a) and (b) are worth zero points) and false tops (e.g., both (c) and (d) are worth the same).

5) Use scales appropriate to the item. For example, amount of experience is often the incorrect scale; type of experience is better.

6) Encourage hiring supervisors to follow up on specific items in their interview.

7) Use a friendly zero point, such as "I do not have any experience but I would be willing to learn" instead of "none." Remember there are egos involved here.

8) Ask questions that are appropriate for a T&E. Don't ask candidates to rate their oral communication skills.

9) Don't ask people to compare themselves to others (e.g., Average, Above Average). Instead use objective measures such as number of times.

10) Seriously consider weighting the items. This is of course dependent upon subject matter expert input, but it's highly likely that your SMEs consider certain training or experience areas more important than others.

11) Before they even get to the T&E, use willingness/pre-screening questionnaires that ask candidates to acknowledge they understand the less-than-perfect conditions related to the job (e.g., mandatory overtime, working outside in the heat).

12) Consider using them as a feedback tool for candidates rather than a scored instrument (e.g., "Your responses indicate you have very little experience and education related to this job. Would you like to continue to apply?").

13) Base everything on SME input. Yes, I realize this probably doesn't need to be stated, but one of the worst temptations is for HR folk to draft T&Es themselves thinking they're easy to write. This is a myth, and helps contribute to poor quality eligible lists.

This is by no means an exhaustive list, and I'm sure there are other methods out there for helping us live with T&Es. Feel free to add your suggestions!

ATP Conference Feb 22-25

The Association of Test Publishers (ATP)'s annual conference is February 22-25 in Palm Springs, CA and if you have even a passing interest in personnel assessment, you owe it to yourself to check it out.

There's a lot more there besides I/O--they have tracks for education, clinical, and certification/licensure as well--but take a gander at some of these sessions:

Test development in a global economy: A practical guide to best practice in test construction, localization and adaptation.

Innovation and going green: Use of web-enabled technologies in applicant screening.

Developments in innovative technology-based assessments.

This is just the tip of the iceberg, there's tons of great content planned. It even looks like they know how to have a little fun. Cost is $800 for non-members ($725 for members).

Old school competencies part 2

I seem to have a thing for noticing signs in museums that remind me of a job analysis. Either I'm developing an appreciation for our nation's history or I'm obsessed with my subject matter. Maybe a bit of both.

Anyway, this weekend I visited the California State Railroad Museum and happened to notice this sign:


It reminded me of several things:

(1) Analyzing job requirements, whether you want to call them KSAs, KSAOs, KSAOPs, competencies, or whatever, is not new.

(2) Despite this, and despite efforts of the feds and well-known vendors, we still don't have a common language when we talk about job requirements. Non-HR folk tend to be more attracted to the words used in this poster--words like "people skills" and "perseverance." We need to seriously de-jargon ourselves.

(3) Many HR professionals are still stuck in the paradigm of (1) analyze job, (2) test for few relevant competencies, (3) choose person with best score. As the last paragraph of this picture points out, we need to remember that many different competencies go into job performance and very few jobs are identical. If a highly competent person drops in your lap, are you able to plug them in where you need them, or do you shunt them to a website where they spend 30 days going through your application process? Are you looking for competencies or people?

(4) I really need to stop using my phone to take pictures.

LAPD uses economy to recruit

Looking for an example of how to take advantage of tight economic times to increase recruitment efforts? Look no further than the Los Angeles Police Department (LAPD), who--accordingly to a recent article--have increased their applicant numbers over 100% compared to last year.

How did they do this? A good career site helps, and high unemployment is certainly a factor, but it's one that LAPD is tapping directly through advertising language such as this that has been posted in newspapers and on the web (bold added):

"During these times of economic uncertainty, the Los Angeles Police Department is always looking for a few good women and men to protect and to serve our communities. Never have to face a layoff again! Start your new career today!"

Subtle, huh? But apparently pretty darn effective.

IPMAAC is now IPAC

As of January 1st, 2009, the professional organization formerly known as the IPMA-HR Assessment Council (IPMAAC) has changed its name to IPAC--the International Personnel Assessment Council.

Who?

IPAC is a group of professionals devoted to the science and practice of employee assessment. That pretty much covers it. Smaller, more accessible, and more focused than SIOP. Larger than say a regional PAC or PTC.

Why the change?

IPMA-HR restructured and decided having IPMAAC a Section no longer fit with their mission. IPMA-HR targets HR heads, while IPMAAC tends to focus on practitioners.

Why join?

1) Networking. IMHO the biggest benefit. Great group of people dedicated to their field, extremely knowledgeable, and willing to lend a hand. IPAC's forum is one of the best.

2) Top notch conference. If you're looking for the latest research and best practices on employee selection, this is where it's at, and members get a discount. Check out some of last year's presentations.

3) Leadership opportunities. Looking to distinguish yourself in the field through a leadership post? IPAC's always looking for people to step up.

4) Training. Staying on top of this field can be challenging. Hooking yourself into an organization dedicated to being current is a great idea.

5) Speaking of staying up-to-date, did I mention the monthly newsletter?

Interested in joining? Head on over to this page.

Most popular posts of 2008

As we say goodbye to 2008, I thought it time to review which topics people seem to be the most interested in--on this blog at least.

So without further ado, here, in descending order, were my top 10 most popular posts of 2008:

1. Adverse impact on personality and work sample tests

2. Can the Dewey Color System be used for hiring?

3. What the new ADA law means for recruitment and assessment

4. What does your selection process say about your culture?

5. A review of situational judgment tests

6. B = f (P, E)

7. One man's journey into Talent Management software

8. Real world value of strategic HRM practices

9. Grading topgrading

10. Staffing.org's 2008 benchmark report

So what do I take from all this? Reader interest varies, but combined with the sidebar poll on the homepage, it remains obvious that personality testing is far and away the most popular topic. Hiring managers continue to be very interested in selecting candidates for more than just their ability, and HR professionals continue their quest for the best way to do this. There are very good personality tests out there, but I get the feeling no one's discovered the holy grail here yet.

What I'd like to see in 2009 is much more emphasis on candidate self-selection. Realistic job previews (descriptions and multi-media), assessments used for feedback rather than selection, and much much more. It's a very exciting direction, so let's get to it!

So Happy New Year, keep up the good work, and thanks for reading!

Oh, and please complete Dr. Charles Handler's survey. Thanks!

Holiday HR Humor

One of my favorite games at work is HRspeak. Using a few words and phrases that seem to be overly popular in some HR circles, we create sentences that sound good but are practically meaningless.

For example:

"Hey, watcha up to?"

"Not much, just transforming our operational cluster into customer-centric business practices that strategically meet our mission-critical goals using knowledge-based ROI-driven solutions."

Why this is funny (at least to a very small segment of the population) is because an even smaller segment of the population actually finds language like this to be practically useful. Typically in my experience it's HR consultants who are trying to convince HR to reinvent itself. The concept may be sound, but the words get in the way.

As usual, Scott Adams picks up on this phenomenon (and makes it much funnier than I can) in a recent Dilbert strip.

This time of year, often when people reflect and plan, let's all try to communicate a little more clearly.

And here's hoping that you and yours enjoy a joy-filled, family and friend-centric season focused around cuisine-based activities that meet your critical soul-based needs.

The future of selection

Last week I presented at the Personnel Testing Council of Northern California's monthly luncheon. The title was "Selection in a changing world: What will we be doing, and who will be doing it?"

The topic was motivated mostly by my own experiences related to what's going on in our field lately. With many of us facing severe budget challenges, I got to thinking about things like:

- What do we need to be doing, what should we be doing, and how can we add value in lean times?

- What impact does automation (e.g., applicant tracking systems) have on the work we do?

- What impact does automation have on the competencies we need?

The field of HR has been talking about transformation for years now, with IMHO only partial success. It's time for assessment professionals to take a look at ourselves and determine if we're where we need to be.

You can see the slides here:



They should also be posted soon on PTC-NC's website.

Resume <> Personality

I don't know about you, but one of my least favorite forms of assessment is pouring through resumes. They're not standardized, they leave out important details, and often provide way too many details about things we don't care about. But most importantly, it just doesn't feel like a very valid way of making inferences about candidates.

There are good reasons to dislike this activity. Not only are there rampant self-inflation problems, the inferences recruiters tend to make about applicant personality are erroneous, according to a recent study. After looking at responses from 244 recruiters, the authors found several important results:

1) Low interrater reliability -- in other words, the recruiters didn't agree with each other very often about what the resume said about an applicant's personality.

2) When correlations were made between recruiters' inferences of personality with actual Big 5 scores from the applicants, low levels of validity were found (slightly better for conscientiousness and openness to experience).

3) Despite the two findings above, rater perception of extraversion, oppenness to experience, and conscientiousness predicted their assessment of employability of the applicants.

Lesson? Be very careful what you imply from a resume. Think carefully about the facts you're using to infer personality. If you must use them, screen out only those who lack the basic qualifications to do the job. Follow up the resume screen with a number of much more valid assessments--work sample tests, structured interviews, in-depth reference checks, etc.

New evidence of the power of GMA

One of the biggest areas of focus for personnel psychologists is uncovering which selection mechanisms do the best job of predicting job performance.

Different researchers have focused on various tests, but perhaps no tests have received as much attention as those that measure general mental ability (GMA). GMA has consistently been shown to produce the highest criterion-related validity (CRV) values and has some very strong proponents. (For those of you not up on your statistics, CRV refers to the statistical relationship between test scores and subsequent job or training performance; with a maximum value of 1.0, the bigger, the better)

One of the most strident advocates of ability testing is Frank Schmidt, who has studied and written extensively on the topic. You may have heard of the widely cited article he co-authored with John Hunter in 1998. In that article, they present a CRV value of .51 for cognitive ability tests, which is considered excellent. Only work samples received a higher score, but this value has been subsequently questioned.

In the latest issue of Personnel Psychology (v61, #4), Schmidt and his colleagues present an updated CRV value, and it's even higher. Using what they claim is a more accurate way of correcting for range restriction, the authors present an overall value of .734 for job performance and .760 for training performance. This value is the highest I've seen reported in a major study such as this and further solidifies GMA as "the construct to beat" when predicting performance.

The article also uses this same updated statistical approach to looking at the CRV of two personality variables that have been generally supported--Conscientiousness (Con) and Emotional Stability (ES). The values presented for these unfortunately were not that much larger than previously reported: for Con: .332 (.367) and for ES: -.100 (-.106) for job (training) performance.

That all being said, there are some things to note:

1) Use of GMA tests for selection are likely to produce substantial adverse impact with most applicant samples of any substantial size, potentially limiting their usage in many cases.

2) CRV coefficients are just one "type" of validity evidence. The calculation is far from perfect and depends greatly on the criterion being used. The authors admit that they were unable to measure the prediction of contextual performance, which could have resulted in substantially higher values for the personality variables.

3) On a related note, some of the largest CRV values for personality tests I've seen were reported in Hogan & Holland (2003), where they aligned predictor and criterion constructs. This study was excluded from the current study because "the performance criteria they employed were specific dimensions of job performance rather than overall job performance."

4) The lower values reported in this study for personality measures may also reflect the way personality is measured, which the authors acknowledge. They suggest using outside raters as well as multiple scales for the same constructs may yield higher CRV values. Interestingly, they also suggest that personality may not be as important because with sufficient GMA, individuals can make up for any weaknesses--such as forcing yourself to frequently speak with others even if you're an introvert.

5) CRV values for GMA continued to vary substantially depending on the complexity of the job, yielding values that ranged .20-.30 apart from one another. This is a key point and is related to the fact that the type of job--and job performance--matters when generating these numbers.

Last but not least, there's another great article in this issue, devoted to (coincidentally) conducting CRV studies by Van Iddekinge and Ployhart--check it out. They go into detail about many issues directly relevant to the study above.

Giving thanks for research

It's almost Thanksgiving here in the U.S., a time to give thanks, and I'd like to thank a largely unsung group of people. Thank you to all the researchers out there who try to help us put some science around the art we call personnel recruitment and selection. Thank you for all your work and insights.

What better way to celebrate this wish of thanks than by talking about a new issue of the International Journal of Selection and Assessment (v16, #4)! As usual it's chalk full of good articles, so let's take a look at some of them.

First, a study of applicant perceptions of credit checks, something many of us do for sensitive positions. Using samples of undergraduates, Kuhn and Nielsen found mostly negative reactions, especially for older participants, but they varied with the explanation given as well as privacy expectations. Worth a look for any of you that conduct large numbers of background checks (and if you do, don't miss the Oppler et al. study below).

Next up, a fascinating study of police officer selection in the Netherlands. Using data from over 3,000 applicants, De Meijer et al. found evidence for differential validity between ethnic majority and minority participants. Specifically, cognitive ability tests predicted training performance for minorities but not for those in the majority. Performance prediction for the latter group was low for cognitive ability tests and somewhat better using non-cognitive ability variables. By the way, the dissertation of the primary author, a fascinating look at similar issues, can be found here.

The third article is one of those articles that almost (...almost) makes me want to pay for it, and anybody interested in electronic applicant issues take note. In this study, Dunleavy et al. used simulations to show the tremendous impact that small numbers of applicants can have on adverse impact (AI) analysis. In fact, the authors reveal situations where AI can be caused or masked by a single applicant applying multiple times! The authors present ways of identifying and handling these cases. Scary stuff. Hope the OFCCP is reading.

Fourth, Lievens and Peeters present results of a study of elaboration and its impact on faking situational judgment tests. Using master students, the researchers found that requiring elaboration on items (i.e., the reason they chose the response) had several positive results. It reduced faking on items with high familiarity. It also reduced the percentage of "fakers" in the top of the distribution. Lastly, candidates reported that the elaboration allowed them to better demonstrate their KSAs. This could be a great strategy for those of you worried about the inflation effects of administering SJTs online.

Next, Furnham et al. with a study of assessment center ratings. The authors found that expert ratings of "personal assertiveness", "toughness and determination", and "curiosity" were significantly correlated with participant personality scores, particularly Extraversion. Correlations with intelligence test scores were low.

Last but definitely not least, Oppler et al. discuss results of a rare empirical study of financial history and its relationship to counterproductive work behaviors (CWBs). Using a "random sample of 2519 employees" the authors found that those with financial history "concerns" were significantly more likely to demonstrate CWBs after hire. Great support for conducting these types of checks.

There are other articles in here, so I encourage you to check them all out. Thank goodness for research!

Does vendor size matter?

Yesterday I attended a demonstration by a smallish firm whose product automates the job application, exam administration, and applicant tracking process. We talked about a lot of different things, including how easy it would be for end users to understand.

After the vendor left, I had a discussion with another of the attendees about the meeting. But our conversation wasn't so much about the product as it was about the company. We talked about the small size of the firm and how much of an issue that is when selecting the vendor.

On the one hand, it would seem small firms are more susceptible to succession planning issues. They're typically run by a charismatic, passionate, and extremely talented individual whose energy continually sustains the business. What happens when they're gone? They may also not have the built in redundancies that larger firms have, as well as the capacity to handle larger projects.

On the other hand, in my experience it's the quality of the product and the support that matters most for an IT implementation, not sheer size. Does the vendor "get" the customer? Do they have experience with the relevant issues? Are they honest about the product's capabilities and time frames? These are the factors I've found to be most important.

What do YOU think? Is vendor size relevant? Is it a make or break issue? I've temporarily turned off the registration requirement for the blog, so anyone can comment. I'm interested in hearing from users as well as vendors.