Does education level matter?

One of the most frequent questions related personnel assessment is does education level predict job performance?

The answer to this question has big implications for setting minimum qualifications for jobs, for screening criteria, leadership development--you name it, across the spectrum of talent management.

It seems like the answer would be fairly obvious--of course more education would result in better performance. People learn good study habits, increase their writing and oral communication skills, etc. etc.

There's just one thing: the research up until now hasn't shown this assumption to be true. For example, in Schmidt & Hunter's well known 1998 study, they found a (corrected) correlation between education level and job performance of .10--the same they found for interests and two and half times less than the predictive validity of reference checks. So not good.

But maybe it was the way they conducted the study. Or maybe education level predicts differently for different types of jobs, or maybe the way you measure performance matters.

That's a lot of maybes. So it was with much anticipation that I read Ng & Feldman's meta-analysis in the most recent issue of Personnel Psychology.

The authors found nearly 300 studies and looked at several measures of performance (both task and OCB) from a variety of sources (e.g., self-ratings, supervisor ratings, objective measures).

Results? Depends on your point of view. According to the authors, "...the results of this study suggest that using education level as a screening device has quite robust validity. In many cases, then, the higher recruitment costs and wage costs that typically accompany hiring highly educated workers are justifiable."

Reading that, you might expect the correlations they found to be quite high. They weren't. The correlation with supervisory ratings was quite similar to Schmidt & Hunter's at .09. Correlations with OCB varied, but the highest was .23. One of the worst correlations was with training program performance, at -.03

So what did education level predict? The highest values were for self-rated on-the-job substance abuse (-.28), creativity ratings (.25), "objective measures" (.24),and general absenteeism (-.22).

What about moderator variables? Nothing for job tenure, organizational tenure, or managerial nature of the job. As far as job complexity, results were mixed and effects not strong.

What's worse, on three out of the six relationships studied, the relationship between education level and job performance was more positive for either men or Caucasians compared to women and minority groups, respectively.

So bottom line? We still don't have good support that education predicts job performance. Certainly it does a much worse job compared to other assessment methods like ability testing or structured interviews. This doesn't bode well for using educational attainment for minimum qualifications nor for creating eligible lists or otherwise screening using education level. It may be that education has a stronger relationship with aspects of task performance that relate to education, such as written communication, presentation, and analytical skills. It's also possible that the type and level of the degree matters (something the authors point out but didn't analyze).

Of course it may be that education level is simply too broad of a measure to accurately predict all of the aspects that go into job performance. When it comes to attracting and screening, there are simply better ways to find the right person.

Strengths-based hiring and job posting customization

I don't usually just point to other articles, but there are two worth special mention.

First: For a while I've been meaning to do a post about the strength-based movement, typified by the StrengthsFinder tool and related books. But a recent post on The Science of Personality does a much better job than I could of discussing the pros and cons of this approach. It's actually an excerpt from an upcoming book on the subject.

And if you're interested in that subject, check out this article that provides a fairly balanced take on the subject.

My 1.5 cents: great idea for managers to focus a little more on strengths; we spend way too much time obsessing over behavioral problems. But not taking weaknesses/growth opportunities/whatever-you-want-to-call-them into account is seeing only half the equation.

Second: Jamie Madigan over at the SelectionMatters blog describes a recent article in Journal of Applied Psychology about the substantial benefits of customizing job postings; it's a great addition to an earlier study I posted about. Jamie has some great ideas on how to implement the results, and the authors provide additional suggestions in the paper, such as providing higher quality information to applicants. I look forward to further studies that investigate other strategies and use a broader subject pool (this study used undergraduates). It would also be interesting to look at using this approach for career development of existing staff. Finally, it will be interesting to see how well the job boards using a similar approach (e.g., JobFox) succeed.

Get in the game

Longtime readers know that I've considered one of the holy grails in our field to be a way of combining the interactivity and engaging content of video game technology (VGT) with recruitment and assessment. Yes, part of this is because I enjoy the occasional Nuka-Cola and killing the occasional troll, but there is so much potential in the marriage of these two fields that we can't ignore it.

Up until now, the best efforts have gone one of two ways. The first is creating an entire first person video game for recruitment purposes--this is what the U.S. Army did. The second is using VGT but in a very basic and limited way--this is what the FAA is doing. But to my knowledge no one has created a web-based tool that showcases the basic functionality of VGT while also serving as an assessment tool. In fact many people may not even know what this might look like.

Well I ran across something the other day (hat tip) that gets us pretty darn close. It's actually an onboarding program designed by Vestas, a Danish energy company. It takes the form of a situational judgment test (SJT) that leads new employees through an orientation of what Vestas does and their approach to their business.

I think once you've watched, you'll agree with me that the potential is vast.


So why this type of technology over, say, existing SJT solutions such as those offered by companies like Ergometrics and Biddle? Those definitely still have a place, and live actors are obviously higher fidelity, but here are some advantages to think about:

1. You can do more, and show more, with VGT. Need to show someone hanging onto the bottom of a helicopter then jumping to a rooftop? Not a problem, no wires required. Need to show someone underwater? Scaling a mountaineous peak? Again, much easier (and cheaper).

2. No screen actors required. No more worrying about makeup or getting the right shot--you create what you want. Of course voice talent is still very important if you decide to use sound.

3. It's just plain more modern. For folks that grew up watching cartoons and playing video games, they will naturally gravitate more toward something that feels familiar. Text job descriptions that link to an ATS? Yawn.

4. It will make you stand out. Yes, I know the unemployment rate is high here in the U.S., but don't think that means the end of competing for the most qualified. Now's the time to plan how you're going to compete when the pendulum swings the other way again.

5. It will stand the test of time. People still watch old cartoons. Very few old shows are on. That video shot of the desktop computer in the background may look outdated sooner than you'd like.

but perhaps most importantly:

6. VGT holds the promise of a truly interactive experience, where candidates explore their future work environment, make decisions, and learn about the organization. This has the potential to be both a realistic job preview that helps candidates decide whether to apply, as well as a measurement tool that gauges how well the candidate meets job requirements. (Yes this sounds a bit like Second Life but need not be so complex)


So what do we need to do moving forward? Here are some things we need to make this work:

1. More education. What do projects like this need to succeed? How much do they cost? What are the challenges and potential roadblocks?

2. Outreach to the VGT industry including the big companies (Activision Blizzard, EA, etc.) as well as the smaller shops, industry groups, schools, etc. No doubt they have much to teach us--but we have a lot to share as well. (As an aside, Activision has a very attractive Careers page that showcases some of their work, but they dump applicants right into their ATS like most companies--failed opportunity to continue the brand experience with a game-like character sheet!)

3. What are the psychometric implications? Is this just another version of unproctored Internet testing, or is there more here? How does this relate to run-of-the-mill adaptive testing? Are there demographic differences in willingness or performance?

Now what may throw a big monkey wrench into this is cost. Video games are not cheap (WOW cost $63M to develop). But we're not talking multi-user, latest video card, and all that stuff. This could be much shorter, more cartoonish, and much simpler.

I think this is the most exciting thing happening in assessment; I hope there are enough developers out there that agree.

Who are your pitchers and catchers?

One of the key pieces of advice I often hear in talent management (TM) circles is to focus on your high performers. But when it comes to selection, we often treat each position the same--each recruitment and hire should be based on job analysis, sound exam development, administration, scoring, etc. We often spend the same amount of time on non-critical roles as we do for critical roles, all in the name of defensibility and merit.

Well, a new study may make you re-think your resource allocation and decide to take that TM advice. The catch? You have to think about your group like a baseball team.

In the study, published in the January '09 issue of the Journal of Applied Psychology, the authors looked at 29 years of data from nearly 800 teams in Major League Baseball. The goal was to determine whether the experience and skill of, and resources allocated to, "core role holders" (CRHs) contributed more to team success than non-core role holders (non-CRHs).

What's a "core role"? The authors define this as one where the incumbent "encounters more of the problems facing the team, handles more of the work than other roles, and is central to the workflow of the team."

Who would this be on a baseball team? The authors chose pitchers and catchers, for several reasons--one big one is they are involved in every play. In many plays, they're the only players involved.

How did they define skill? Using rate statistics and a measure of defensive skill. The rate statistics were on base percentage for (catcher) and on base percentage against (pitcher). Choice of metric is obviously a big sticking point, but consider that the latter was substantially correlated (r=.69) with earned run average, another common way of measuring pitcher performance. The defensive skill chosen was error rate. The authors used a one-year lag so skill from one year was correlated with team performance in the next.

How about resource allocation? This was defined simply as the sum of the salaries of all players who filled the core positions from 1985-2002 (no salary data prior to '85).

Okay, enough about measures. What did they find (using hierarchical linear modeling)? Well, quite a lot of interesting things actually (particularly if you're a baseball fan), but for our purposes:

- Career experience, job-related skill, and team experience average were all positively related to team performance.

- Career experience of CRHs was more strongly correlated with team performance than career experience of non-CRHs.

- Job-related skill of CRHs was more strongly related to team performance than job-related skill of non-CRHs.

- Core resource allocation (salary of pitchers & catchers) added a small, but significant amount to team performance prediction (3%).

So what are the implications?

1) Certain positions are worth more of your time in terms of recruitment and selection (sounds intuitive but do you practice this?). When conducting job analysis, we should consider the overall strategic importance of the position, not just the tasks performed and competencies required.

2) Certain general qualifications are more important for certain positions. This has major implications for things like recruitment messages, minimum qualifications and other screening and assessment methods. It even begs the question of whether we need to revisit how we calculate criterion-related validity statistics.

3) Performance management for these positions, including top-notch onboarding, appraisal, compensation, and career development plans, are critical.

4) The selection of individuals onto teams should pay particular attention to the CRH positions.

For those of us that don't work for a professional baseball organization, what roles might be considered "core"? Obviously the organization in question matters a great deal, but considering the definition the authors use, this might include people such as receptionists, lead and senior positions, and people in QA. The situation is a little tricky since most of us don't work in groups that are directly competing against other groups, so the measurement of success might also impact the results.

One downside of the study is it didn't account for (nor does it claim to) other factors, such as the importance of leadership (in this case, the coach), personality factors, resources, reward and information systems, etc. Using their best model, the authors were able to account only for 30% of variance in team performance. I'd also be curious to see if the offensive and defensive measures differentially predicted success and whether pitchers' performance predicted team performance any better (or worse) than the catchers'.

Want to read more? Check out the in press version here.

Obama nominations illustrate lessons

The recent high profile withdraw of two of President Obama's nominees yields several lessons for employers and HR professionals about the hiring process. Here are a few:

1) Determine the real essential qualifications of the job. This includes more than just competencies; it includes things about the person that will add to their ability to get things done and work with others. For example, Daschle was widely seen as an expert on health care issues, but his tie to lobbyists and his tax problems made him an easy target.

2) Understand the environment the position is in. In Obama's case, the public was fed up with a perceived double standard that while average Americans struggled but paid their taxes, the wealthy--incredibly--failed to fulfill their tax obligations. Nominating three (if you include Geithner) individuals who exemplified this was sure to raise the ire of many Americans.

3) Don't rush the hiring process. A full assessment, which always includes reference checks, are vital. The more important the position, the longer the hiring process should be. It's not for me to judge Obama's vetting process, it's likely it was quite thorough. But for certain positions, no rocks should be left unturned.

4) Don't count your chickens. Planning the onboarding process is fine, but don't get too carried away with your plans for your new hire because they can pull out at any time. The hiring process is a two-way contract, and even in these difficult economic times, we shouldn't forget that.

5) You'll make mistakes--everyone does. Finding the right hire is just plain difficult in some cases. The more specific and rare the competencies, the harder it will be. Don't be too hard on yourself, just learn from your mistakes and try again.

6) People are watching. In Obama's case, every decision is broadcast on the 24-hour news cycle. While none of us live in that fishbowl, your current employees are always watching what you do when you make a hire (did they choose that person because they're friends? because they're attractive?). Oftenttimes your customers will be watching as well, to see what this says about your devotion to making merit-based hires. Have a sound hiring plan, and make sure it's transparent.

What's hot in I/O?

You can often get a sense of what's hot in a field by looking at conference presentations or workshops.

Recently I got the registration materials for the 2009 SIOP conference, and it includes a description of the seminars and workshops. Here's a sample:

Frontiers of personality research and practice

Making HR measurement strategic (by the authors of this book)

Reliability, ratings, and reality: Oh my!

Development in place: Leveraging the other 90% of your organization's talent

The future of HR metrics: It's a brave new world

O*NET products and tools: What's new and what's useful for your research and practice (O*NET is here)

Selection of first-line supervisors: What we know

Exploring new frontiers in test security: Approaches for protecting your testing program

Hmmm....sounds like metrics are here! As always, I'm sure the conference will be a great learning and networking opportunity. Feel like going to New Orleans in April?