Don't ask candidates to judge themselves

Imagine you're buying a car. The salesperson throws out a price on the car you're interested in. And here are the questions you ask to try to determine whether it's a good deal:

- Is this a good price?
- How good of a salesperson are you?
- Compared to other sales you've made, how good is this one?

Think this is silly? Well it's essentially what many employers are doing when they interview people or otherwise rely on descriptions of experience when screening. They rely way too heavily on self-descriptions when they should be taking a more rigorous approach. Think these types of questions, which should be stricken from your inventory:

"What's something you're particularly good at?"

"How would you describe your skills compared to other people?"

There are two main problems with asking these types of questions in a high-stakes situation like a job interview (or buying a car):

1) People are motivated to inflate their answers, or just plain lie, in these situations. You know that. I know that. But it's surprising how many people forget it.

2) People are bad at accurately describing themselves. We know this from years of research, but if you're interested, check out a recent study published in the Journal of Personality and Social Psychology that compared Big 5 personality ratings from four countries and found that people generally hold more favorable opinions of themselves compared to how others see them.

But it's even worse than it appears. It's not just that people inflate themselves, it's that some of your best candidates deflate themselves. Think about your star performers: if you asked them how good they were in a particular area, what do you think they'd say?


You essentially want to know two things about candidates:
1) What they've done
2) What they're capable of doing

To answer the first issue, you have several options, including:

1a) Asking them to describe what they've done--the so-called "behavioral interviewing" technique. Research shows that these types of questions generally contribute a significant amount of validity to the process. But they're not perfect by any means, particularly with people with bad memories about themselves. And keep in mind at that point you're taking their word for it.

1b) Asking them for examples of what they've done. Best used as a follow-up to a claim, but tricky in any situation where there's even a remote possibility that someone else did it or did most of it (so practically everything outside of the person being videotaped).

1c) Asking others (e.g., co-workers, supervisors) what the candidate's done. Probably the most promising but most difficult data to accurately capture. Hypothetically if the person has any job history at all they've left a trail of accomplishments and failures, as well as a reliable pattern of responding to situations. This is the promise of reference checks that so often is either squandered ("I don't have time") or stymied ("They just gave me name, title, and employment dates"). Don't use these excuses, investigate.

As for the second issue, you have several options as well, including:

2a) Asking knowledge-based questions in an interview. For whatever reason these seem to have fallen out of favor, but if there is a body of knowledge that is critical to know prior to employment, ask about it. At the worst you'll weed out those who have absolutely no idea.

2b) Using another type of assessment, such as a performance test/work sample, on-site writing exercise, role play, simulation, or written multiple choice test (to name a few). Properly developed and administered, these will give you a great sense of what people are capable of--just make sure the tests are tied back to true job requirements.

2c) Using the initial period of employment (some call it probation) to throw things at the person and see what they're capable of. It's important not test their ability to deal with overload (unless that's critical to the job), but get them involved in a diverse set of projects. Ask for their input. Ask them to do some research. See what they are capable of delivering, even if it's a little rough.


Whatever you do, triangulate on candidate knowledge, skills, and abilities. Use multiple measures to get an accurate picture of what they bring. Consider using an interview for a two-way job preview as much as an assessment device.

But above all, don't take one person's word for things. Unless you like being sold a lemon.

September 2010 IJSA (those considering SHRM certification, read on)

The September issue of the International Journal of Selection and Assessment (IJSA) is out with a boatload of content. Let's check out some of the highlights:

First up, a piece by Gentry, et al. that has implications for self-rating instruments. The authors studied self-observer ratings among managers in Southern Asia and Confucian Asia and found an important difference: the discrepancy between the ratings was greater in Southern Asia. Specifically, the difference appears in self-ratings rather than observer ratings, indicating differences in how managers in the different areas perceived themselves. Implication? Differences in self ratings may be due to cultural differences in addition to things like personality and instrument type.

The second article is a fascinating one by Saul Fine in which the author analyzed differences in integrity test scores across 27 countries. Fine found two important things: first, there are significant differences in test scores across countries. Second, test results were significantly correlated (r= -.48) with country-level measures of corruption as well as several aspects of Hofstede's cultural dimensions.

Next, an article by De Corte, et al. that describes a method for creating Pareto-optimal selection systems that balance validity, adverse impact, and predictor constraints. This article continues the quest for balancing utility and subgroup differences. A link to the article is here but it wasn't functional at the time I wrote this; hopefully it will be soon.

Next, in an article that SHRM will probably place on their homepage if they haven't already, Lester et al. studied alumni from three U.S. universities to analyze the relationship between attainment of the Professional in Human Resources (PHR) certification offered by SHRM and early career success. Results? Those with a PHR were significantly more likely to obtain a job in HR (versus another field) BUT possession was not associated with starting salary or early career promotions. I'll let you decide if you think it's worth the time (and expense).

If you need another reason to focus on work samples and structured interviews, here ya go. Anderson, et al. provide us with the results of a meta-analysis of applicant reactions to selection instruments. Drawing from data from 17 countries, the authors found results similar to what we've seen in the past: work samples and interviews were most preferred, while honesty testing, personal contacts, and graphology were the least preferred. In the middle (favorably evaluated) were resumes, cognitive tests, references, biodata, and personality inventories.

Fans of biodata and personality testing may find the article by Sisco & Reilly reassuring. Using results from over 700 participants, the authors found that the factor structures of a personality inventory and biodata measure were not significantly impacted by social desirability at the item level. Implication? The measures seemed to hold together and retain at least an aspect of their construct validity even in the face of items that beg inflation.

Speaking of personality tests, Whetzel et al. investigated the linearity of the relationship between the OPQ and job performance. Results? Very little departure from linearity and where present the departure was small. This suggests that utility gains may be obtained across the spectrum of personality test results.

Are you overloading your assessment center raters? Melchers et al. present the results of a study that strongly suggests that if you are using group discussions as an assessment tool, you need to be sensitive to the number of participants that raters are simultaneously observing.

There are other articles in here you may be interested in, including ones on organizational attractiveness, range shrinkage in cognitive ability test scores, and staffing services related to innovation.

July 2009 J.A.P.: SJTs and more

Situational judgment tests (SJTs) have a long tradition of successfully being used in employment tests. These types of (typically multiple-choice) items describe a job-related scenario then ask the test-taker to endorse the proper response. The question itself usually takes one of two forms:

1) What SHOULD be done in this situation? ("knowledge instruction")

2) What WOULD you do in this situation? ("behavioral tendency instruction")

What are the practical differences between the two? Previous meta-analytic research, specifically McDaniel et al.'s 2007 study, revealed that knowledge instruction items tend to be more highly correlated with cognitive ability, while behavioral tendency items show higher correlations with personality constructs. In terms of criterion-related validity, there appeared to be no significant difference between the two.

But there were limitations to that study, and two of them are addressed in a study found in the July 2009 issue of the Journal of Applied Psychology. Specifically, Lievens et al. addressed the inconsistency in stem content by keeping it the same while altering the response instruction, and also looked at a large population of applicants, rather than incumbents, which tended to dominate McDaniel et al.'s 2007 sample.

Results? Consistent with the 2007 study, knowledge instructions were again more highly correlated with cognitive ability, and there was no meaningful difference in criterion-related validity (the criterion being grades in interpersonally-oriented courses in medical school). Contrary to some research in low-stakes settings, there were no mean score difference between the two response instructions.

Practical implications? The authors suggest knowledge instruction items may be superior due to their resistance to faking. My only concern is that these items are likely to result in adverse impact in many applied settings. Like all assessment situations, the decision will involve a variety of factors, including the KSAs required on the job, the size and nature of the applicant pool, the legal environment, etc. But at least this type of research supports the fact that both response instructions seem to WORK. By the way, you can see an in-press version of this article here.

Other content in this journal? There's quite a bit, but here's a sample:

Content validity <> criterion-related validity

More evidence that selection procedures can impact unit as well as organizational performance

Self-ratings appear to be culturally bound

Free monograph on test validation

IPAC (the International Personnel Assessment Council) is making available, free of charge, a monograph by Dr. Charles Sproule titled Rationale and research evidence supporting the use of content validation in personnel assessment.

Having seen a copy, I can tell you it's chalk full of great content, spanning much of the field of personnel assessment, including updated information on validity coefficients and special sections for several different types of tests (e.g., interviews, training and experience exams).

It's a great primer for anyone who wants to learn more about what it means to "validate" an exam, and it's a worthy addition to the library of any seasoned professional.

The monograph can be accessed here if you are a member, or you can request a copy here if you are not. Check it out!

Exploring the wild west of staffing

Last week I gave a presentation at the WRIPAC meeting in Burlingame, CA. The topic was "Using the Internet to effectively attract and screen the right applicants." Here are a few observations given my conversations with a very engaging group of participants:

1. There is immense interest in using the Internet for more than just posting jobs. But many folks are unsure how to start or what's out there. For individuals new to Web 2.0 concepts (e.g., social networking sites) there is a healthy--not necessarily unwarranted--skepticism.

2. Our worlds can shift relatively rapidly from "how do we get more applicants?" to "how do we get fewer applicants?" and then back again. As recruitment and assessment professionals we need to be able to pivot quickly and make sure our organizations are up to snuff in both realms.

3. In order to use the Internet effectively, most of us need to work with our IT staff. But this can be challenging given built-in resistances to change, security concerns, and assumptions. My advice was to push them as long as you have a good business case.

4. We have a lot of work to do on the most basic use of the Internet for recruitment and self-screening: our career portals. Many are cluttered and most likely have not been reviewed critically. More white space, more links, more pictures and video.

5. Most folks have some type of applicant tracking system, but many are unsure what its capabilities are and are not 100% sure how to use it to screen applicants. Oftentimes we rush to use built-in training and experience questionnaire functionality without first understanding the best way to use them. I just hope we move away from time-in-grade.

6. People seem to be moving away from the big job boards and towards cheaper--and usually more effective--alternatives, like craigslist or more targeted postings. People are also taking a harder look at whether their advertising dollars are really bringing in the candidates they need.

7. The balance of power has shifted in some ways from the employer to the applicant. They're not just responding to our (often uninspired) job postings, they're asking their friends, looking us up on glassdoor, connecting to people through LinkedIn, etc. As partial keepers of the organization's reputation, we need to be aware of all the different ways applicants research us.

8. These are exciting times. At several points in the presentation I mentioned that right now we are in the Wild West of staffing. Technology changes constantly, and along with them so do expectations placed upon us. We learn as we go. And it's challenging, but opportunities abound for those that are willing to take some risks.

Hang on, we're in for a heck of a ride.

March '09 Journal of Applied Psychology

The March 2009 issue of J.A.P. is out with a lot of great content; for example:

To maximize diversity and validity, try recruiting on cognitive ability and selecting on conscientiousness

In a multistage assessment scenario (which exists for practically every hire), there are several options for maximizing diversity and validity

What determines the success of word of mouth recruitment strategies? Turns out, several things...
(read the draft version here and see Van Hoye's IPAC presentation here)

Interested in improving your T&Es? Check out what this study has to say about self and supervisory agreement of performance ratings.

More clues about what GMA is all about

How P-E fit is related to performance dimensions such as OCB

The importance of job offer negotiation: Subjective value predicts long-term employee outcomes

Does education level matter?

One of the most frequent questions related personnel assessment is does education level predict job performance?

The answer to this question has big implications for setting minimum qualifications for jobs, for screening criteria, leadership development--you name it, across the spectrum of talent management.

It seems like the answer would be fairly obvious--of course more education would result in better performance. People learn good study habits, increase their writing and oral communication skills, etc. etc.

There's just one thing: the research up until now hasn't shown this assumption to be true. For example, in Schmidt & Hunter's well known 1998 study, they found a (corrected) correlation between education level and job performance of .10--the same they found for interests and two and half times less than the predictive validity of reference checks. So not good.

But maybe it was the way they conducted the study. Or maybe education level predicts differently for different types of jobs, or maybe the way you measure performance matters.

That's a lot of maybes. So it was with much anticipation that I read Ng & Feldman's meta-analysis in the most recent issue of Personnel Psychology.

The authors found nearly 300 studies and looked at several measures of performance (both task and OCB) from a variety of sources (e.g., self-ratings, supervisor ratings, objective measures).

Results? Depends on your point of view. According to the authors, "...the results of this study suggest that using education level as a screening device has quite robust validity. In many cases, then, the higher recruitment costs and wage costs that typically accompany hiring highly educated workers are justifiable."

Reading that, you might expect the correlations they found to be quite high. They weren't. The correlation with supervisory ratings was quite similar to Schmidt & Hunter's at .09. Correlations with OCB varied, but the highest was .23. One of the worst correlations was with training program performance, at -.03

So what did education level predict? The highest values were for self-rated on-the-job substance abuse (-.28), creativity ratings (.25), "objective measures" (.24),and general absenteeism (-.22).

What about moderator variables? Nothing for job tenure, organizational tenure, or managerial nature of the job. As far as job complexity, results were mixed and effects not strong.

What's worse, on three out of the six relationships studied, the relationship between education level and job performance was more positive for either men or Caucasians compared to women and minority groups, respectively.

So bottom line? We still don't have good support that education predicts job performance. Certainly it does a much worse job compared to other assessment methods like ability testing or structured interviews. This doesn't bode well for using educational attainment for minimum qualifications nor for creating eligible lists or otherwise screening using education level. It may be that education has a stronger relationship with aspects of task performance that relate to education, such as written communication, presentation, and analytical skills. It's also possible that the type and level of the degree matters (something the authors point out but didn't analyze).

Of course it may be that education level is simply too broad of a measure to accurately predict all of the aspects that go into job performance. When it comes to attracting and screening, there are simply better ways to find the right person.

Who are your pitchers and catchers?

One of the key pieces of advice I often hear in talent management (TM) circles is to focus on your high performers. But when it comes to selection, we often treat each position the same--each recruitment and hire should be based on job analysis, sound exam development, administration, scoring, etc. We often spend the same amount of time on non-critical roles as we do for critical roles, all in the name of defensibility and merit.

Well, a new study may make you re-think your resource allocation and decide to take that TM advice. The catch? You have to think about your group like a baseball team.

In the study, published in the January '09 issue of the Journal of Applied Psychology, the authors looked at 29 years of data from nearly 800 teams in Major League Baseball. The goal was to determine whether the experience and skill of, and resources allocated to, "core role holders" (CRHs) contributed more to team success than non-core role holders (non-CRHs).

What's a "core role"? The authors define this as one where the incumbent "encounters more of the problems facing the team, handles more of the work than other roles, and is central to the workflow of the team."

Who would this be on a baseball team? The authors chose pitchers and catchers, for several reasons--one big one is they are involved in every play. In many plays, they're the only players involved.

How did they define skill? Using rate statistics and a measure of defensive skill. The rate statistics were on base percentage for (catcher) and on base percentage against (pitcher). Choice of metric is obviously a big sticking point, but consider that the latter was substantially correlated (r=.69) with earned run average, another common way of measuring pitcher performance. The defensive skill chosen was error rate. The authors used a one-year lag so skill from one year was correlated with team performance in the next.

How about resource allocation? This was defined simply as the sum of the salaries of all players who filled the core positions from 1985-2002 (no salary data prior to '85).

Okay, enough about measures. What did they find (using hierarchical linear modeling)? Well, quite a lot of interesting things actually (particularly if you're a baseball fan), but for our purposes:

- Career experience, job-related skill, and team experience average were all positively related to team performance.

- Career experience of CRHs was more strongly correlated with team performance than career experience of non-CRHs.

- Job-related skill of CRHs was more strongly related to team performance than job-related skill of non-CRHs.

- Core resource allocation (salary of pitchers & catchers) added a small, but significant amount to team performance prediction (3%).

So what are the implications?

1) Certain positions are worth more of your time in terms of recruitment and selection (sounds intuitive but do you practice this?). When conducting job analysis, we should consider the overall strategic importance of the position, not just the tasks performed and competencies required.

2) Certain general qualifications are more important for certain positions. This has major implications for things like recruitment messages, minimum qualifications and other screening and assessment methods. It even begs the question of whether we need to revisit how we calculate criterion-related validity statistics.

3) Performance management for these positions, including top-notch onboarding, appraisal, compensation, and career development plans, are critical.

4) The selection of individuals onto teams should pay particular attention to the CRH positions.

For those of us that don't work for a professional baseball organization, what roles might be considered "core"? Obviously the organization in question matters a great deal, but considering the definition the authors use, this might include people such as receptionists, lead and senior positions, and people in QA. The situation is a little tricky since most of us don't work in groups that are directly competing against other groups, so the measurement of success might also impact the results.

One downside of the study is it didn't account for (nor does it claim to) other factors, such as the importance of leadership (in this case, the coach), personality factors, resources, reward and information systems, etc. Using their best model, the authors were able to account only for 30% of variance in team performance. I'd also be curious to see if the offensive and defensive measures differentially predicted success and whether pitchers' performance predicted team performance any better (or worse) than the catchers'.

Want to read more? Check out the in press version here.

Living with T&Es

There are a number of ways to use the Internet to perform personnel assessment. Examples include timed cognitive and job knowledge tests, biodata instruments, and personality inventories. But one of the easiest--and thus most tempting to use--types of tests is known as training and experience questionnaires, or T&Es (also sometimes called E&E for education and experience).

A typical (poor) example of a T&E item might be something like this:

How much experience do you have conducing job analysis?

a) None
b) Less than 2 years
c) 2-4 years
d) 4 or more years

I could go on at length about the challenges inherent with using this type of assessment, but I'll spare you. Instead I'll point you to Jim Higgins' December 2008 newsletter, HR Rampage, in which he addresses this topic, among others (see page 2).

Jim points out several problems with this type of assessment, including the overwhelming urge to self-inflate in high-stakes scenarios (we already have problems with outright cheating), the inability of highly qualified individuals to give themselves sufficient credit, and the work required to validate responses.

So given all these challenges, what can we do to mitigate them? The solutions investigated so far (e.g., elaboration, warnings) have met with very limited success. But there are a number of tactics we can take in this situation. Here are some other methods to consider as we wait for more research in this area:

1) Accurately describe the job and requirements to prevent an unqualified individual from applying in the first place.

2) Clearly word stems and responses to avoid legitimate mis-reads.

3) Include lie items (e.g., "I have experience using the HR Tests Job Analysis Technique") and deduct points when candidates endorse them.

4) Use false bottoms (e.g., both (a) and (b) are worth zero points) and false tops (e.g., both (c) and (d) are worth the same).

5) Use scales appropriate to the item. For example, amount of experience is often the incorrect scale; type of experience is better.

6) Encourage hiring supervisors to follow up on specific items in their interview.

7) Use a friendly zero point, such as "I do not have any experience but I would be willing to learn" instead of "none." Remember there are egos involved here.

8) Ask questions that are appropriate for a T&E. Don't ask candidates to rate their oral communication skills.

9) Don't ask people to compare themselves to others (e.g., Average, Above Average). Instead use objective measures such as number of times.

10) Seriously consider weighting the items. This is of course dependent upon subject matter expert input, but it's highly likely that your SMEs consider certain training or experience areas more important than others.

11) Before they even get to the T&E, use willingness/pre-screening questionnaires that ask candidates to acknowledge they understand the less-than-perfect conditions related to the job (e.g., mandatory overtime, working outside in the heat).

12) Consider using them as a feedback tool for candidates rather than a scored instrument (e.g., "Your responses indicate you have very little experience and education related to this job. Would you like to continue to apply?").

13) Base everything on SME input. Yes, I realize this probably doesn't need to be stated, but one of the worst temptations is for HR folk to draft T&Es themselves thinking they're easy to write. This is a myth, and helps contribute to poor quality eligible lists.

This is by no means an exhaustive list, and I'm sure there are other methods out there for helping us live with T&Es. Feel free to add your suggestions!

2008 PTC-NC Conference, Day 1

On March 20th and 21st I was lucky enough to attend (and present) at the 2008 PTC-NC Conference. There were several great presentations and I'm going to break the summary up into two days for ya.

The conference started off with Michael Harris, professor of International Business at the University of Missouri-St. Louis, with an update on what the courts have been saying in the area of adverse impact and employment testing. Some major points:

- Although criterion-related validity has been discussed a lot lately, Dr. Harris predicted that a content validation strategy will continue to be the most popular choice of employers.

- They're not perfect, but courts will continue to rely on the Uniform Guidelines when judging employers' efforts to validate their tests.

- Employers should be prepared to answer what alternative tests they considered before choosing the ones they did (the third prong of this type of case).

The next session was a breakout, with Chris Wright and Louis Xavier (SF State) presenting on stereotype threat while I went over the results of a demographic analysis I conducted on applicants to an on-line T&E system. Bottom line of my presentation: there were some clear demographic differences in the jobs applied for but actual instances of adverse impact (using 4/5ths rule) were rare.

Next up was another breakout, with Jim Kuthy (Biddle) presenting on AutoGOJA while Shelley Langan (CPS) presented on succession planning. I attended Shelley's, which focused on the importance of workforce planning given current demographics and provided some practical tips on how to put together a successful plan. Some key takeaways: limit succession planning to certain positions, consider inviting everyone to apply, and conduct a future-oriented job analysis as part of your planning process.

The last breakout featured an introduction to competency modeling by Nathan Ainspan (independent consultant) and a presentation on using personality testing by Shelley Langan and Howard Fortson (CPS). I attended the latter, where we had a spirited discussion of the state of personality testing and how to introduce personality testing to your selection process (hint: rhymes with job analysis). Example business measures they mentioned included the CPI, HPI, NEO, and 16-PF. They also mentioned an upcoming article by Hough & Oswald where the authors list all of the various outcomes that personality tests have been able to successfully predict.

Last but not least that day was an outstanding keynote address by James Outtz, president of Outtz and Associates and international expert on employment testing and discrimination. Dr. Outtz went over a wide range of issues related to those subjects, including the balance between validity and adverse impact (so well covered in the most recent issue of Personnel Psychology). He introduced some fascinating research that showed that while multiple choice formats showed adverse impact against African Americans and Hispanics (favoring Caucasians and Asians), the opposite was the case for multiple list (divergent) items. Perhaps most interesting was his description of a questionnaire he developed called the Job Perception Index that served as both a realistic job preview and a selection device for firefighter positions. Some great stuff from a fabulous speaker.

That's it for now--those of you that attended, feel free to comment or add! Tomorrow: Day 2.

The Acting Quotient

Think about your favorite actor or actress. Think about their performances and the different characters they've played.

Now think about what you know about that person given their performances:

- How quickly would they learn?
- How good of a team player would they make?
- What are their customer service skills like?
- Would they show up to work punctually?

Do you have any idea what the answers to these questions are? Probably not. That's because they're good at what they do--modifying their behavior to fit different roles.

Why do I bring this up? Because lately I've been thinking about how much stock people place in the most common screening mechanisms--applications, resumes, and interviews. These selection mechanisms all allow enormous opportunities for candidates to alter their behavior to fit what they think you want to hear, or to downright fib. Another up-and-coming selection method, on-line training and experience (T&E) measures, offer similar opportunities.

This is one reason why testing doesn't perfectly predict performance--because what you see isn't always close to what you get. What's worse, we tend to think we can spot the fibbers--which just plain isn't true. We're not nearly as good at most things as we think we are.

A person's real performance can be seen as true score or typical performance, and researchers have looked into the extent to which people self-inflate. The academic term for this is impression management. But for our purposes let's just think about an individual's ability to pull the wool over the employer's eyes--their Acting Quotient (AQ).

What causes someone's AQ to be higher?

- The person is a good actor or actress. It's just a natural skill. They know what their strengths are, and the play to them. They're good at reading you, and they modify their behavior to fit the subtle cues you're giving them. In fact people can vary their acting method depending on what kind of question you ask!

- They're very motivated to get your job. This could be good (my skills match your needs), this could be bad (I'm about to get fired).

- They understand what you're looking for. Again, this could be good (they did their homework) or less impressive (they know someone who works for you already).

Okay, now the important part. What can we do to mitigate the impact of someone's AQ?

- Ask difficult technical questions whenever possible. Not all jobs lend themselves to this, but whenever you can ask questions that require job knowledge and aren't easily predictable.

- Conduct extensive reference checks. Rely heavily on off-list checks. Talk to co-workers and customers, not just supervisors.

- Triangulate and be patient. Make application/resume review and interviews a small portion of your selection process. Make no decisions until all the information is in.

- Have a bias for work sample/performance tests. Make someone show you they can do things--not just tell you they can.

Now on the bright side, many if not most people (including myself) aren't very good at acting. Sure, I can put a positive spin on my accomplishments, but I'm not very good at making things up out of whole cloth. Which is good, because it means when I interview, like most people, what you see is generally what you get. But that doesn't mean there aren't Academy Award nominees out there, and it doesn't mean we can't do better at hiring the right person.

New MSPB study has gems

A new study out by the U.S. Merit Systems Protection Board (MSPB), "Attracting the next generation: A look at federal entry-level new hires," while focused on the federal government, has good information for all of us. The study of recent federal hires uses data from over 1,000 survey participants as well as personnel records. Some of the insights include:

- Both those under 30 and those 30+ were most likely to find out about the job through a friend or relative. On the other hand, those 30+ were more likely to use USAJOBs and agency web sites, whereas those under 30 (not surprisingly) were more likely to discover jobs through school job fairs or placement offices. In addition, those under 30 were more likely to use "non-government job sites."

- The most common forms of assessment used were (mostly unstructured) interviews and reviews of transcripts. Narrative statements, written exams, and job simulations were (unfortunately) used much less often--despite evidence of their superior ability to predict job performance.

- Agencies also relied heavily on the "point method" for reviewing training and experience--i.e., the more training/experience you have, the higher your score. This despite evidence that the correlation between experience and performance is complex and peaks fairly quickly, while education does not generally predict performance very well. Not to mention that they make little sense when hiring for entry-level jobs.

- There were very few differences between age groups when it came to what applicants wanted in a job. For both the under 30 and 30+ groups, salary and benefits came out tops. The only significant differences came when looking at pensions (more popular among those 30+) and health insurance and tuition reimbursement (more popular among those under 30). Job security was also mentioned by both groups as a big draw to the federal government.

- Those under 30 were much less likely to predict staying with the feds for their entire career (although still an impressive 45%), and more likely to predict eventually moving to the private sector.

- "Length of the process" was by far the biggest obstacle faced by applicants in their pursuit of a federal job.


MSPB recommends federal agencies make several improvements to their recruiting and assessment efforts, including:

- Use more predictive assessment tools instead of relying on training and experience measures.

- Use a balanced set of recruitment practices to reach all the various segments of the job market. MSPB points out that conducting a proper job analysis and identifying true minimum qualifications can help cut down the number of unqualified applicants.

- Market what is important to applicants. For federal agencies this means job security, benefits, and the ability to make a difference.

- Evaluate the hiring process and remove any obstacles that are unnecessarily lengthening the process.

- Avoid stereotyping based on generational assumptions, since the data indicate substantial similarities between age groups in what they want out of a job.

Again, good information for all of us. There's a lot more information in the report.

Et tu, HR?

It's easy to give out advice. HR does it all the time.

But how well does HR practice what it preaches?

In the most recent issue of the U.S. Merit System Protection Board's (MSPB) newsletter, Issues of Merit, in an article titled "Taking our own advice", the author describes attempts by MSPB to ensure they are walking the walk when it comes to hiring procedures.

So what did they find?

1) Their job announcements weren't all that attractive. They contained jargon and too much information. Perhaps more importantly, they didn't "sell" the job.

Solutions: Job description was refined and rewritten so qualifications were easier to understand, got rid of excess information, and made easier to read. In addition, they added a job preview component that helps applicants decide if the job is a good fit.

2) They were using questionnaires as an initial screen that had low validity.

Solution: Questionnaires were replaced with an accomplishment record, which they hope (and research suggests) will better predict who will succeed in the job.

3) They were using the "rule of three" for external hires which limited their ability to consider a broad candidate group.

Solution: "Rule of three" replaced with category rating, which allows them to consider more candidates.

4) Recruitment methods weren't as broad as they could be.

Solution: MSPB worked with OPM to feature their jobs prominently through USAJOBS, the federal government's online job posting site. In addition they made greater efforts to actively seek out qualified candidates whose resumes were in USAJOBS. Finally, they used professional organizations to help advertise their opportunities.

An honest review of recruitment and assessment procedures in any organization will undoubtedly result in areas for improvement. Kudos to MSPB for following their own advice.

January '08 issue of J.A.P.

The January 2008 issue of the Journal of Applied Psychology is out. Unfortunately there are only three articles directly related to recruitment and assessment, but they're pretty good ones, so let's dive in.

First up, a Monte Carlo investigation of the impact of faking on personality tests by Komar, et al. "What is a Monte Carlo investigation?" you may ask. Essentially it's when researchers use computers to simulate data scenarios rather than collecting actual data from participants/subjects/victims. Anyway, the researchers looked at the impact on the criterion-related validity (as measured by supervisory ratings) of conscientiousness scores adjusting for various "faking" scenarios. They found that the validity is impacted by a variety of factors, most notably proportion of fakers, magnitude of faking, and the relationship between faking and performance. Another shot across the bow of self-report personality inventories, methinks, although the debate will no doubt continue!

Next a fascinating study of motherhood bias in both expectations and screening decisions by Heilman and Okimoto. The researchers found a bias against both male and female parents when it came to anticipated job commitment, achievement striving, and dependability, although anticipated competence was uniquely low for mothers and seemed to be the major contributing factor to lowered expectations and screening recommendations. An unfortunate reminder that these factors do matter and something to watch out for. The results are reminiscent of negative behavior toward "pregnant" women found in a previous study.

Finally, Zyphur, Chaturvedi, and Arvey present a discussion of job performance. They address two subjects: the impact of past performance on future performance and individual differences in performance trajectories. Analyzing past literature, the authors note that performance feedback influences future performance directly and different individuals do have different latent performance trajectories, which has big implications for selection. Why? Because many assessment techniques (e.g., T&Es, behavioral interviews) rely on an general assumption that more experience equals better performance. This study adds ammunition to those that argue that assumption has serious flaws (or at least is overly simplistic).

In addition to these three, you may find the following interesting as well:

Challenging conventional wisdom about who quits: Revelations from corporate America. (great stuff for those of you interested in retention)

Effectiveness of error management training: A meta-analysis. (for all you trainers out there)

Effects of task performance, helping, voice, and organizational loyalty on performance appraisal ratings. (for those interested in performance ratings)

You don't know you

In a recent interview with Gallup Management Journal, Cornell psychologist David Dunning talked about why people aren't very good at judging themselves. Why is this important? Because it has a great deal to do with how we recruit and assess applicants.

A big part of the reason we're so bad at accurately judging ourselves is due to our self-serving bias--our tendency to take credit for successes but blame outside factors (other people, equipment) for our failures. This helps our ego out--if we were always blaming ourselves for failures and attributing success to outside factors, we wouldn't be very happy campers. But it has the downside of oftentimes blinding us to the real reason why things happen.

Dr. Dunning covers a wide range of topics in the interview, including gender differences, when overconfidence may be a good thing, employee training, providing feedback, and the serious implications of this phenomenon (e.g., think about doctors judging their skills as being better than they are).

As I said, this is important because a great deal of recruitment and selection is about self-assessment--a prime example is the growing movement toward online training and experience (T&E) questionnaires made easier with the spread of ATS products. Many of these questionnaires are chock full of questions that (no joke) aren't much different than: "How great are you at X?"

But it's not just about T&Es. People make judgments about themselves when deciding what jobs to apply for in the first place. They describe themselves in certain ways during job interviews (when the motivation to make yourself look good is even stronger).

What can we do about it? Simply put, verify, verify, verify. If someone claims to be the greatest Java programmer on the planet, make them show you. If they claim to be a great orator, make an oral presentation part of the hiring process. Then talk to folks that know their work to establish a history of competence. Don't take someone's word at face value because (a) they may be trying to snow you, but more subtly (b) they may not know themselves.

By the way, Dr. Dunning is co-author of one of my all-time favorite articles, "Flawed self-assessment: Implications for health, education, and the workplace" which describes how the inability of people to judge themselves accurately can result in very serious problems.

Last thing: if you're not already familiar with it, check out the fundamental attribution error, which is one of the other big things our brain is constantly doing. It has huge implications for how we judge others.

Great July 2007 Issues of Merit

The U.S. Merit Systems Protections Board (MSPB) puts out a great newsletter focused on staffing called Issues of Merit.

The July 2007 edition has some great stuff in it, including:

- Risks inherent with using self-assessment for high-stakes decisions, such as hiring (hint: people are horrible at it)

- Tips for workforce planning

- How to write good questions

- Analyzing entry hires into the federal workforce

- An introduction to work sample tests

Good stuff!

JPSP, Vol. 92, Issue 6

There's a new issue of the Journal of Personality and Social Psychology out (volume 92, #6), with some juicy research for us...


First up, a fascinating study by Kawakami, et al. that may assist with efforts to eliminate or minimize discriminatory behavior. Participants in the study were trained to either pull a joystick toward themselves or push it away when shown pictures of Black, Asian, or White individuals. They then took the Implicit Association Test (a measure of how connected things in your memory are, used in this context to measure bias) or were observed for nonverbal behavior in an interracial context. Results suggested that simply engaging in approach behavior reduced "implicit racial prejudice" (as measured by the IAT) and increased "immediacy" in the nonverbal situation. Could this be incorporated into some type of training to reduce recruitment and selection bias? We'll see. (Mere exposure may be the more likely training route)


Second, an article that directly relates to the current focus in assessment circles on measures of training and experience (dovetailing with the increase in ATS). Moore & Small note that people generally believe they are better than others on easy tasks and worse on difficult tasks. The authors propose that these difference occur because people have much more information about themselves than about others. The result is even stronger when people have accurate information about themselves (!). The solution, it would seem, is to provide people with accurate information about how others perform.

What might this look like? A simplistic example would be instead of having people simply select categories such as Expert-Journey-Learning-Beginner, provide some data on how many folks tend to fall into each category. Unfortunately, I doubt this would be enough to overcome our built-in inaccuracy when it comes to self rating--but everything helps.


Last but not least, a study of a non-cognitive trait--and it's not one of the Big Five! No, this time it's grit, defined by Duckworth et al. as "perseverance and passion for long-term goals." Using various measures, the authors show that measures of grit added incremental variance to the prediction of a variety of criteria, including:

- Educational attainment among two samples of adults
- GPA among ivy league undergrads
- Retention in two classes of West Point cadets
- Ranking on the National Spelling Bee

Grit was not correlated with IQ, but was highly correlated with conscientiousness. It only accounted for about 4% of the variance in predicting the above outcomes, but the incremental validity added was beyond both IQ and conscientiousness. Is this practically meaningful? Depends on your point of view. If you're dealing with a large candidate group, or a particularly sensitive one (e.g., peace officers), could be worth a second look. Methinks more research is needed, particularly research on any subgroup differences.