Want better prediction? Gather more data.

That's the bottom line from a study in the November 2009 issue of the Journal of Applied Psychology.

Oh & Berry looked at how adding personality ratings from peers and supervisors added incremental validity to self-ratings using a five-factor model measure. What were the results? Increases of 50-74% in operational validity across personality facets. They also looked at differential prediction of task and contextual performance (unfortunately those results weren't reported in the abstract). Bottom line? If you're using a personality assessment for promotions, strongly consider gathering data from co-workers.

Speaking of self-presentation, in the same issue Barrick et al. report the results of a meta-analysis of how self-presentation tactics (e.g., appearance, non-verbal behavior) impact interview ratings and later job performance. Results? "What you see in the interview may not be what you get on the job and...the unstructured interview is particularly impacted by these self-presentation tactics." An important reminder of how who the candidate seems to be impacts your assessment, and another reason to collect multiple sources of data.

There are a number of other great articles in this issue, such as:

How Major League Baseball CEO personalities impact important outcomes (like, um, winning).

How SJT and biodata measures add to the prediction of college student performance.

How personality scale validities change over time among a group of medical students.

Differences among letters of recommendation in academia between genders.

HR: C-level or AAA?

Just when we thought we were C-level, Dogbert reminds us how old-school CEOs view HR.

p.s. I think I like CPO better than HCO.

Explain away

Of all the low-hanging fruit in recruitment and selection, perhaps none are easier to implement than explaining your process. It's shocking how few selection processes are fully (and coherently) explained, not just in terms of getting from point A to point B, but why the points exist in the first place.

Turns out explaining the process to candidates matters. A lot. And while we already knew that applicant perceptions were important, a recent meta-analysis by Truxillo, et al. published in the International Journal of Selection and Testing clarifies the impact that explanations have. Specifically, explanations were related to:

- Fairness perceptions (important in their own right)

- Perceptions of the hiring organization

- Test-taking motivation

- Performance on cognitive ability tests

Furthermore, fairness effects were greater when paired with personality tests rather than cognitive ability exams.

What does all this mean? It's pretty simple, really--communicate, communicate, communicate. Explain in clear terms to all applicants what the full selection process is, and why. Imagine something like this:

"Thank you for your interest in applying for the Blog Reader position. The selection process will consist of the following:

Step 1. You submit your application and work sample by December 5.

Step 2. Your application is reviewed to ensure you meet our minimum qualifications found on the job posting.

Step 3. If so, your work sample is scored by internal subject matter experts. It will be judged on relevance to the position, complexity, and contribution to the profession. The top scorers move on to Step 4.

Step 4. You are contacted by Human Resources to set up an interview. The interview will last approximately 2 hours and will take place at our San Jose campus. They will take place in mid-January.

Step 5. You interview with the hiring supervisor and 2-3 potential co-workers. You will be asked a series of questions designed to measure your knowledge of Blogs. You will also be asked to complete a writing sample which will be judged for style as well as content. Top performers in these steps will be asked for a final interview with the head of the Blogger Division.

Should you fail to proceed to any Step, you will be contacted and told why."

That wasn't so hard, was it?

For more about applicant perspectives in selection, check out the entire December '09 issue.

And while I'm on the subject of research, check out the latest issue of the International Journal of Testing for good stuff on test compromise, DIF, and P-O fit. Oh, and don't forget about the entire issue devoted to test adaptation.

Last but not least, Practical Assessment, Research, and Evaluation (PARE) has put out several things lately worth reading, check 'em out.

What employers can learn from Twilight

Not since Harry Potter have I seen such obsessed fans. The buzz started several months ago in anticipation. And November 20th is almost here.

Don't know what happens on November 20th? Ask your daughter. Or granddaughter. Or, heck, pretty much any woman between the age of 16 and 30. That's the day that New Moon, the second installment in the wildly popular Twilight series, hits theaters. Why should employers care about this, other than anticipating that certain staff members will be out of the office that day? Read on.

Like Harry Potter, the Twilight books (written by Stephenie Meyer) are enormously popular, and the movies are too. Also like Harry Potter, fans range the demographic spectrum, although the most rapid fans seem to be women (not surprising given the protagonist and the love triangle she's in the middle of).

Most employers would kill to have the kind of brand devotion that Twilight fans have. If Twilight was an employer, they would be competing with Google for top talent. So what can we learn from this phenomenon that can help us with branding our organization?

1. People like good stories. Twilight was a phenomenon way before Robert Pattinson and Taylor Lautner (the male lead actors). Whether you're in banking, IT, public utilities, or a flower shop, you have stories to tell about how your organization has impacted others--or how your employees have impacted each other. Are you telling these stories, or letting stories be told about you?

2. People still read. Related to #1, the Twilight phenomenon, like Harry Potter, began with the books. There's a lot of hype about video these days, but given something interesting, people have no problem spending time reading it. What does your recruitment material look like--is it entertaining? Educational? Would you read it even if you weren't interested in a job there?

3. People like fantasy. There's an awful lot of reality out there right now--the recession, H1N1, wars--and people like to take a mental break. Don't be afraid to break out of the mold and try telling a story that takes people away from their day-to-day lives.

4. People like contests and "sides". One of the biggest, possibly the biggest, dramas within the world of Twilight is the competition between the two main lead male characters. Fans identify themselves as being on "Team Edward" or "Team Jacob". This isn't something you see employers do very often, and it requires a bit of built-in loyalty, but it's something that can engage fans even more.

5. People like being fans. There's something primal about being part of a group of people who share the same interest. If you give people something to be a fan of, they'll enjoy connecting with others who share their passion. This is what the Facebook fan pages are all about. Google has 300k+ Facebook fans. Twilight has over 4 million.

6. You can brand almost anything. Branding is about more than your website, or your recruitment fliers. It can become part of everything your organization does, if it's strong enough. And it's more than just a logo, it's about the organization's philosophy and accomplishments. Look around and you can probably see Twilight branded on almost everything. I'm surprised they don't have Twilight adhesive bandages. Oh wait, they do.

As November 20th approaches, be prepared for a media onslaught about Twilight. Whether you're a fan or not, use it as an opportunity to think about how your organization could garner that kind of excitement. After all, that's what leads high potentials to want to apply.

New Job Simulations Report

The U.S. Merit Systems Protections Board (MSPB) just released a great, easily digestible, report on job simulations.

The report includes several things, such as:

- Job simulations defined and advantages/disadvantages

- Types of job simulations (SJT, work samples, etc.) and concrete examples

- Benchmark data on satisfaction with candidate quality as well as how federal agencies currently use simulations (just don't look at GPA compared to job knowledge tests in Figure 2)

- Survey data on why simulations aren't used more often in the federal government (time and expertise, sadly, were the top reasons)

- A 5-step strategy for using job simulations

- References to support the use of simulations (and good selection in general)

A great, free resource for anyone wanting to learn more about one of the best selection mechanisms you can use. And particularly relevant as more and more organizations move to using training and experience (T&E) questionnaires as their first (quick but not particularly valid) hurdle.

Personality tests: Situation matters

Is a personality test the right selection mechanism for your needs? In trying to answer that question, an important consideration is: Does the job allow for the expression of personality facets?

There's a concept in psychology called situation strength. It refers to how the "strength" of a situation impacts the display of personality via behavior, and it's something important to remember when using measures of personality to predict job performance.

A situation's "strength" refers to the environment under which personality aspects are displayed; think of it like a rulebook. If Job A is described perfectly in exacting detail with very little room for deviation ("Place container A over part B..."), does one's personality really matter when it comes to successfully performing the job? Or thought of another way, if the rulebook repeatedly emphasizes using an aspect of personality (e.g., extraversion), how will people without the ability to express that behavior consistently fare?

If you're a software programmer, it probably matters little in the grand scheme of things how extraverted you are; analytical ability is likely much more important. Similarly, if you work in a call center and follow a script, differences in openness to experience probably don't mean much; extraversion is likely more important.

But what about something like conscientiousness--could the strength of a situation impact the relationship between this aspect of personality and job behavior? That's the question that Meyer and colleagues set out to answer, and an answer is published in a recent issue of the Journal of Organizational Behavior.

The way they went about trying to solve this puzzle was to conduct a meta-analysis at the occupation (rather than job) level. What did they find? A few things:

1) Uncorrected correlations between conscientiousness and performance varied widely between .06 and .23.

2) Correlations appear slightly stronger when using overall performance as the criterion rather than task performance (not surprising given previous research).

3) Stronger correlations were found in "weak" occupations.

What does this mean? Well, for one it reinforces the fact that the answer to "Do personality tests work?" varies greatly depending on what the job is and how you measure performance. But perhaps more interestingly, it suggests that at an occupation level we can expect that for jobs that come with built in rules regarding behavior ("strong" occupations), measures of personality aspects such as conscientiousness may not predict performance as well as for jobs with more flexibility ("weak" occupations).

So the next time you're thinking about using a personality inventory for selection purposes, consider: To what extent will incumbents be allowed to express their personality?

Is recruiting using SNS discriminatory?

I keep reading/hearing about how recruiting using social networking sites (SNS) opens employers up to discrimination lawsuits because of who uses the sites. For the most part, this just plain isn't true.

A recent Pew study is the latest to show that when it comes to using SNS like Facebook, MySpace, and LinkedIn, you really should have one primary demographic concern when it comes to ensuring a diverse candidate pool: age.

Not gender, at least not in traditional sense. While four years ago SNS users tilted slightly male (55%), the balance has essentially flipped today (54% female).

Not race, there simply do not appear to be generalizable differences in racial groups when it comes to these sites (in fact I've seen some data that suggest the user base on these sites is more diverse)--but things change, and this may vary with particular sites, so keep an eye on this one.

But when it comes to age, SNS users are disproportionately younger than the overall Internet population. In the words of the Pew report, "[this] doesn't mean that more older adults aren't flocking to SNS--they are--but younger adults are ALSO flocking to the sites, so the overall representation of the age cohorts in the SNS user population has actually gotten younger."

One demographic difference I don't see a whole lot about: disability status. Are individuals with disabilities more/less likely to use SNS? I think that's an important question we need to address if we're truly trying to diversity our candidate pools.

Myths about assessment

Despite plenty of evidence and commentary otherwise, several myths persist about personnel assessment:

1) Some tests are "objective", others are "subjective." This is a myth reinforced by no less than the U.S. Supreme Court on a regular basis. The reality is even the choice to use certain selection methods is a judgment. Sure, certain methods involve more ongoing judgment, but a multiple-choice test can be highly "subjective" and an interview highly "objective" depending on how they are made and used.

2) Only certain selection methods are legally considered "tests" and therefore vulnerable to legal scrutiny. Wrong. Anything you do to narrow down your candidate pool is technically fair game. This includes how you advertise, screen, and interview.

3) Good hiring is an art more than a science. Actually we have decades of research showing the opposite. Human judgment is full of flaws. Combine this with the fact that most people think they are experts, and you have a perfect storm of personal overconfidence. The time and effort spent creating standardized instruments targeting competencies relevant for a particular position will be well spent.

4) Even the best assessments can predict only a small fraction of job performance. It's true it's a fraction, but it's not small. Research indicates that close to 40% of the variation in performance can be predicted with assessments. That's nothing to sneeze at when you consider all the other things that impact performance (organizational climate, quality of supervision, reward structures, team composition, role clarity, resources, mood, etc.).

5) Good assessment will solve all your people problems. Yes, this is sort of the flip side of the above. Consultants like to pretend that with the right assessment instrument every person you hire will be the most productive, friendly, team-oriented person ever. The reality is that performance depends not only on what someone brings to the job, but leadership, organizational norms...all that stuff I mentioned in #4.

Do we have all the answers when it comes to hiring the right person? Nope. Is there enough best practice out there so that any hiring supervisor should be able to get the expertise they need to do significantly better than chance? Yep.

Latest EEO Insight

EEO Insight is quickly becoming a great resource for anyone interested in issues related broadly to equal employment opportunity. And this isn't just affirmative action plans--it includes anyone interested in recruitment and assessment.

In the latest issue (v1, #3), you'll read about:

- Alternatives to RIFs such as wage freezes and job sharing and the EEO implications

- Analyzing layoff decisions for statistical evidence of adverse impact

- Using multiple regression to detect race and gender differences in compensation

- Ricci in retrospect and lessons learned

- Reaching out to veterans and individuals with disabilities

- Results of the EEO best practices survey and (very good) recommendations

By the way, if you're interested in EEO issues and you're not already reading OFCCP Blog Spot, I highly recommend starting.

SIOP Name Change: Will They or Won't They?

The Society for Industrial and Organizational Psychology (SIOP) is considering a name change. For those that don't know, SIOP is a division of the American Psychological Association and has thousands of members devoted to research into a variety of phenomena, including recruitment and assessment.

SIOP was established in 1982 and a significant number of members have been asking on and off for years whether the name continues to accurately describe what they do. The problem is mainly with the "Industrial" part--it's just not a word that gets as much attention as it used to (remember when everything was "industrial strength"?).

The name change is something they've tossed around for years, but beginning in October they're going to survey their members and ask them to consider three alternatives:

1. The Society for Organizational Psychology (TSOP) - name's okay but continues implication that focus is on cleaning our offices; acronym sounds like a rapper; oh, and URL is taken.

2. Society for Work Psychology (SWP) - a little bland but definitely simpler; how do you say the acronym? Is it like "swap"? Or maybe "swip"? Oh, and restricts the field to "work", which is a little narrow.

3. Society for Work and Organizational Psychology (SWOP) - the most complete name in terms of description; acronym easy to use but has some interesting brethren. Too bad we couldn't come up with SWOT (ya know, as in strengths, weaknesses, opportunities, threats). Oh, and URL is taken. Sort of.

The winner between these will take on SIOP for the final determination.

My bet? Members will split among the three, none will receive a passionate endorsement and it will lose against SIOP.

Any takers?

Screening on personality: Legal loophole or pothole?

In a recent post over at ERE, the author mentions a website that provides employers with the ability to screen candidates based on a measure of personality that applicants complete online.

My reading of this article prompted the following internal debate:

Me #1: Boy oh boy, is that ever a bad idea. The Uniform Guidelines clearly state (Q&A #75) that a measure of a trait or construct cannot be validated based on content validity, which is what most employers are likely to rely on in this situation.

Me #2: Ah yes, but because personality tests typically result in much less adverse impact than traditional cognitive tests, are the Uniform Guidelines even likely to come into play?

Me #1: Maybe not, but you never know until you go through a selection process, so why take the risk?

Me #2: What about in cases where the numbers being screened are so small that adverse impact analysis is likely to be wonky? (that's a technical term)

Me #1: Well that may well be different, but you're missing the point.

Me #2: What is the point?

Me #1: That employers should use caution before screening based on constructs such as personality. They need to take validation seriously.

Me #2: But isn't it good that they're using an instrument that's at least based on an evidence-based theory of personality (the Big 5)?

Me #1: Absolutely, and props to them. But it is still incumbent on employers to realize the legal risks as well as the implications of using a self-report personality screen as a first hurdle.

Me #2: Fine, but aren't you being a little hypocritical? Haven't you said one of your goals is a giant database where applicant information can be matched with employer needs?

Me #1: True, but I was thinking more along the lines of verifiable skills testing, not self-report inventories.

Me #2: Actually in that post you specifically refer to Big 5 assessments.

Me #1: Hey! This isn't about me. This is about warning employers to make sure they know what they're doing when they screen based on personality measures.

Me #2: Aren't you making an awful lot of assumptions about the website's process without having actually used it or talked to the owners?

Me #1: Well, yes, but I'm a blogger. That's what we do.

Me #2: Ugh.

2009 IPMA/IPAC Conference Material

Just got back from Nashville after having attended the 2009 IPMA-HR/IPAC conference. Great information, great people, and really enjoyed Nashville.

For me, the conference was largely about one thing: leadership. Makes sense in times like these, people are focusing on how leaders can help guide organizations through rough waters. People like Bob Hogan made great arguments for the importance of leadership and how far we have to go in doing a good job selecting them.

If you weren't able to attend, you can see many of the presentations here. Here's a sampling of topics:

- The selection interview

- Assessment centers for supervisors and managers

- Employee engagement

- Online testing

Can't wait for next year's IPAC conference in Newport Beach!

Considering employee testimonials? Go video.

In the September 2009 issue of the Journal of Applied Psychology, one study stood out: the authors studied employee testimonials shown on recruitment websites. Results strongly suggest that:

(1) Including some type of testimonials increases your attractiveness as an employer; and

(2) Using more complex multimedia (video with audio) is clearly superior to simply pictures and text in terms of both attractiveness and credibility. This also helps mitigate any perceptual differences that occur when you increase the number of testimonials from minorities.

This is great validation for organizations that have put the time and effort into putting quality videos on their site.

Check out these other studies while you're at it:

Does recruitment method impact turnover? (short answer: yes, in the short run)

Interested in P-E fit? Check out this review and model development.

Like vocational interest inventories and statistics? You'll like this.

R&A Software Failures Hurt Taxpayers, Too

We tend to think of successes and failures of applicant tracking systems and other recruitment- and assessment-related technologies as impacting businesses--much of what's written is about large organizations such as Microsoft and Google and what software they decide to adopt.

But public sector organizations are using these technologies as well. And when they fail, it hurts not only the organization but taxpayers as well.

Case in point: the state of Washington recently decided to abandon their efforts to implement SAP E-Recruiting after nearly three years and millions of dollars. The state will now go with a hosted solution which is estimated to be $700-800,000 a year cheaper (and hopefully much easier) to maintain.

Having been ringside for some of this, I can tell you the problem was not with motivation or energy, or even IT knowledge. I suspect that a lion's share of the problem was related to the complexity of the program. This would match reports I've read that a significant number of organizations are moving away from single-vendor HR solutions and going with simpler, targeted products. It's also possible that businesses find it easier to implement these programs because resources (particularly internal experts) are easier to move around and buy-off is easier to obtain.

I wish them luck on their next purchase, and hope they do due diligence in their research (you can often find others who have had problems). Some type of audit may help them determine exactly what went wrong and how to prevent it the next time around. It's not just a matter of time, energy, and expense on the part of the organization, these failures impact applicants, hiring supervisors, HR staff, and ultimately taxpayers.

SNWs for R&A professionals

In the August 2009 issue of IPAC's Assessment Council News I write about how recruitment and assessment professionals can take advantage of social networking websites (SNWs) such as LinkedIn, Facebook, and Ning, and offer some cautionary notes about using them.

The article covers what will be familiar ground to many of you, but I tried to also talk about how we can use these sites for professional development, not just for sourcing or selection.

The article ends with a very Web 1.0 idea--a solicitation of letters to the editor. I'll make the same request here: what do YOU think about these websites--flash in the pan or here to stay? Approach with kid gloves or jump right in?

Latest IJT and PP

The latest issue of the International Journal of Testing has some good stuff in it, particularly if you're in education. Here's a sample of what's available:

Correcting fallacies in (construct) validity, reliability and classification


Then head on over to Personnel Psychology to check out the Autumn issue. Here's some of what it covers:

Conscientiousness and KSAs predicts leadership in the military

Job component validation, meta-analysis, and DOT ratings

Personality variables and job search behavior and success

The importance of specificity and observability in job analysis ratings

Ricci presentation

On August 13th I gave a presentation at a PTC-NC luncheon about the Ricci decision. We had a great discussion about the implications (which remain to be seen) and dissected several passages from the decision.

One of the questions that came up had to do with the pass point. I didn't know the answer at the time, but looking back at the case it turns out that the City charter mandated a 70% pass point for these exams. Which is funny, because I made a joke about how 70% is the magic cutoff score given its ubiquity, particularly in the public sector.

In a perfect testing world would the pass point be based on an analysis of the minimum competency level required for the job? Yep. Did the 5-member majority in this case care? Nope.

You can view the (mostly visible) slides below.

September '09 IJSA

The September 2009 issue of IJSA (International Journal of Selection and Assessment) is chalk full of good stuff. Let's dive in.

1) An important update of the "guidelines and ethical considerations for assessment center operations"--a must for anyone interested in the appropriate use of assessment centers.

2) Speaking of assessment centers, here's a meta-analysis of how they correlate with cognitive ability and personality, as well as the proper way to weight the results.

3) Speaking of cognitive ability, curious about the correlation between ability and faking? Check out this large-sample study of faking on a biodata measure.

4) Worried about what your applicants think of your selection method? Frame it as select in (accept) rather than select out (reject).

5) Want to make sure your raters are rating accurately? You may want to re-think stocking your panel with agreeable people (sounds like a lot of fun for the exam analyst!).

6) Before you put the finishing touches on your new online job application system, make sure you pay attention to its features, user friendliness, and efficiency. I like to think of this as "Googley."

7) Looking for a measure of person-job fit that relates equity of contribution to reward? Check this out.

That's all for now!

Research update

Here are some research studies that you may have missed:

Predicting applicants' job pursuit behavior from their selection expectations: the mediating role of the theory of planned behavior

Complexity in the relationship among the subdimensions of extraversion and job performance in managerial occupations

The use of personality test norms in work settings: Effects of sample size and relevance

Networking as a job search behaviour: A social network perspective

Bonuses:

Avue recently published a white paper titled "The state of the federal workforce: A case for insourcing" that, while focusing on the federal government, has some good information and tips for all of us.

Can't get enough analysis of the Ricci decision? Check out this webinar on 8/14.

Reminder: Want more frequent updates? You can follow my shared items here.

July 2009 J.A.P.: SJTs and more

Situational judgment tests (SJTs) have a long tradition of successfully being used in employment tests. These types of (typically multiple-choice) items describe a job-related scenario then ask the test-taker to endorse the proper response. The question itself usually takes one of two forms:

1) What SHOULD be done in this situation? ("knowledge instruction")

2) What WOULD you do in this situation? ("behavioral tendency instruction")

What are the practical differences between the two? Previous meta-analytic research, specifically McDaniel et al.'s 2007 study, revealed that knowledge instruction items tend to be more highly correlated with cognitive ability, while behavioral tendency items show higher correlations with personality constructs. In terms of criterion-related validity, there appeared to be no significant difference between the two.

But there were limitations to that study, and two of them are addressed in a study found in the July 2009 issue of the Journal of Applied Psychology. Specifically, Lievens et al. addressed the inconsistency in stem content by keeping it the same while altering the response instruction, and also looked at a large population of applicants, rather than incumbents, which tended to dominate McDaniel et al.'s 2007 sample.

Results? Consistent with the 2007 study, knowledge instructions were again more highly correlated with cognitive ability, and there was no meaningful difference in criterion-related validity (the criterion being grades in interpersonally-oriented courses in medical school). Contrary to some research in low-stakes settings, there were no mean score difference between the two response instructions.

Practical implications? The authors suggest knowledge instruction items may be superior due to their resistance to faking. My only concern is that these items are likely to result in adverse impact in many applied settings. Like all assessment situations, the decision will involve a variety of factors, including the KSAs required on the job, the size and nature of the applicant pool, the legal environment, etc. But at least this type of research supports the fact that both response instructions seem to WORK. By the way, you can see an in-press version of this article here.

Other content in this journal? There's quite a bit, but here's a sample:

Content validity <> criterion-related validity

More evidence that selection procedures can impact unit as well as organizational performance

Self-ratings appear to be culturally bound