National Work Readiness Credential

Have you heard about the National Work Readiness Credential?

It's a 3-hour pass-or-fail assessment delivered over the web that is designed to measure competencies critical for entry-level workers, and consists of four modules:

1. Oral language (oral language comprehension and speech)
2. Situational judgment (cooperating, solving problems, etc.)
3. Reading
4. Math

I love the idea of a transferable skills test; kinda like the SAT of the work world. I think this approach, combined with assess-and-select-yourself notions are two of the truly creative directions we're going in.

Downsides?
(1) Right now it's not available in all areas of the country.
(2) A searchable database (either as a recruiting tool or as a verification) would be great.
(3) Last but not least, employers have to be cautious that the position they're hiring for truly requires the competencies measured by this exam.

But all that aside, a promising idea. It will be interesting to see where this goes.

Here are links to some of the many resources available:

Brochure
Training guide
Candidate handbook
Assessment sites
Appropriate uses
FAQs

August ACN

The August, 2007 issue of Assessment Council News is out and Dr. Mike Aamodt provides his usual great writing, this time in article titled, "A Test! A Test! My Kingdom for a Valid Test!" where he goes over what you need to look for when selecting a commercially available test...in two easy steps!

Some of my favorite quotes:

"Previously, [the] clients had their supervisors create their own tests, and we advised them that this was not a good idea." (I just like the idea of saying that to clients, aside from the fact that it's true 99% of the time)

"Creating a reliable, valid, and fair measure of a competency is difficult, time consuming, frustrating, costly, and just about any other negative adjective you can conjure up. Think of the frustration that accompanies building or remodeling a home and you will have the appropriate picture." (So it ISN'T a coincidence that I enjoy testing and home remodeling. Whew.)

"...it is essential to remember that no test is valid across all jobs and that criterion validity is established by occupation, and depending on who you talk (argue) with, perhaps by individual location." (Just don't tell this to Schmidt and Hunter.)

More info about ACN, including links to past issues, here.

And by the way...major kudos to Dr. Aamodt for offering so much of his work online. This is rare and to be commended.

June 2007 IJSA

Yep, it's journal time again...the June 2007 issue of the International Journal of Selection and Assessment is out...

This time the articles fall into two main camps: those focused on applicant reactions to selection procedures, and those focused on personality testing.


Applicant reactions

First up, Landon and Arvey looked at ratings of test fairness by 57 individuals with graduate education in HRM. The test characteristics included validity coefficient, mean test score difference between minority and majority candidates, and test score adjustment for the minority group. Results? Fairness ratings depended almost as much on how raters rated as they did on the test characteristics! Implications? "Extensively cross validated with a global random sample of over 3 million with criterion-related validities exceeding .90!" may not be worth a hill of beans to some folk.

Next, Bertolino and Steiner with a similar study of test fairness using responses from 137 university students (ahh, where would our research be without students). Students rated 10 selection methods on 8 procedural justice dimensions. Results? Work sample tests were rated highest (as they usually are), followed by resumes, written ability tests, interviews, and personal references. All things that are expected by most job seekers. What wasn't perceived as well? Graphology--thankfully. The most important predictors of procedural justice were opportunity to perform and perceived face validity. Nothing earth shatteringly new, but some international confirmation of previous, mostly U.S., findings.

Speaking of international support for the importance of opportunity to perform and face validity among an international sample judging fairness of selection methods (say that five times fast), Nikolaou and Judge analyzed responses from 158 employees and 181 students in Greece. Methods rated highest (drumroll...): interviews, resumes, and work samples, across both groups. However, students reported more positive reactions to "psychometric tests" (i.e., ability, personality, and honesty tests) than did employees--an important distinction. Also, although there does appear to be individual differences in rating (see previous study), core self evaluations didn't appear to explain much.

In summary: Work samples and interviews--high fairness ratings and (potentially) high validity. A great combination.


Personality testing

First up in the personality section is Kuncel and Borneman's study of a new method for detecting "faking" during personality tests by looking at the particular way "fakers" respond. The authors found support for this method (sample was not described), with 20-37% of fakers identified. May not seem like a lot, but they had a false positive rate (incorrectly labeling someone a faker when they're not) of only 1% with a baserate of 56% honesty. Not too shabby. Interestingly, the "faker" pattern did not correlate with personality or cognitive ability test results.

Second, a very interesting study of the "dark side" of personality by Benson and Campbell. Using two independent samples of managers/leaders (N=1306 and 290), the authors found support for a non-linear (inverted U) relationship between "dark side" scores and assessment center ratings as well as performance ratings. So, for example, having a moderate amount of skepticism or caution is good--but too little or too much creates a problem. The instruments used were the Global Personality Inventory and the Hogan Development Survey.


Okay, there's a third category

Okay, one more study. de Mejier et al. analyzed results from over 5,000 applicants to Dutch police officer positions. The researchers were interested in rating differences when comparing ethnic minority and non-minority applicants. Results? Similar (if not more) assessment information was used to judge the two groups, but a large number of "irrelevant cues" were used to judge minority applicants. One other difference: when rating minority candidates, assessors relied more on the ratings of others. Another argument for a standardized rating process!

Monster Performance Assessment

Yesterday I posted about an assessment that Monster.com offers job seekers in order to help them understand their strengths, values, etc. I mentioned that I wished that Monster offered a more valid tool that employers could use to match position requirements to job seekers.

A comment to my post yesterday pointed out that Monster offers such a tool to employers, the Monster Performance Assessment. This "online behavioral screening application" was developed in conjunction with DDI and the idea is that an invitation to the assessment is e-mailed to potential candidates and the results are stored by Monster. The tool can be built into Monster job postings and is designed to integrate with several ATSs. The tool is available for several job families, such as customer service, health care, IT, and sales.

Cost? $100 per posting--very cheap when you think about price per applicant.
Completion time? 10-15 minutes--very important given short attention spans.
Items? Looks like situational judgment--thank goodness its not just T&E.

In terms of getting the right people, I'm generally a fan of this formula: brand management + realistic job previews + constant recruiting (especially through referrals). But if ya have a situation where you've got a heck of a lot of active candidates, this is a pretty elegant way to go, although I'd like to see a technical report. Could also use biodata or pesonality tests, but those are trickier. I also like that they explicitly measure face validity.

A brochure on the product is available, as is a very informative webinar (Internet Explorer only).

JASPER

A friend of mine sent me a link to an "assessment" that Monster is using (through MyMonster) called JASPER--Job Assets and Strengths Profiler. According to the website:

"Based on over 60 years of research, this fun and enlightening test will uncover your job strengths and preferences and help you use them to your advantage.
- Discover your work and leadership style
- Gain confidence in your job related skills
- Enhance your ability to work with others
- Improve your resume, job search and more."

The assessment was developed by Tickle, which appears to be one of these outfits that has a number of "pop" tests of questionable validity, some of which apparently are "PhD-Certified" (whatever that means).

Anyway, it does appear some work went into the development of this test (including factor analysis of survey results, which apparently gives the test "content validity, face validity, and reliability"!), although it looks to be a perfect example of a-theoretical test development. From Tickle's website:

"Led by a team of Ph.D. behavioral scientists, the research team determined which psychological and career concepts were supported by research and would also provide test takers and employers with the most accurate, useful, and in-depth feedback. The team turned to general career research as well as referenced tests such as the Self-Directed Search (SDS), the Strong Interest Inventory, the Myers-Briggs Type Indicator (MBTI), and Campbell's Interest and Career Survey (CISS). "

To their credit, they aren't claiming the assessment is the be-all and end-all:

"JASPER does not have what is called predictive validity, meaning that this test doesn't claim to predict one's future success. Rather, its purpose is to accurately assess the test taker's strengths as it relates to the work environment. "

So why am I even talking about this? Two reasons. First, Monster, one of the most trafficked job sites, is using it. Second, it does make use of a variety of different testing formats and measures, including (literally) sliding scales, picture comparisons (wonder how they would deal with accessibility issues?), and scrolling words that must be selected. Nothing earth-shattering, but it's nice to see some exploration of different methods.

Of course I HAD to take the test. Results? Apparently my type is "Mentor", which 10% of test-takers are. My leadership style is "Innovative", my work personality "Rousing", my universal skill "Communication", and my work style "Questioning." What, you thought my results would say "Crazy megalomaniac time-waster?" (Yes, this is a whole heck of a lot of different measurement concepts tossed into a single assessment. Yes, this is very Myers-Briggs-like in that you are a "type." No, reliabilities are not reported anywhere.)

One last thing: the results report is more substantial than many "pop" assessments and makes for interesting reading. This type of thing would be a lot more useful if it was actually tied to specific job requirements, which seems like something Monster could take advantage of for job matching...

Additional information include this whitepaper ("technical manual") from Tickle and this blog post from Monster.

Human Performance, Vol. 20, #1: The EQ-i:S

March journal madness continues with the latest issue of the journal Human Performance.

There's really only one article in here related to selection, but it's an interesting one, so let's take a look.

In this study of 229 students from an southeastern U.S. university, Grubb and McDaniel looked at the constructs measured and fakability of the Emotional Quotient Inventory Short Form (EQ-i:S), a popular proposed measure of emotional intelligence.

The bottom line results:

- Participants were able to "fake" their scores, raising them substantially (by .83 standard deviations). Not particularly surprising as it's fairly well established that non-cognitive measures can be "faked." (What's not clear is whether it matters...)

- The two "screens" built into the EQ-i:S to try to identify fakers correctly identified only 31% of the fakers.

- EQ-i:S scores were predicted by the Big Five measure with a multiple correlation of .79. This result, say the authors, "casts doubt on the construct of emotional intelligence as operationalized in the EQ-i:S."

But there's some additional goodness in this article, largely because the authors also had the participants take the Wonderlic Personnel Test (WPT) and a measure of the Big Five personality factors (IPIP):

- WPT scores were correlated strongly with only one Big Five factor--Openness.

- The correlation between WPT scores and gender was small.

- Black-White score differences on the Big Five factors were small.

- Gender was correlated with Big Five scales, but the nature varied depending on the condition (honest or faking).

- The ability to fake on the EQ-i:S was a function of cognitive ability and personality (mostly agreeableness).

There were other articles, one of which I'll discuss over at HR Coal.

New issue of IJSA--vol. 15, #1

The March journal madness continues with the latest issue of the International Journal of Selection and Assessment.

There's quite a lot in this issue (and plenty for you personality testing junkies), so let's take a look at some of the articles...

The main section

1. Hulsheger and colleagues report a meta-analysis of the operational validity of cognitive ability (general mental ability, g, whatever ya wanna call it) in Germany. After analyzing data from 54 articles and reports, the authors report a population correlation of .467 with training success and .534 with job performance. Interestingly (and counter to other research I've seen), they found the relationship was higher for low-complexity jobs.

2. Next, a study of situational judgment tests (SJTs) by O'Connell and colleagues. Using data from seven U.S. manufacturing companies (total N of around 1,000), the authors had findings in three areas. First,they found a mean Black-White difference on the SJT of .38 and a gender difference of -.27 (favoring females). Second, for task performance they found that the SJT added incremental validity to a cognitive ability test or a personality test, but not if both were already being used. Last, for contextual performance, they found the SJT added validity only if a cognitive ability test was being used as the sole instrument. Good stuff, and kudos as always to Dr. Michael McDaniel for making this (and other articles) available through his website.

3. Next up, a very interesting concurrent study of 154 customer service employees from DeGroot and Kluemper. Why very interesting? Because they looked at the impact of "vocal attractiveness." Results? Vocal attractiveness correlated with both situational interview scores and job performance. AND, two Big 5 factors (agreeableness and conscientiousness) predicted performance more strongly for people with more attractive voices. Interesting! I knew all those vocal lessons would pay off.

Let's see...what else looks good...

4. How about another concurrent study of person-organization (P-O) fit? McCulloch and Turban looked at the incremental validity of a measure of P-O fit, defined as the match between managers' description of the work culture with incumbent work preferences. Results? The measure added significant incremental variance over a cognitive ability measure in terms of predicting turnover (important given the historically high levels for these jobs) but was not related to job performance. A great illustration (along with #2 above) of the importance of the criteria being studied.

5. A study of Hartman's Color Code Personality Profile by Ault and Barney. If you don't know much about this test (I didn't), it classifies people into four colors--Red (motive: power), Blue (intimacy), White (peace), and Yellow (fun). Anyhoo, the authors state this is the first psychometric research on the highly popular instrument (don't even get me started), and found "high" test-retest reliability and support for the instrument measuring "some" personality traits. They note, however, that the instrument has high error variance and suggest caution when using it at the individual level. This is one of those studies you'll want to have in your back pocket when someone comes up to you and says, "Hey, I read about this cool personality test on my flight. Let's use that from now on."

Speaking of personality testing...

This issue has a special section on personality. Let's see what we've got:

6. A study that looks into response distortion on personality tests. Berry, Page, and Sackett found support among a sample of 261 managers for enhanced prediction of job performance when self-deceptive enhancements (SDE) were accounted for, but not impression management (IM) scores. Why is this important? Because most research on "faking" personality tests has looked at IM--intentionally inflating your scores to score better on the test. SDE, on the other hand, happens when you honestly believe you are greater than you are. I think I need this article.

7. The brain stimulation continues with a meta-analysis by Connolly, Kavanagh, and Viswesvaran. The authors were looking at the relationship between self-ratings and observer-ratings of Big 5 personality dimensions. Results? Correlations in the .5-.6 range, suggesting each contributes substantial unique variance. Oh yes, and duration of acquaintance had a large moderating effect, but source of observer ratings did not. I may need this one too.

8. Last but not least (yeah, I skipped a couple, but I know your e-mail Inbox is filling up as we speak), Smithikrai found, in a sample of 2,518 Thais employed in seven different occupations, that the Big 5 factor of neuroticism was significantly negatively correlated with job success across the board, while extraversion and conscientiousness showed significant positive correlations. Conscientiousness was the only trait to predict success across occupations (sounds familiar!).

That's all folks! Enjoy your weekends!

Q&A #4: Joe Murphy

This is the fourth in a series of Q&As I'm doing with thought leaders in the area of recruitment and selection.

This edition features Joe Murphy, VP and Co-Founder of Shaker Consulting Group. Brief bio: Joe is a former principal at SHL USA, Inc. and co-founder of Olsen, Stern, Murphy & Hogan, Inc. He has published articles in a variety of professional publications and presented seminars on staffing metrics at a number of conferences. Joe is very active in the recruiting space (e.g., ERE) as well as an active member of SHRM, having conducted the 2004 survey of assessment practices and a national web cast on Quality of Hire.

I've been meaning to post about Shaker's Virtual Job Tryouts for a while because I like what I've seen, so this gives me a chance to highlight it. Joe's thought-provoking responses are longer than previous Q&A'rs, but well worth the read.

(Note: as always, some links are provided by yours truly)

BB: What do you think are the primary recruitment/assessment issues that employers are struggling with today?

JM: Education. Both for consumers of assessment products and providers of assessment resources. The retailer SYMS uses the phrase: “an educated consumer is our best customer.” This is also true in the field of assessment.

Just as recently as last year a VP of HR asked me if using assessment was legal in the US. I was a bit shocked.

I walked the exhibit hall at the EMA conference in San Diego and the SHRM conference in DC last year and stopped into many booths that offered some form of testing or assessment. I asked a few simple questions that I thought might trigger a good dialogue and allow me to gauge the quality of the resource. I varied my probing based upon the responses I received.

1. What do you do to establish job relevance of your assessment?
2. What steps do you take to conform to the uniform guideline on employee selection procedures?
3. Describe your approach to job analysis.
4. What criteria have been used in your validation work?
5. Do you have a sample validation technical report I could examine?
6. What norm groups do you have?

With the exception of a few providers, those questions brought blank stares or “What do you mean?, type responses. One of my favorite responses was: “Well we didn’t use prison inmates like one firm did.”

My take aways were:

• Many assessment providers have no meaningful results or best practices to share
  Training personnel working in an exhibitor’s booth on knowledge of assessment fundamentals is not valued
• And last, but most telling - Assessment providers do not expect a well educated assessment consumer

I have been involved in objective candidate evaluation practices as a user or provider for over 25 years. From my experience HR practitioners have limited knowledge regarding the use of objective candidate evaluation methods. This is most evident when exploring one word: Valid.

Validity is not an easy construct to wrap your arms around. However, many assessment consumers fall prey to the generic offering “Our test is valid.” It never dawns on many to ask: “Valid for what?” The answer to that question is far more important than just being valid.

I strongly recommend that any one considering the use of assessment go to a nearby university and take two courses: one on Tests and Measures and another on Personality Theory or Measurement. These courses will raise the caliber of users significantly.

BB: What is an example of an innovative or creative recruitment/assessment practice that you've seen recently?

JM: A focus on the nature of the candidate experience has been leading companies to be innovative. There are a number of companies that have taken the resume out of the front end screening and candidate evaluation process. The resume is then only used as context for the structured interview. By using scorable applications or biodata questionnaires, companies have become more objective at putting candidates into “Yes” and “No” piles. This also sends a nice message: “No resume needed to apply here.”

At TalUncon [Ed: Joe's link] in Redwood City, CA Gerry Crispin was talking about competency based interviews captured on video. A candidate could receive a set of questions, record their responses and post them to a YouTube like location for the recruiter. This raises questions about what is really being evaluated, but is certainly can provide side-by-side comparisons.

We like to think our Virtual Job Tryout™ is out on the leading edge. This approach to objective candidate evaluation transforms the candidate experience into an interactive, informative two-way exchange. The candidate learns about the job by performing work samples, and making choices on how they might handle specific interactions. Recruiters learn about work style and job related competencies, all from one experience.

BB: What is an area of research that you think deserves increased attention?

JM: Three things have been gnawing at me recently:

1. Attitudes and beliefs around un-proctored, on-line assessment.

I believe industry has to accept that there are sound and effective methods to administer and interpret un-proctored assessments.

2. Why do organizations place such low expectations to be measurement oriented on HR?

I believe we hide behind the term “soft costs” as an alternative to doing the rigorous work of developing the infrastructure and discipline of measurement required for the task at hand

3. How can the misguided opinion of one executive prevent an organization from realizing the gain available from objective candidate evaluation methods?

I believe we need to find better methods of documenting and presenting the benefits of assessments for executive audiences.

Here are three examples that illustrate my points.

One of our clients has the enviable, yet challenging scale of a 500 to 1 applicant to hire ratio. Their current methods of screening have been deemed both unfair and un-scalable. Unfair because the recruiter wrestles with looking at 50 candidates and makes a hiring decision, and wonders what talent was in the 450 with only a resume in the data base. Un-scalable because their growth plans simply make this semi manual process and the scope of needed recruiters unreasonable. The marketplace demands a faster and more objective data gathering process from the candidate. Relying on resumes searches or forcing the candidate into an on-site experience adds steps and negates cycle time reduction initiatives.

I heard a participant at a conference on HR Metrics openly state she went into HR because she was “bad with numbers.” The unfortunate part is that being light on that competency was OK with the company’s executives. The term “soft costs’ shows up a great deal in HR discussions. Some might say that term is used because you can’t measure it. If that is the case, will your boss mind if you double your soft costs next year?. There are sound methods to make the intangible more tangible. At the other end of the measurement continuum, one recruiter I met at an EMA conference stated her company captured and reported on 27 variables that made up their cost per hire figure. It can be done.

The now retired CEO of one of a company had a bad experience with an assessment center for executive development early in his career. As he rose through the ranks, he was able to forbid the further use of assessment in the organization. This edict lasted several decades. Last year, objective candidate evaluation was implemented at a new plant start-up. The financial implications were described by the VP of HR as one of the highest ROI projects in the company last year. Objective candidate evaluation has economic value.

BB: As someone who has their own consulting business, have you noticed any changes or patterns in the types of requests you're getting from clients?

JM: I see two camps. The novice user is still seeking the silver bullet. They want the omnibus assessment that can be used for many positions and predict any job behaviors. This is exasperated by the providers who lead under-educated HR practitioners to believe such an assessment exists.

Practitioners with a passion for process improvement get excited about taking measurement more seriously and seek more job relevant approaches to objective candidate evaluation. Project teams with six sigma training see the application and value of assessment clearly. The experienced user has achieved some degree of success and has learned that the more job relevant the assessment, the more accurate and valuable the results. Sound analytics help a user sees both the limitations and value from assessments.

BB: Have you read any books or articles lately that you would recommend to the professional community?

JM: I am in the middle of reading Susan Conway’s The Think Factory (Wiley 2007) [Ed: Joe's link]. Her mission is to bring Lean and Six Sigma practices to Information Worker (I Worker)process models. In reading it from a recruiting perspective (one of my major frames of reference) I continually see how formal and objective data gathering methods such as job analysis, assessment, and predictor-criteria correlation analysis fit right into her framework. The hiring process is a perfect example of her definition of I-Work: gathering and/or transforming data and making decisions,.

I have been pushing forth the concept of measuring the financial impact of waste and re-work in staffing processes for years. Conway addresses these issues, re-work in particular, as being significant drains on profitability. The high turnover in many entry level jobs causes re-work to the tune of millions of dollars. Yet company presidents have looked me in the eye and said, “That’s just the way it is in our business.” Responses like that change quickly when someone owns the budget for staffing waste.

Susan also addresses issues of efficiency and effectiveness in a manner that reinforces the real value driver is the later. While cost per hire and cycle time factors are important to manage, the high gains come from increases in effectiveness. This translates into making hiring decisions based on candidate data that correlates to performance outcomes.

I can’t wait to finish the book and apply a range of her assertions and models in communicating what we do. The alignment is fascinating.

BB: Is there anything else you think recruiters/assessment professionals should be focused on right now?

JM: I conducted a small survey (N = 558,) with SHRM in February of 2004. 85% of respondents stated they had not conducted an ROI analysis of their selection practices. This forced me down two paths:

1. Why do organizations allow so many resources to be invested in a business process without demanding some measure of return or contribution?
2. Why do staffing and HR professionals jeopardize their credibility by not holding themselves accountable with sound economic measures of performance?

The most common metrics in staffing are based upon cost and time. When you report cost, you get pressure to make it smaller. In essence budget constraints on staffing functions might be directly attributed to the nature of reporting.

I suggest, if not compel recruiters to partner with other in-house disciplines such as finance and process improvement. Use these combined resources to focus on value metrics, and invest in the rigor and discipline of capturing business process data that can be used to document the contribution of recruiting to the top line and the bottom line.

If a tree falls in the woods and nobody hears it, does it make a sound?

If a recruiter adds value but nobody knows it, does it make a difference?

Indeed. Thank you Joe!

Personality testing basics: Part 2

There continues to be great interest in using personality tests for personnel selection. In Part 1 of this series I reviewed some of the major modern research on personality testing. In this post I will review some of the major commercial personality tests that can be used for selection.

This is by no means an exhaustive list, but these tests have been around for a while, have generally withstood professional scrutiny, and are used by many organizations as part of their hiring process. While not all of them are strict measures of the Big 5, all are based or strongly related to those factors. I mentioned that while not everyone agrees that the Big 5 is the best way to describe normal personality, it does have considerable support.

Below I have provided a brief description of these tests along with any evaluative comments (if I had them) from the Mental Measurements Yearbook (MMY), one of the few objective reviewers of commercial tests. All can be administered via computer or paper-and-pencil and vary somewhat in price (approximately $10-30 per test--contact publisher for details).

The Tests

1. Hogan Personality Inventory (HPI) - The HPI, created by the well-known Dr. Robert Hogan, is based on the Big 5 but actually has seven scales--Adjustment, Ambition, Sociability, Interpersonal Sensitivity, Prudence, Inquisitive, Learning Approach (note that several names have changes in the last few years). The HPI consists of 206 true/false items that according to Hogan Assessment Systems are written at a 4th grade reading level and take 15-20 minutes to complete.

MMY (1998) quotes on HPI:
"...[the HPI] appears to be a theoretically sound, carefully conceptualized, and well-validated instrument offering practical utility for organizations."
"...[the HPI] is a valid and reliable test..that is recommended for use as part of a battery in vocational and personnel selection settings."

2. 16PF - The 16PF is published by IPAT and is based on the research of Dr. Raymond Cattell, one of the most famous psychologists who (among other things) is credited with theorizing the existence of fluid and crystallized intelligence. The 16PF measures sixteen "primary factors" as well as the Big 5, contains 185 three-choice items, and IPAT states it is written at a 5th grade reading level and takes 35-50 minutes to complete.

MMY (1995) quotes on 16PF:
"This psychometrically sophisticated measure is a valuable contribution to the testing repertoire of counselors and clinicians."
"...this well-known research instrument has stood the test of time and is supported by a vast body of data."

3. Personal Characteristics Inventory (PCI) - The PCI is a measure of the Big 5 (and subscales) and was developed by Drs. Michael Mount and Murray Barrick, authors of a very influential study on personality testing. It consists of 150 three-choice items that Wonderlic claims are written at 3rd-4th or 8th-9th grade level depending on the scale, and takes about 25-30 minutes to complete.

MMY (2005) quotes on PCI:
"The authors have devised a simple, clear measure of [the Big 5] that may be helpful in some personnel work. Presently available evidence suggests more development work would be necessary...before this reviewer could recommend its use."
"...a good first step in the development of a personnel selection instrument...There are some important reliability and validity issues the authors must address in their revisions."

4. NEO Personality Inventory-Revised (NEO PI-R) - The NEO PI-R is one of several tests of the Big 5 offered by Psychological Assessment Resources (PAR). The test authors, Drs. Paul Costa and Robert McCrae, are well-known researchers and authors of several influential studies on personality testing (hey, who said psychological research doesn't pay?). The test (available in both self-report and observer-report versions) contains 240 items that PAR claims are written at a 6th grade reading level; the test takes 35-45 minutes to complete.

MMY (1995) quotes on NEO PI-R:
"These scales should be considered a standard set of useful tools for personality assessment and may provide a useful bridge between basic research in personality psychology and applied psychology."
"...a reliable and well-validated test of personality features...an instrument that represents a comprehensive operational translation of the Five Factor Model of personality."

So which do I use?

Good question, and there's no easy answer. My advice? First analyze the job(s) and make sure that a test of the Big 5 makes sense given job requirements. Then contact the test publishers and get demonstrations (they should all be willing to do this)--this will allow you to get a feel for the exam as well as the customer service offered by the company. Whatever you do, don't try coming up with a personality test yourself!

Wonderlic releases revised Personnel Test

Any day I get a chance to post about the testing company with my favorite name, it's a good day.

That day is today, and that company is Wonderlic.

Many of you are no doubt familiar with Wonderlic, one of the oldest testing companies around. It's also one of the most famous or infamous depending on how you look at it. The Wonderlic Personnel Test (WPT), a test of cognitive ability (intelligence if you prefer) has been around since the 1930s and has been used by many organizations, including the military and (more recently) professional football teams.

In 1971, the Wonderlic received some unwelcome media (along with the Bennett Mechanical Comprehension Test) when the U.S. Supreme Court ruled in Griggs v. Duke Power that the company had misused the test (essentially they didn't do the leg work to tie WPT scores to successful job performance). And yes, Griggs is also the case where the Supreme Court articulated the "adverse impact" theory of employment discrimination. In fact, if you haven't read Griggs, I highly recommend it, for historical purposes if nothing else.

Anyhoo, Wonderlic recently announced the release of a revised version of the WPT, appropriately named the WPT-R. What's new? Some changes to content, and some changes to format. It's still a 12-minute test, available via bubble sheet or online, scored by Wonderlic.

Click here to see some sample questions from the WPT-R.

New issue of Personnel Psychology

The Winter issue of Personnel Psychology showed up in my mailbox yesterday, and although there's only one article that directly addresses recruitment/assessment, it's a good one.

The study by McDaniel, Rothstein, and Whetzel analyzed whether publication bias exists in the validity information presented in technical manuals of 4 test vendors. To be honest, I was shocked that only 2 of the 4 showed evidence of moderate-to-severe bias. In both cases, the publisher tended to report only statistically significant correlations.

Unfortunately the names of the test vendors were not given; they were specifically omitted to avoid stigmatizing the vendors and to make the article easier to review. IMHO both of these reasons are weak, but that's neither here nor there. Still, a great introduction to the "trim and fill" method and provocative results that have implications for any meta-analysis.

Another reason this article is interesting is it references a chapter written in 2005 by Sue Duval where, using the same "trim and fill" method, the criterion-related validity reported by McDaniel et al. in 1994 for structured interviews (.27) was found to be likely be an overestimate, and the true value (.21) is closer to that of unstructured interviews. As McDaniel himself points out in an upcoming publication, this is not the last word on the validity of structured interviews, but it certainly gives one pause! That study, combined with the 2005 study by Roth, Bobko, and McFarland on work sample tests really cry out for a re-do of Hunter & Schmidt's seminal paper.

By the way, big props to Dr. McDaniel for allowing access to so many of his publications.

Unproctored online testing

There's a very interesting thread going on over at IPMAAC's listserv about the appropriate and potential use of unproctored internet testing that started with this post from Eric Palmer from the City of Forth Worth.

Some of the back-and-forth has been about how common unproctored internet testing is (which says something about the profession's knowledge of its own practices), but the more interesting aspects have to do with candidate perception, logistics, reliability, and validity.

IMHO internet testing, along with modular testing, is the future of professional assessment for many hiring scenarios, particularly as we move more and more towards an employment society based on knowledge work and providing services. You simply cannot beat the utility of administering quality on-line exams--regardless of how you verify the scores (seems to me the best way is to re-administer similar items to the final candidate pool).

That said, decisions regarding how to assess should always be based on the KSAs/competencies required by the job and operational needs. But the ability to gather and track data, the ability to quickly link test results with job performance and other candidate information, and the flexibility of a "one stop shop" for testing simply cannot be ignored.

10 tests to consider

An article on Inc.com describes 10 tests (including cost from select vendors) that are, in their words, "extensively validated, highly respected."

The list includes several that I would also recommend folks consider, including the Watson-Glaser and Wonderlic for cognitive ability and the NEO, 16PF, HPI, and OPQ for personality. (By the way, if you're going to invent a test, make sure you consider the acronym; in fact you may want to come up with the acronym first)

But, as pointed out on EASI-HR blog you need to think carefully about the situation before running out and buying one. Do you have documentation showing the competency/KSA measured by the test is required for the job? What are you going to do about pass points? These are just some of the questions that need to be addressed.

By no means should that dissuade the use of tests. But like any other high-stakes purchasing decision, conduct a risk assessment and cost/benefit analysis before making the leap. And use resources like Buros test reviews to check out what the experts have to say.