Unvarnished Unwrapped

A few months ago I mentioned a website called Unvarnished, which was getting a lot of mixed press. The basic concept is (as they describe it) Yelp mixed with LinkedIn. You provide anonymous reviews of people you've worked with. Call it a social resume, call it a web-based reference, I call it fascinating. And something that anyone interested in recruitment and assessment should pay attention to.

I had a chance recently to test out the site, and then I met with the co-founder, Peter Kazanjy. I'm still not sure which direction this will go, but I think you'll agree after reading what follows that the concept merits our attention.

The Test Drive
First, the test drive, which started with an invitation through Facebook from Peter. After spending some time on the site, I'm more optimistic about Unvarnished in some respects, more cautious in others.

Why optimism? The site hits it out of the ballpark on two accounts: it's simple and fast (at least someone learns from Google). Simple and fast is good, because one of the biggest challenges will be building a large community and making reviews easy helps immensely.

Even better, the ratings are relevant. This isn't a popularity contest, it's an honest attempt to provide a useful description of someone's performance. While it's unlikely the rating scales were developed after reading a Personnel Psychology meta-analysis, I was pleased to discover that they pass the smell test and some even have benchmarks.

The ratings consist of an overall performance rating (5-point, anchors described), what job you are rating the person in, four 10-point scales that are described but not anchored (skill, relationships, productivity, integrity), and an open-ended strengths/areas for improvement box. That's it. It's like a super-basic reference check form that takes all of about a minute. You can see what this process looks like below.

Your Unvarnished homepage is fed by your network and generates suggestions for your review (PMYK--People You Might Know). You can review people at any time (even people who haven't claimed a profile) or request reviews using your Facebook contacts. The open comments section is limited to 500 characters to encourage people to review and move on.

The site was developed to be heavily reliant on algorithms. A reviewer's reputation is based in part on the pattern of reviews they have generated as well as how their reviews have been rated. Recommendations for reviews are backed by similar math. A smaller (but important) feature is a profanity filter, which may allay some concerns regarding people looking to settle a score.

Speaking of Facebook, one of my major concerns is it relies HEAVILY on Facebook (not unlike Quora). At least in its current iteration, your identity is verified through having a Facebook account, you invite people through your Facebook contacts, and invitations are posted to Facebook. This is both a potentially good thing (e.g., cuts down on spammers) as well as a bad thing (e.g., not everyone wants to use their Facebook profile in the service of another website). It also begs the question of what would happen if Facebook went belly up. Read on to see what Peter had to say about this.

The interview

I happened to be in San Francisco (interestingly enough to go the Exploratorium) and had an opportunity to talk with Peter about the product and his company. After talking to him for about 10 minutes, one thing became abundantly clear: this guy has thought a great deal about online reputation management. I'd love to get him and Bob Hogan together.

Background
The impetus for the site came from a couple directions. One was a previous job where he was continually surprised that competitors (e.g., a certain company we'll call Bicrosoft) would recruit away the lesser talented employees. Why? His theory is they lacked important information--namely the reputation people had within the organization. (One could argue that they should have done better reference checks, but we all know how easy/productive those can be)

On the other end of the spectrum, he saw excellence not being properly recognized and questioned whether upper management really knew what their talent looked like (a point not lost, btw, on purveyors of performance management software with their 9-box grids and helicopter views).

His ah-ha moment (or one of them) came when he realized that if social ratings can work for things like books and software, couldn't they work for people? If he could develop a site that aggregated high quality information about people's performance, talent decisions would be higher quality as well as more fair (it's hard to argue with that goal).

Peter also believes there is an important employer segment not being served by existing background/reference checking processes. Employers that hire hourly workers rely largely on criminal/credit checks. Those hiring for executive-level positions often rely on high cost search firms. But for employers hiring large number of employees in the middle, there isn't really a good option.

Such a site would have three primary users: those being reviewed, those providing reviews, and those using the information (e.g., employers checking out candidates and vice-versa). And quite commonly a single individual could be in all three roles at various times. The site would have to accommodate all three perspectives.

Concerns/criticism
So what about my concerns? When it comes to the reliance on Facebook, Peter pointed out that it's a good bet Facebook will be around for a while, but the site is not being built to rely solely upon it. It has--or will have--the ability to use contact information from sites like Gmail and LinkedIn. I still have a concern about forcing people through Facebook, so it will be interesting to see whether this impacts the ability to generate reviews.

Concern from others has focused on the potential for abuse. But Peter made several important points. First, this isn't like an online newspaper comment space--those are anonymous with no repercussions for inaccuracy. In Unvarnished your reputation would suffer and your reviews become less valid (assuming your reviews are themselves reviewed). Second, this information is largely out there (or potentially at least) in the form of places like Twitter. But it's not in a central location that can be easily managed, and it's not objective. By guarding invites and relying on anonymity, the goal is to make legitimate reviewers feel safe in leaving honest feedback (whether it's an A+ or a D-) without worrying about the interpersonal implications.

What about stickiness--why would someone want to keep coming back? They're working on several (re)engagement initiatives. One idea is to provide people with periodic updates letting them know when people in their network have been updated (let's hope it doesn't turn into those updates from LinkedIn that are easy deleted). They're also working on the ability to follow an individual based on your "gestures" (e.g., you've reviewed them).

Future directions
The team is considering adding several features. One is the ability for reviewers to better define their relationship--e.g., describing not only the organization they worked together in but their relationship. This would factor into behind-the-scenes algorithms but would not be published.

They're also discussing allowing people to identify themselves, but this raises all of the associated issues such as accuracy, thoroughness, and feeling a need for reciprocity.

Another is allowing access to premium features (e.g., at a cost) such as making trusted reviews more obvious, something that "super users" such as recruiters would likely be willing to pay for.

In terms of opening up access, they're in no big rush to expand access beyond Facebook invites. While this may hinder their growth, it helps keep the data quality high, and they're willing (smartly, I think) to make this trade off.

Conclusion
Currently the company is focused on acquiring the talent it needs to succeed (check out the way they recently advertised for new engineers). One of Peter's primary concerns is that the community evolve in the right direction. Right now it's somewhat of a "love fest" with lots of positive reviews. The site will gain in usefulness when reviews are a combination of pros and cons.

Given what we know about performance ratings, it will be interesting to see if the existing invitation and rating process is sufficient to generate that depth. It also remains to be seen whether some type of incentive will need to be given to generate reviewers.

My overriding concern continues to be the size and diversity of the user group (right now primarily filled with Silicon Valley IT folk). Accuracy, something other writers have been obsessed with, is less of a concern for me after kicking the tires and talking with Peter. But we'll see how the promise and concerns ebb and flow with the user base as well as changes to the service.

At the very least, I hope you'll agree with me that this website is a fascinating development and one we should watch. After all, you might want to use Unvarnished to provide feedback on someone. Or research a potential boss. Or research an applicant. Or...you may be the applicant.

Three on EI

A couple of posts ago I wrote about the most recent issue of IOP and the focal article on emotional intelligence (EI).

Now there's another meta-analysis out by O'Boyle et al. in JOB which may add some support to fans of EI. Here are the main findings:

1) Corrected correlations of between .24 and .30 with job performance.

2) The three "streams" of measures (ability, self- or peer-report, and "mixed models") correlated differently with cognitive ability and personality measures.

3) Perhaps most interestingly, self-/peer-report and "mixed models" had the most incremental validity beyond cognitive ability and personality.

More evidence that these measures--whatever they're measuring--are correlated with job performance, and they seem to be picking up something new. But do we know what's being measured? And what aspect of job performance is being predicted (contextual rather than task performance seems likely)? And what about other considerations, such as face validity/applicant reaction?

For those of you into EI, here are a couple articles that you may have missed:

Mayer et al.'s overview in the 2008 issue of Annual Review of Psychology (PDF; full text)

Cote and Miners' 2006 piece in Administrative Science quarterly (PDF; full text).

June '10 IOP Part II: "Test bias"

In Part I of this two-part post, I described the first focal article in the June 2010 issue of Industrial and Organizational Psychology (IOP), devoted to emotional intelligence. In this post, I will describe the second focal article, which focuses on test bias.

The focal article is written by Meade & Tonidandel and has at its core one primary argument. In their words:

"...the most commonly used technique for evaluating whether a test is biased, the regression-based approach outlined by Cleary (1968), is itself flawed and fails to tell us what we really want to know."

The Cleary method of detecting test bias is conducted by regressing a criterion (e.g., job performance) on test scores along with a dichotomous demographic grouping variable, such as sex or race, and looking at the interaction. If there is a significant difference by group variable (e.g., different slope, different intercepts), this suggests the test may be "biased". This situation is called differential prediction or predictive bias. The authors contrast this with "internal methods" of examining tests for bias, such as differential item and test functioning and confirmatory factor analysis which do not rely upon any external criterion data.

Meade & Tonidandel state that while the Cleary method has become the de facto method for evaluating bias, there are several significant flaws with the approach. Most critically, the presence of differential prediction does not necessarily mean the test itself is the cause (which we would call measurement bias). Other potential causes are:

- bias in the criterion
- reliability of the test
- omitted variables (i.e., important predictors are left out of the regression)

Another important limitation of the Cleary method is the susceptibility of slope difference tests to low power--this can result in findings of no slope differences due to small samples rather than a true absence. In addition, because of the type I error rate of the intercept test, one is likely to conclude that intercept differences are present when none truly exist.

Because of these limitations, the authors recommend the following general steps when investigating a test:

1. Conduct internal analyses examining differential functioning of items and the test.
2. Examine both test and criterion scores for significant group mean differences before conducting regression analyses.
3. Compute d effect size estimates for group mean differences for both the test and the criterion.

The authors present several scenarios where tests may be "biased" as conceived in the traditional sense but may or may not be "fair"--an important distinction. For example, one group may have higher performance scores, but there is no group difference in test scores. Use of the predictor may result in one group being hired at greater or lesser rates than they "should be", but our judgment of the fairness of this requires consideration of organizational and societal goals (e.g., affirmative action, maximum efficiency) rather than simply an analysis of the tests.

The authors close with several overall recommendations:
1) Stop using the term test bias (too many interpretations, confounds different concepts).
2) Always examine both measurement bias and differential prediction.
3) Do not assume a test is unusable if differential prediction is indicated.

There are several commentary articles that follow the focal one. The authors of these pieces make several points, ranging from criticisms of techniques and specific statements to suggestions for further analyzing the issue. Some of the better comments/questions include:

- How likely is it that the average practitioner is aware of these issues and further is able to conduct these analyses? (a point the authors agree with in part)

- This approaches advocated mainly work with multi-item tests; things get more complicated when we're using several tests or an overall rating.

- It may not be helpful to create a "recipe" of recommendations (as listed above); rather we should acknowledge that each selection scenario is a different context.

- We are still ignoring the (probably more important) issue of why a test may or may be "biased". An important consideration is the nature of group membership in demographic categories (including multiple group memberships).

Meade & Tonidandel provide a response to the commentaries and acknowledge several valid points raised but end this article with the same proposition that they started the focal article with:

"For 30 years, the Cleary model has served as the dominant guidelines for analyses of test bias in our field. We believe that these should be revisited."

June '10 IOP Part I: Emotional Intelligence

The July 2010 issue of Industrial and Organizational Psychology has two very good focal articles with, as always, several thought-provoking commentaries following each. This post will review the first article and I'll cover the next in the following post.

The first focal article, by Cary Cherniss, attempts to provide some clarity to the debate over emotional intelligence (EI). In it, Cherniss makes several key arguments, including:

- While there is disagreement over the best way to measure EI, there is considerable agreement over its definition. Cherniss adopts Mayer, et al.'s definition of "the ability to perceive and express emotions, assimilate emotion in thought, understand and reason with emotion, and regulate emotion in the self and others."

- EI can be distinguished from emotional and social competence (ESC), which explicitly links to superior performance.

- Among the major extant models, the Mayer-Salovey-Caruso model (MSCEIT) represents EI, while the others (Bar-On; Boyatzis & Goleman; Petrides, et al.) primarily consist of ESC aspects. Importantly, this does not make the Mayer, et al. model "superior", just more easily classified as an ability measure.

- There are significant problems with all of the major measures of EI/ESC, such as convergent/discriminant validity and inflation (depending on the measure). According to Cherniss, "it is difficult at this point to reach any firm conclusions--pro or con--about the quality of the most popular tests of EI and ESC."

- Emerging measures, such as those involving multiple ratings (e.g., Genos EI), appear promising, although they may be more complex and expensive than performance tests or self-report inventories.

- Other, even more creative, measures hold even more promise. This includes video and simulation tests. This point was echoed in several commentaries. I've posted a lot about the promise of computer-based simulation testing, and EI may be one of the areas where this type of measurement holds particular promise.

- Context is key. I think this is one of Cherniss' most important points: the importance of EI likely depends greatly on the situation--i.e., the job and the "emotional labor" required to perform successfully. This point was also echoed in several commentaries. EI may be particularly important for jobs that require a lot of social interaction and influence, team performance, and in jobs that involve a high level of stress.

- Several studies have shown modest correlations between EI/ESC measures and job performance, but there are issues with many of them (e.g., student samples, questionable criteria). Newer meta-analyses (e.g., Joseph & Newman, 2010) suggest "mixed-model EI" measures may hold more promise in terms of adding incremental validity.

The commentaries provide input on a whole host of points which would be difficult to summarize here (I will say I enjoyed Kaplan et al.'s the most). Needless to say there is still an enormous amount of disagreement regarding how EI is conceptualized, measured, and its overall importance. Then again, as Cherniss and others point out, EI as a concept is in its infancy and this type of debate is both healthy and expected.

Perhaps most importantly, users of tests should exhibit particular caution when choosing to use a purported measure of EI as the scientific community has not reached anything close to a consensus on the appropriate measurement method. Of course pinpointing the exact moment when that occurs will be--as with all measures--a challenge.

Research update

Here are a few research articles you may have missed. Just links this time because...well, I'm really enjoying World Cup.

McGrath, et al.: Evidence for response bias as a source of error variance in applied assessment.
(May Psych. Bulletin)

Hulleman, et al.: A meta-analytic review of achievement goal measures: Different labels for the same constructs or different constructs with similar labels? (May Psych Bulletin)

Thornton & Potemra: Utility of assessment center for promotion of police sergeants. (March Public Personnel Management)

Bright: Why age matters in the work preferences of public employees: A comparison of three age-related explanations (March Public Personnel Management)

Thomas, et al.: Employee proactivity in organizations: A comparative meta-analysis of emergent proactive constructs (June JOOP)

Bjørkelo, et al.: Predicting proactive behaviour at work: Exploring the role of personality as an antecedent of whistleblowing behaviour (June JOOP)

Zibarras & Woods: A survey of UK selection practices across different organization sizes and industry sectors (June JOOP)

What we can learn from the baseball draft

Longtime Major League Baseball commentator Peter Gammons was recently interviewed on National Public Radio. What does baseball have to do with recruitment and selection? Quite a bit, actually, but in this case the conversation was even more relevant because he was discussing the accuracy of the baseball draft.

I think you will agree with me after reading/listening to his observations that the draft has several lessons for all kinds of employers:

- More information is better. As scouts have been able to gather and crunch more data about prospects, the accuracy of predicting how a pick will fare in the big leagues has increased. We know from assessment research that more measures are better (up to a point) in predicting performance.

- The type of information matters. Scouts used to focus on relatively narrow measures such as running speed. Today they consider a whole host of factors. Similarly, modern assessment professionals consider a wide range of measures appropriate to the job.

- Personality matters. While lots of data about skills is important in prediction, personality/psychological factors also play a big role in determining success. Personality has also been one of the hottest topics in employment testing over the last 20-30 years.

- It's rare for a single candidate to shine head and shoulders over the rest. Making a final selection is usually a challenge.

- Assessment is imperfect. Even with all the information in the world, a host of other factors influence whether someone will be successful, including which team the person is on, their role, and how they interact with other teammates. This also means "low scoring" candidates can--and do--turn into superstars (Albert Pujols of the St. Louis Cardinals was a 13th-round draft pick).

- Ability to learn is important. We know from assessment research that cognitive ability shows the highest correlation with subsequent job performance (esp. for complex jobs), and many have suggested that it is the learning ability component of cognitive ability that matters most.

- Good recruitment and assessment requires resources. Organizations that take talent management seriously are willing to put their money where their mouth is and devote resources to sourcers, recruiters, and thorough assessment procedures.

By the way, 2009's first overall draft pick, Stephen Strasburg, had an excellent debut last night for the Washington Nationals, striking out 14.

May '10 J.A.P.

Summer journal madness continues with the May issue of the Journal of Applied Psychology. It's a diverse issue, check it out:

- Taras et al. conducted a very large meta-analysis of the association between Hofstede's cultural value dimensions (e.g., power distance, masculinity, individualism) and a wide variety of individual outcomes. One interesting finding is the stronger relationship between these values and emotions (organizational commitment, OCBs, etc.) compared to job performance.

- Are high performers more likely to stay or leave? In a study of over 12,000 employees in the insurance industry over a 3-year period, Nyberg found the answer was: it depends. Specifically, it depends on the labor market and pay growth.

- Think g (cognitive ability) is just related to job performance? In a (albeit small) study by Judge, et al., it turns out it was also related to physical and economic well-being. Maybe their next study will address my personal hypothesis: g is related to choice of car.

- A study by Lievens, et al. (in press version here) found with a sample of 192 incumbents from 64 occupations that 25% of the variance in competency ratings (like you might find in a job analysis) was due to the nature of the rater's job, such as level of complexity. Not surprisingly, the greatest consensus was reached for jobs that involved a lot of equipment or contact with the public.

- Self-efficacy (i.e.., confidence) has been proposed as an important predictor of job performance. In a study by Schmidt & DeShon, the authors found that this relationship depends on the ambiguity present in the situation--in situations high in ambiguity, self-efficacy was negatively related to job performance; in situations low in ambiguity, the opposite was true.

- Finally, for anyone citing Ilies, et al.'s 2009 study of the relationship between personality and OCB, there have been a couple corrections.

Lewis case emphasizes need for valid tests

On Monday, May 24, the U.S. Supreme Court ruled in Lewis v. City of Chicago that plaintiffs filing an adverse impact discrimination claim under Title VII of the Civil Rights Act have 300 days to file from each time the test is used to fill a position--not just 300 days from when the test was administered.

For the most part, this mainly impacts employers that run an exam and use the results to fill positions for several years. In this case it was a firefighter entry-level exam, and my guess is it will mostly be public sector agencies and large employers that should pay particular attention to the ruling.

Why? Well for one it means more potential adverse impact lawsuits. If you were counting on being safe after 300 days from the exam, that is no longer the case. Second, it emphasizes the need to follow professional guidelines when developing an exam. Employers can successfully defend against an adverse impact case by showing that the selection practice is "job related for the position in question and consistent with business necessity..." (and that no alternatives with similar validity with less adverse impact were available). This means your exams need to be developed and interpreted by people who know what they're doing.

The City claimed that evidence related to an employer's business necessity defense might be unavailable by the time the lawsuit is brought--the court was not swayed. This means you'll want to hang on to your exam development records for at least a year beyond the last time you use the results to fill a position.

Another point worth noting: this case boils down to the validity of a cut score used by the City, which they themselves admitted wasn't supportable. Proper exam development and interpretation includes setting a job-related pass point based on subject matter input and/or statistical evidence that it is linked to job performance.

The Lewis ruling doesn't fundamentally change what we should be doing. It just emphasizes that we need to do it right.

Click here for a quick overview of the facts of the case.

June 2010 IJSA

The summer journal season continues with the June 2010 issue of the International Journal of Selection and Assessment. Take a deep breath, there's a lot of stuff packed into this issue:

- Roth et al. provide evidence that women outperformed men on work sample exams that involved social skills, writing skills, or a broad array of KSAs. To the extent that an employer is trying to avoid discriminating against female applicants, this provides support for work sample usage.

- In a study of managers in Taiwan, Tsai et al. show that the most effective way an applicant can make up for a slip in an interview is to apologize (vs. attempting to justify or use an excuse).

- Jackson et al. strive to add some clarity on task-based assessment centers

- Blickle & Schnitzler provide evidence of the construct and criterion-related validity of the political skill inventory

- Colarelli et al. studied how racial prototypicality and affirmative action policies impact hiring decisions. Results of a resume review indicated more jobs were awarded to black candidates as racial prototypicality and affirmative action policy strength increased, but stronger AA policies decreased the percentage of minority hires attributed to higher qualifications.

- In my personal favorite article of the issue, Karl et al. found in a study of U.S. and German students that those low on conscientiousness (especially), agreeableness, and emotional stability were more likely to post "Facebook Faux Pas". This provides some support for employers who screen out applicants based on inappropriate social networking posts. I'll talk more about this in my upcoming webinar.

- Denis, et al. provide support for the NEO PI-R's ability to predict job performance in two French-Canadian samples.

- Bilgiç and Acarlar report results of a study of Turkish students and perceptions of various selection instruments. Interviews were rated most highly and there were some differences in terms of privacy perceptions depending on the goal orientation of the student.

- Trying to figure out how to hire better direct support professionals (e.g., those providing long-term residential care or care to those with disabilities)? Robson, et al. describe the development of a composite predictor composed of various measures (e.g., agreeableness, numerical ability) that predicted performance, satisfaction, and turnover.

- Ahmetoglu et al. provide support for using the Fundamental Interpersonal Relationship Orientations-Behaviour (FIRO-B) to predict leadership capability.

- Ispas et al. describe results of a study that showed support for a nonverbal cognitive ability measure (the GAMA) in predicting job performance in two samples.

- Last but not least, in another win for context-specific assessments, Pace & Brannick show how a measure of openness to experience tailored to specific work outpredicted the comparable general NEO PI-R scale. IMHO this is how personality measures will eventually become more prominent and accepted as pre-hire assessments.

IPAC conference to feature Campbell, McDaniel, Highhouse, and more

Those of you on the fence about attending the 2010 International Personnel Assessment Council (IPAC) conference on July 18-21 may be interested to know that a preliminary schedule has been released that reveals some great speakers and topics. For example:

- David Campbell's provocatively titled opening session, The Use of Picture Postcards for Exploring Diversity Issues Such as Bias and Prejudices, or "How Can We Keep Our Grandchildren From Going to War With Each Other?"

- Not to be outdone, Michael McDaniel kicks things off Tuesday morning with Abolish the Uniform Guidelines.

- Scott Highhouse closes things up Wednesday with A Critical Look at Holistic Assessment

- Great pre-conference workshops on everything from job analysis to fairness

- Wonderfully diverse concurrent sessions on topics such as public service motivation, leadership coaching, simulations, engagement, online testing, charging for exams, test transportability, cross-cultural personality assessment, measuring workforce gaps, adverse impact analysis, faking and lie detection, and succession planning. And that's just a sample!

Staying current on assessment through professional education is one of the commandments of our field. I hope you'll be joining your friends and colleagues in Newport Beach. Early bird registration ends June 1st.

Personnel Psychology, Summer 2010

The latest issue of Personnel Psychology (v63, #2) marks the beginning of summer journal season. Let's take a peek at some of what's inside:

Practice makes...better. John Hausknecht studied over 15,000 candidates who applied for supervisory positions (and 357 who repeated the process) over a 4-year period with a large organization in the service industry. The selection process included a personality test. He found that candidates that failed the first time around showed practice effects on dimension-level scores of .40 to .60. Candidates that passed the first time, but were taking the test again for other reasons, generally showed no difference in scores. More interestingly, on several subscales low scores the first time around were associated with practice effects that exceeded one standard deviation. A good reminder that personality inventories are susceptible to "faking", but certainly not a nail in their coffin as they still work quite well in many situations.

Another reason to structure your interviews. As if you needed more convincing, McCarthy et al.'s study of nearly 20,000 applicants for a managerial-level position in a large organization found that the use of a structured interview resulted in zero main effects for applicant gender and race on interview performance. Similarly, there were no effects of applicant-interviewer similarity with respect to gender and race.

Users of the CRT-A take note. The conditional reasoning test of aggression (CRT-A) is used to detect individuals with a propensity for aggression. Previous studies have suggested the criterion-related validity of this test is around r=.44. In this study, by Berry et al., the authors meta-analyzed a large data set and found much lower values, in the .10-.16 range, that rose to .24-.26 when certain studies were excluded.

Assess your way into a job. Last but not least, Wanberg et al. describe the development of an inventory for job seekers called Getting Ready for your Next Job (YNJ, available here). The authors present results tying inventory components (e.g., job search intensity, Internet use) to subsequent employment outcomes.

Stay tuned, new issues of JAP, IJSA, and others should be out soon!

Grade your co-workers.com?

It was only a matter of time.

We know people like judging other people. Heck, we even like watching other people judge people. We also know people like interacting with websites rather than simply reading them. The natural result? Websites where you can judge other people.

Collective judgments are nothing new when it comes to restaurants, books, or even employers. But we're entering a new era where your online reputation may in large part be determined by other people. Assuming this trend takes off.

On one end of this spectrum, we have websites like Checkster, which is more of a reference checking or 360-degree feedback tool. It's what I would consider a "closed" system in its current iteration, because unless you're part of the process (applicant, employer, or reference-giver) you don't interact with it. The information remains relatively private and it's for a specific situation.

In the middle are sites liked LinkedIn, which allow you to "recommend" people. LinkedIn is a bit more open in that you can view someone's profile, but to see someone's recommendations, you need to be connected to them in some way, which is generally tricky for an employer unless they're already very connected. The other problem is the title--recommendations. This precludes other types of, shall we say, more constructive feedback.

On the far end of the spectrum is Unvarnished, which has recently gotten a lot of press. It's the most open system in that people's profiles are readily available (presumably; it's still in beta). Any unvarnished user can add a profile of a person to the site or comment on an already existing one. And it's all anonymous, although reviews can be rated and moderated. Finally, you can "claim" your profile and receive notification of new reviews, comment on ratings, and request reviews from specific people.

One of the big questions about this model is how accurate the information is. Are people just using this as an opportunity to get back at someone? Do they really know the person? To some these concerns are so overwhelming that they can't imagine using such a site. So it might be helpful for us to look at some recent research on a similar site, RateMyProfessors, which shares the open feel of Unvarnished.

You're probably familiar with RateMyProfessors. It's a simple way for students to provide feedback about their teachers. Teachers are rated on things like helpfulness and clarity and can provide comments as well. Those being rated can even provide responses.

Sounds like a way for failing students to rant about their professors, right? Well you might be surprised. In a new study published online, the authors looked at several hundred students and professors at the University of Wisconsin-Eau Claire. Here are some of their results:

1) Ratings were more frequently positive than negative.

2) "Popularity" (or lack thereof) of teacher was not correlated with frequency of feedback.

3) Students are not using the site to "rant" or "rave" as their primary motivation.

4) Those who posted were no different than those who hadn't in terms of GPA, year in school, or learning goal orientation. They were more likely to be male and their program was correlated with likelihood of feedback (e.g., those in the social sciences were more likely than those in the arts and humanities).

These results, if generalizable to other similar sites like Unvarnished, suggest that the results may be more accurate than we fear, and thus more useful. We know that peer reviews have at least moderate validity in terms of predicting performance. But there a still a lot of questions to be answered in terms of how the feedback is structured and how the information will be used by a potential employer.

So...might there be hope for crowdsourcing one's reputation? Or are we headed down a dangerous road? Would this make employers' lives easier--or just more confusing? Are defamation suits a possibility?

As an applicant yourself, here's something else to think about: would you rather your online reputation be determined by what an employer finds out about you while randomly surfing, or would you rather have a site where you can--at least partially--manage it?

Finally, consider this: If such a website became popular and filled with information about applicants...would you look someone up before hiring them?

Webinar on Internet snooping

Okay, that's not the title. The real title is actually much lengthier but more accurate: "They posted what? Promises and pitfalls of using social networking and other Internet sites to research job candidates." Yours truly will be presenting this webinar--IPAC's first--on June 9th.

I believe Internet snooping is one of the elephants in the room when it comes to personnel selection--most people are doing it, but we don't talk about it. The way to deal with this is to get things out in the open and provide hiring supervisors with some informed guidance rather than pretend they're not doing it or deluding ourselves into thinking that blocking these websites at work takes care of it.

I'll be mainly focusing on two points: (1) why websites like Facebook and LinkedIn hold so much promise when it comes to gathering additional data on candidates, and (2) what the drawbacks are if you're going to do this. The latter includes things like ensuring authenticity, uncovering information you wish you hadn't, and finding the information in the first place.

It's free for IPAC members and $75 for non-members that includes a membership for the rest of the year. More details are here. Hope to "see" you there!

Assessments: More than meets the eye

So I quasi-randomly completed the 'personality quiz' over at the LA Times page. I did it just for fun, but it actually did something useful with my results--tailored my news. Although the label it gave me ("hot shot") is questionable, the stories it returned were based on the pictures I selected, things like politics, vacationing in Hawaii, and spending time with family. This led to a couple thoughts:

1) Why aren't more tests like this? Those of us on the professional side of testing often forget that there are tests that people actually enjoy taking. In fact people take "personality tests" all the time, through Facebook or on random websites. It's the kind of thing people pass around via e-mail. When was the last time you looked forward to taking an employment test?

2) I wonder what theory (if any) this is based on? It uses Imagini's VisualDNA technology, but I wasn't able to determine much from their website other than it took over three years to develop. Oh, and that apparently it's used by a number of sites, including match.com and hotels.com, for marketing purposes.

Taking this quiz also made me think not only about the "fun" side of testing but about alternate uses of assessment tools. These measures don't have to be used for selecting in and out. They can be used for many purposes, including some that are obvious (development) and some that perhaps aren't, like placement.

Using assessments for placement is something career counselors do all the time, but it's relatively rare for organizations. It shouldn't be. Imagine the value of putting some of your old-but-still-good assessments on the web and allowing people to take them, get feedback about their results, and receive some information that would allow them to self select in or out of various positions. It's a tool for insight, a realistic job preview, and an efficient way to populate the top of your selection funnel--all at the same time.

But wait, there's more. Imagine if you could populate your applicant tracking system with the results of said assessments. Imagine if, at the end of the assessment(s), the results strongly indicated the individual would be a good fit for a certain type of job. You could store their results for contacting in the future, provide them with additional recruiting material, lead them to relevant vacancies, and/or encourage them to apply.

Aside from some of the bleeding edge video game-type assessments, I haven't seen any selection tests that come close to fun (yes, I know we like to think that assessment centers are "fun" for applicants but we're fooling ourselves). And I don't recall seeing anyone using tests for placement in the way I described.

Have you?

The perils of testing incumbents

Many of us plan or administer promotional tests on a daily basis. It's a normal way for the organization to figure out if people are ready for the next step--as journey person, lead, supervisor, manager, etc.

What isn't so usual is testing incumbents to determine if they can keep their jobs (drug testing aside). Amtrak, who is poised to take control of Metrolink, Southern California's commuter rail service, ran into fierce opposition recently when they announced that they will require train crews to take and pass two personality tests traditionally used only for screening applicants.

Why now? Amtrak is trying to prevent a repeat of the Chatsworth crash, where a Metrolink engineer crashed head on into a freight train, resulting in 25 dead (including the engineer) and 135 injured. Records revealed the engineer had a history of sending and receiving text messages hundred of times while operating trains (violating safety rules), including seconds before he ran a red led and crashed. Amtrak officials hope to identify "psychological issues", particularly those that manifest themselves during times of stress.

The tests in question are the Applicant Personality Inventory (API) and the Hogan Personality Inventory (HPI). The tests have been used since 2004 (API) and 2002 (HPI) to select among applicants for engineer and conductor positions, and the pass rate for the API is around 80%.

Union leaders have aggressively resisted the idea, claiming that the tests aren't valid or "relevant" measures of a trained and experienced employee's ability to safely operate trains. Instead, they claim the tests are a "witch hunt."

There are several things I find interesting about this situation:

1) Union leaders don't believe the tests are valid for incumbents, but have no problem with using them to select among potential hires. Assuming the personality aspects tested for by these measures are relatively stable, why would they be okay with testing applicants but not members? Methinks the issue is less validity and more membership.

2) If the vast majority of applicants, who are presumably a more heterogeneous group than incumbents, pass these exams, why would the union think that any incumbents would "fail" them? If it is indeed a witch hunt, what evidence do they believe management is relying upon to pre-judge certain individuals?

3) On a related note, given the high pass rates why would Metrolink/Amtrak think that any incumbents will "fail" these exams? (One wonders if the pass point would remain the same) This could be a situation where much political capital is expended with very little utility as a result of the assessments.

Incumbent testing is always a risky business. Even in the best of situations, those that fail or are passed over may harbor feelings of resentment, even anger. HR professionals must treat these selection situations with particular care and plan for how to communicate about the exam, before and after results are announced and used. But as this case demonstrates, when the tests in question are used to determine continued employment in one's current position--and the tests are personality inventories--tempers may flare particularly high.


Hat tip to my friend Warren Bobrow for this story.

April 2010 TIP and latest EEO Insight

The April 2010 issue of The Industrial/Organizational Psychologist (TIP) is now available here. Let's check out some of what's inside:

- Several great, touching dedications to the late great Frank Landy. I found Rick Jacobs' eulogy particularly moving. My first real job out of graduate school was working for Frank doing research for expert witness testimony. Fabulous experience, amazing individual.

- Joel Wiesen proposes a novel approach to promotional selection in fire department settings.

- A great little article about moving from an I/O position to an HR generalist one. Very timely given all the changes HR departments are experiencing.

- The current status of legal protection for sexual minorities in the workplace

- A great article about the Bridgeport, CT case and why both the media and the city got things wrong.

- A fascinating breakdown of the major activities of I/O consultants v. internal practitioners v. academics. Check out Table 3.


In addition, there's a new issue of EEO Insight with some great content, and one article in particular I'd like to point out: A comprehensive look at how to successfully develop diversity initiatives and testing programs post-Ricci, including the essential elements of a Croson study (see Table 2) and the "strong basis in evidence" standard as applied to several practical examples (Table 3). It's a keeper and starts on page 27.

Some other articles to check out include:

- Strategies for defending and framing the issue of adverse impact in selection

- How to avoid adverse impact when choosing a test

Some blogs to follow + FB group

Here are three blogs you have missed:

Shaker Consulting Group (makers of the Virtual Job Tryout) has a new blog out written by my friend Joe Murphy. Recent posts covered the ERE conference, the importance of data-based decision making (can't agree with that enough!), and the recruiting value proposition.

Another is I/O at work, sponsored by HR Catalyst. They do a great job summarizing recent research across a wide spectrum of topics. Recent posts covered evidence-based management, the importance of positive feedback, and what job ads say about organizational culture.

Finally, those of you that subscribe to my shared items know I'm a fan of the blog from Recruitment Directory, an Australian consulting firm. Recent topics include checking out your job site on the iPad, website security, and the Australian HR Tech Report.

On a different note, if Facebook is a big part of your life, you may want to join the HR Tests fanpage. It's a great way to get your news in your feed, learn who else reads HR Tests, and have the ability to comment, without waiting for moderation, on any post.

ClicFlic offers assessment innovation

A while back I posted about a creative use of technology that Vestas was using for onboarding. At the time I wrote about the potential I saw for the use of such technology for assessment, but actually creating these videos was a bit of a mystery from the customer side. Now I've come across a vendor that allows us to create these tools.

I don't post about specific products very often--usually I focus on research and best practices--but I have made occasional exceptions. When I see a product that I think has the potential to be innovative, highly effective, and highly valid, I want to share the wealth.

Such is the case with ClicFlic. In a nutshell, ClicFlic allows customers to create customized interactive web-based videos that can be used for things like situational judgment tests (SJTs). But we've seen that before, right? What I hadn't seen was the branching ability of ClicFlic.

Historically, video-based testing, whether Internet-enabled or not, presents all candidates with the same content. A situation is presented, and the candidate is provided with either several pre-determined responses or an open-ended response area. But much like traditional computerized adaptive testing (CAT), ClicFlic allows for the creation of branching videos. In other words, what the user sees in the next segment will vary depending on how they respond on the current one.

Although most of the examples you'll see on their website involve customer service or training applications, the technology is easily adaptable to assessment situations, as you can see from this example.

I had an opportunity to speak with Mike Russiello, President and CEO of ClicFlic (and co-founder of Brainbench) and he allowed me to peek "under the hood"--what I saw looked plug-and-play easy. The scripting branches are easy to generate, videos simple to upload, and you can quickly assign points to different responses. The videos are flash-based and you can easily generate the HTML to place it on a webpage.

Want to learn more? Check out the examples on their website--the demo on the front page will give you a good feel for the technology. Here are some others that will give you an idea of the possibilities. For assessment-specific usages, here you can select several different types of items with some characters you may recognize.

Questions? You can learn more about how the tools are built here. You may also run into Mike at SIOP if you have questions. Finally, they're also planning on an upcoming webcast through tmgov.org.

I hope this sparks some interest for you and maybe even some ideas about where this technology could be taken even further (RJPs anyone?).

This and that

I follow several journals, several of which aren't specifically devoted to recruitment and selection. But if you believe, as I do, that organizational structure and behavior have implications for what we usually talk about on this blog, I think you might find the following recently published articles interesting. I've also included a couple directly on point that you may have missed:

Got meetings? Turns out they're a key aspect of job satisfaction.

Thinking about work-life balancing measures? Consider the type of employee.

GLBT nondiscrimination policies may impact overall organizational performance.

Wrap your mind around this one: The ability to recognize opportunities may have a genetic component, similar to the personality aspect of openness to experience.

Are formal HR policies bad for morale? This study certainly suggests so. It also suggests that we need to "think small" when it comes to organizational units.

What makes someone "employable"? Willingness to change jobs--yes. Willingness to develop new competencies--not so much.

Interested in presenteeism (people coming to work sick)? Here's a good overview.

Maybe the New London police department wasn't so wacky. Turns out being overeducated negatively impacts job satisfaction--the good news is experience appears to moderate the relationship.

Bothered by the "criterion problem" in measuring the utility of assessments? This study won't make you feel any better, but it does help explain our challenge.

Want to do better on a test? Think positive.

Need more evidence that off-list checks are important? Check this out.

March 2010 J.A.P.

The March 2010 issue of the Journal of Applied Psychology is out. Let's take a look:

Do women make better leaders? According to a study by Rosette and Tost, it varies with how success is attributed, the level of the position, perceptions of double-standards, and expectations. So the answer? A very solid "it depends."

Who should determine SJT scoring? Motowidlo and Beier suggest in their research study that situational judgment test (SJT) scoring keys based on input from subject matter experts (SMEs) contribute differentially to the prediction of job performance compared to keys based on general knowledge about trait effectiveness. What does this mean? That your ability to predict performance using SJTs depends in part on who is determining the scoring, and getting SME input may boost the effectiveness.

Do Americans work to live or live to work? Based on an analysis from Highhouse, et al., it's looking more and more like the former.

Need more evidence of the value of confirmatory testing? Naquin, et al. performed three experimental studies that demonstrated higher levels of lying when using email compared to pen and paper.

Do you like your leaders proactive? According to research conducted in China by Ning et al., you're not alone.

Finally, a slight correction to an article by Ilies et al. published last July on the relationship between personality and organizational citizenship behaviors.

2010 PTC -NC Conference

Last week I was fortunate to attend and present at the 2010 Personnel Testing Council of Northern California (PTC-NC) conference. Several of the presentation slides are now available.

My presentation was a legal update, primarily focusing on last year's big case, Ricci v. DeStefano. While I think the case received a fair share of its publicity simply because Sonia Sotomayor was one of the circuit court judges who ruled for the city, the case itself has some interesting implications for assessment. I gave my two cents last year after the decision.

Some of the points I made during the presentation included:

- Test validation standards as judged by the courts are generally very attainable. Following best practice (i.e., beginning with a thorough job analysis) is a recipe for a defensible process.

- Employers should spend the vast majority of their time before the assessment is given, figuring out what and how to test. Minimal time should be spent after the test making decisions about test usage--you should know that already.

- Employers, and assessment professionals, are expected to be familiar with and consider a wide range of testing mechanisms when planning a selection process. This includes non-cognitive assessments such as situational judgment tests, personality inventories, and biodata measures.

There were many great presentations; I always enjoy hearing what Wayne Cascio has to say, Dale Glaser has a way with statistics, and Deniz Ones and Stephan Dilchert's presentation on personality profiles of leaders was fascinating (they also happen to be very pleasant people to have lunch with). I plan on printing out slide 33 and placing it within reach--it does a great job of pointing out that criterion validity depends greatly on what you're trying to predict!

It's worth reviewing all the slides to get a flavor of what was discussed; there will likely be other presentations added over time.


On a side note, I'd like to acknowledge my readers at Baruch College -- thanks for reading!

Hiring Right Presentation

Late last year I did a webinar through BCGi titled "Hiring right: How to hire the right person for the job" and the slides are now available.

For many of you it will be review, but I tried to cover a variety of topics relevant for newer professionals as well as hiring supervisors, including:

- accuracy of our perceptions regarding hiring

- reviewing applications and resumes

- types of assessment

- interview questions

- best practices

You can see the PDF version here; content begins on page 5.

Personnel Psychology, Spring 2010: SJTs, affect, and job offer timing

The Spring 2010 (v.63, #1) issue of Personnel Psychology is out. Let's look at the highlights:

First out of the gate, a great meta-analysis for anyone interested in situational judgment tests (SJTs; and who isn't?). Christian, et al. looked at 84 studies and found some pretty interesting things:

1) SJTs reported in the literature have been used to measure a variety of things, including leadership skills (37%), some type of composite (33%), interpersonal skills (12.5%), personality tendencies (9.6%), teamwork skills (4.4%) and job knowledge/skills (3%).

2) Criterion-related validity depends--as you might expect--on the match between predictor and performance measure. Conscientiousness measures, for example, predicted task performance much better than managerial performance (rho=.39 and .06 respectively). The highest correlations (albeit based on relatively small samples) were for teamwork skills and personality composites predicting task performance (.50 and .45 respectively).

3) Video-based SJTs tended to have stronger criterion-related validity values compared to paper-based measures. This was particularly true when measuring interpersonal skills (.47 compared to .27).

Second, a small but interesting study by Johnson, et al. on the relationship between trait affect (i.e., being generally disposed to feeling positive or negative emotions) and job performance. Results from 120 matched employee-supervisor pairs from a variety of jobs using both explicit (survey) and implicit (word fragment completion) measures of affect found substantial correlations, particularly between positive affect and performance (in the .50 range), and particularly when using implicit measures.

Something to add to a selection battery, perhaps? Could be perceived negatively by applicants, however, and I can see some questions being raised about the link to medical issues. But the same types of concerns were originally leveled at personality tests and were mitigated by creating measures specifically tied to work behavior. Definitely an area for more research.

Third, check out this study by Becker, et al. on the impact that job offer timing has on acceptance, performance and turnover. The authors found (using data from a Fortune 500 engineering technology company) that for both student and experienced samples, faster offers were associated with higher acceptance rates. Specifically, for experienced candidates, the difference between 2 weeks and 3 weeks taken to make the offer was substantial, whereas for the students 3 weeks versus 4 weeks was important. But, no differences were found in terms of either performance ratings or turnover among employees hired through different offer speeds.

Implication? The study suggests that offer time does impact the likelihood that the offer will be accepted, but viewed broadly this may not have long-term impacts in terms of how employees do on the job. Maybe in cases of good candidate-employer fit, candidates are willing to wait.

Last but not least are the book reviews. Two books are particularly relevant for us, The Structured Interview (Pettersen & Durivage) and Outliers (Gladwell). The first is received very positively and sounds like a great source for anyone wanting more details about the support for and use of structured interviews. The latter is "well worth [a] few evenings" but requires you to overlook the lack of evidence and convenient inferences.

Final notes: those of you interested in multisource performance ratings should check out Hoffman, et al.'s article, which reinforces the impact of having raters from different levels. Chuang and Liao's article also includes a useful measure of a high-performance work system.

IPAC Conference + Innovations Award

What are you doing July 18-21? I assume if you enjoy good weather, good company, and--most importantly--great information on state-of-the-art selection practices, you'll be joining me at IPAC's annual conference in beautiful Newport Beach, California.

If you're not only going but have something to present, by all means respond to the call for proposals. It could be a workshop, panel discussion, symposia--pretty much any format you can think of. Don't wait too long, the deadline is this Friday, March 5.

And speaking of the conference, IPAC has announced that nominations for the Innovations in Assessment Award are being accepted from 5/17-6/18. The winner receives not only formal recognition (and bragging rights), but a free pass to the conference.

IPAC's a great group, full of people that are extremely knowledgeable and passionate about using the best selection practices to get organizations the talent they need. Plus, it's the only international (or national) organization I know of devoted exclusively to the topic.

Hope to see you there!

Book review: Strategy-Driven Talent Management

A thought-provoking collection of essays and ideas; but it won't solve all our problems.

The value of a book lies as much with the reader as it does with the content. A book about advanced programming does little good to the person who has problems turning a computer on. A collection of cooking recipes is largely useless to someone who exclusively uses a microwave.

The same is true about business and HR books. Depending upon who you are and where you're at in life, some books may help you, some may be beyond your reach. Such is the case with SIOP's latest entry into its Professional Practice series, Strategy-Driven Talent Management: A Leadership Imperative, edited by Bob Silzer and Ben Dowell.

The book (tome, actually, at nearly 900 pages) is full of thought-provoking pieces from a variety of authors, including some familiar faces such as John Boudreau and Allan Church. There are academics present, but the majority of authors are practitioners in private sector organizations, such as Aon, Ingersoll Rand, HP, Sara Lee, Merck, and Bristol-Myers Squibb.

The book is roughly broken up by major topic area, although the distinction can be hard to maintain. There are chapters on recruitment, executive onboarding, engagement, measurement, and global issues. There's even a 40-page annotated bibliography. But the editors do an admirable job of keeping the topics all related to the broad field of talent management (TM), which they define as, "an integrated set of processes, programs, and cultural norms in an organization designed and implemented to attract, develop, deploy, and retain talent to achieve strategic objectives and meet future business needs" (p. 18).

The book is described as a "comprehensive [collection of] state-of-the-art ideas, best practices, and guidance." It shines on the first two but failed me on the last, although not for lack of trying. The problem is the book is so long, full of so many ideas and case studies, that it's very easy to get lost and not come away with any clear guidance based on the consensus of authors. To some extent this is endemic in any collection of works by separate authors, but it's clearly a collection of "what you might do" rather than a solid prescription for "how to", although some authors do a better job than others.

Another problem is that many authors seem to presume that current TM practices are sub-optimized because they aren't linked to business strategy and results. This may be true if the process is based on non-validated assumptions, but as long as there is a link between job success and specific practices, we're already there. We just haven't made a particularly good link between job success and organizational success, which may explain the attraction to concepts like competencies (mentioned many times in the book).

But my main problem, and this goes for the book as well as the field, is that it treats the concept of talent management as a logical process to be managed. Somewhere in the transition from HR to TM, we lost the H--human. Talent management (and HR) is messy because it involves people. It's political. It changes every day. And you're dealing with emotions, not lines of code. The real challenge--which is discussed but to my mind not driven home--is how to get the talent mindset into the organizational DNA.

There is value to thinking broadly and philosophically about the topic. It helps us plan. But what people really need are concrete suggestions for establishing a self-sustaining high-performance system. In order to do this, we must address the fundamentals (the basic needs of Maslow's hierarchy, if you will), such as:

- HR must learn "the business" and stay close to their customers
- Supervisors must be selected and trained with their talent management role at the forefront
- Success in HR must be defined and measured. It must be communicated, understood, and valued
- Sustained attention to HR success and significant resources must be expended by both HR and line managers

The book does a passable job of presenting these, but you may have to dig for them. The bigger problem is that there seems to be an assumption that what keeps organizations from having a top-notch TM system is a lack of understanding, either of the organizational strategy or best practices in TM, rather than the very real daily troubles that organizations experience, such as:

- Supervisors that hire people they know/like rather than the most qualified person
- People placed into HR with little or no background, interest, or passion for it
- Insufficient resources devoted to TM/HR
- HR managers who are just that--managers--rather than real HR leaders (Avedon and Scholes present a great assessment in Chapter 2 that helps separate these)

Until organizations have these types of "minor"--but real--flaws ironed out, all the charts and good intentions in the world will have very little impact.

Finally, I was also disappointed that there wasn't more in here about evidence-based TM and HR (which may say more about the field than the authors/editors, who acknowledge this lack in Chapter 22). The field desperately needs more research to tie the hard science of assessment with the more anecdotal/consultant practices such as recruiting, retention, and performance management. This will require significantly more research using methods beyond surveys in order to show what works and what doesn't. There are some ties to good research in here, but the hole is significant.


To summarize, the book contains a lot to like, particularly for individuals already schooled in this area looking to optimize their shop, or for graduate students seeking to understand the big picture. But for most HR practitioners (and, I would expect, executives), this book is akin to a collection of recipes for advanced Italian cooking--fabulous for those used to making their own pasta, but beyond the reach of those struggling to make their own sauce.

Webinar on 21st Century Assessment

Went to a pretty darn good webinar yesterday put on by HCI and featuring Ken Lahti (PreVisor) and Charles Handler (Rocket-Hire). The topic was 21st century assessment.

Some of the topics covered included:

- increased functionality and usability of testing platforms

- increased sophistication of security methods

- off-the-shelf tests and "I/O psychologists in a box"

- integrating assessment with your overall talent strategy

And my two favorites:

- advanced simulations (such as those using video game technology)

- candidate data that follows them

The webinar is going to be re-broadcast several times today and tomorrow, if you have a chance check it out. You can also see a copy of the slides for free if you're an HCI member (which is free).

Free, short, and full of information--that's my kind of training.

Recruitment v. Assessment, Round 1: Fight!

I'm sure some of you are avid readers of ERE (Electronic Recruiters Exchange), but for those of you that aren't (and don't receive my shared items), there's a lively discussion going on over there regarding the practical value of recruitment versus assessment practices.

It started with Wendell Williams' first post on how to identify a bad test (the second part is also worth reading). The comments begin relatively benignly, debating the strength of various predictors of performance (e.g., P-O fit versus behavioral interviews), but turns into, let's say, a lively debate that includes a discussion of the Gallup 12, the limitation of assessments, the Uniform Guidelines, and a lot more. The most heated exchange occurs between Wendell and Lou Adler, where accusations and sarcasm fly.

Speaking of Lou, he continues to advocate his perspective with his next post on whether increasing interview accuracy increases quality of hire (yes, he's suggesting that's an open question). While the comments following are fewer in number, the debate continues regarding the value of assessment and the evidence used to support it (e.g., Schmidt & Hunter's 1998 piece).

Who said HR is boring?

Latest IJSA: Emotional intelligence, multiple-choice formats, and lots more

The March 2010 issue of the International Journal of Selection and Assessment (IJSA) is out, and the research covers a wide variety of recruitment and assessment topics as well as being truly international:

Unproctored internet-based testing (UIT) response distortion may be less than we fear (sample included cognitive and personality measures)

What factors are most important to organizations when choosing a test? This study suggests applicant reaction, cost, and diffusion of the test type in the field.

Personality (esp. core self evaluation) is related to the type of work preferred, and hence P-O fit

Career site features may differentially attract men and women

Corporate images do matter when it comes to organizational attractiveness

Who uses job-search websites and how to improve them (the sites, not the people)

Support for performance-based (as opposed to self-report) measures of emotional intelligence

Work samples, interviews, and ability tests perceived best by employees (why? because they work, say the participants)

...and last but definitely not least:

A "2 of 5" multiple-choice format seems superior than traditional "1 of 6" (you just have to make sure you can score them that way!)

Feds new jobs site is Googlish

The U.S. Government has revamped its jobs page, www.usajobs.gov, and in the process shown everyone else how its done.

Take a look at their old site. Not horrible, but cluttered with lots of features that distracted from the main reason people visit the site: to look for a job.


This picture actually doesn't do it (in)justice; there was additional content below the bar.

Now look at their new website:


This new website is what I would call "Googlish": simple, lots of white space, no scrolling required, and a single search box. The design focuses less on being pretty, and more on being functional. If you're interested in learning more about careers, or if you'd like information related to specific groups, like veterans or those with disabilities, its still there. And there's even more functionality up top in the form of drop-down menus.

Job seekers don't need a magazine ad. They need to quickly and easily find information. And this new website fits the bill.

How does yours compare?

Lessons from NYC Fire case - part 2

Part 2 of 2

Last time I discussed five important lessons we can take away from recent rulings in the Vulcan v. City of New York case. In this post I'll review the remaining lessons and also discuss the relief order.

----

6) The city failed to provide sufficient evidence that the exam(s) tested for a sufficient number of the critical KSAs. They also failed to explain why they chose not to measure several KSAs identified as critical.

Lesson: the courts do not require employers to measure every single critical KSA. But there is an expectation that employers attempt to measure a sufficient number that represent a significant portion of the job requirements. In this case, that included non-cognitive abilities such as resistance to stress, teamwork, and conscientiousness, that were not measured.

7) The city failed to adequately consider how to measure a significant number of essential KSAs. While some of their concerns were valid (e.g., structured interviews for all applicants would be an operational nightmare), there are many different forms of testing that should have been considered, including situational judgment tests (SJTs) and biodata, which can be used to measure non-cognitive components.

Lesson: triers of fact expect employers to be up on the various assessment methods available and be able to explain why they chose not to use certain ones. This includes tests that are relatively easy to develop (e.g., SJTs) as well as ones that require substantial resources and statistical expertise (e.g., biodata).

8) The city failed to conduct a reading level analysis on the exams to ensure that it was not "pointlessly high." The plaintiff introduced evidence suggesting the reading level was above 12th grade; in addition, it appeared to exceed the reading level of materials at the academy.

Lesson: never forget that every assessment method is in some sense measuring additional KSAs beyond those you intend. For written exams, reading comprehension is always a requirement (barring accommodation). It's quite easy to conduct a reading level analysis (MS Word has it built in) to ensure that the level is reasonable and matches other job-related material.

9) The city failed to show that the cutoff scores (pass points) established for the exams were based on adequate rationale, namely "the necessary qualifications for the job of entry-level firefighter." Instead, the cutoff scores were based on operational need (the number of job openings expected). This is particularly important in multiple-hurdle selection processes such as in this case, where a failure on one exam component precludes an applicant from participating in the rest of the (potentially compensatory) assessment process.

Lesson: ultimately applicants have to pass the test(s) to be considered for employment. Cutoff scores should be established using the expertise of both SMEs and test developers and should be based on the minimum competency levels required upon entry to the job. At a minimum (and I would not rely solely upon this), the scores should be analyzed to identify any logical "break-points."

----

After ruling for the plaintiffs on both the adverse impact and disparate treatment claims, the judge issued a relief order on 1/10/10. In it, he imposes several things, including the following:

1) The city must develop a new testing procedure for entry-level firefighter in conjunction with the relevant parties. Following the development of the test, there will be a hearing to determine if this test should be used rather than the current test (developed in 2007 and not at issue in this litigation).

2) The court shall develop a process by which the approximately 7,400 applicants covered by this case can file a claim for monetary relief.

3) The city will identify 293 black candidates on the eligibility list and offer them priority hiring. (No quotas are being imposed, although the judge leaves this possibility open)

4) Retroactive seniority for those hired.

In addition, several other issues are up for debate, including the appointment of a special master or monitor, standards that will be relied upon in constructing the new exam, and the need for additional relief.

---

So what did we learn from all this? If you follow--fairly closely--best practices when developing and administering exams, you will be on solid ground defending them. If you don't, and your exam has a discriminatory effect, you may be called on it--and it's not a pleasant process. I'll leave you with this quote from the January ruling on disparate treatment:

"The history of the City's efforts to remedy its discriminatory firefighter hiring policies can be summarized as follows: 34 years of intransigence and deliberate indifference, bookeneded by identical judicial declarations that the City's hiring policies are illegal."