Friday, May 14, 2010

Personnel Psychology, Summer 2010


The latest issue of Personnel Psychology (v63, #2) marks the beginning of summer journal season. Let's take a peek at some of what's inside:

Practice makes...better. John Hausknecht studied over 15,000 candidates who applied for supervisory positions (and 357 who repeated the process) over a 4-year period with a large organization in the service industry. The selection process included a personality test. He found that candidates that failed the first time around showed practice effects on dimension-level scores of .40 to .60. Candidates that passed the first time, but were taking the test again for other reasons, generally showed no difference in scores. More interestingly, on several subscales low scores the first time around were associated with practice effects that exceeded one standard deviation. A good reminder that personality inventories are susceptible to "faking", but certainly not a nail in their coffin as they still work quite well in many situations.

Another reason to structure your interviews.
As if you needed more convincing, McCarthy et al.'s study of nearly 20,000 applicants for a managerial-level position in a large organization found that the use of a structured interview resulted in zero main effects for applicant gender and race on interview performance. Similarly, there were no effects of applicant-interviewer similarity with respect to gender and race.

Users of the CRT-A take note. The conditional reasoning test of aggression (CRT-A) is used to detect individuals with a propensity for aggression. Previous studies have suggested the criterion-related validity of this test is around r=.44. In this study, by Berry et al., the authors meta-analyzed a large data set and found much lower values, in the .10-.16 range, that rose to .24-.26 when certain studies were excluded.

Assess your way into a job. Last but not least, Wanberg et al. describe the development of an inventory for job seekers called Getting Ready for your Next Job (YNJ, available here). The authors present results tying inventory components (e.g., job search intensity, Internet use) to subsequent employment outcomes.

Stay tuned, new issues of JAP, IJSA, and others should be out soon!

Wednesday, May 05, 2010

Grade your co-workers.com?


It was only a matter of time.

We know people like judging other people. Heck, we even like watching other people judge people. We also know people like interacting with websites rather than simply reading them. The natural result? Websites where you can judge other people.

Collective judgments are nothing new when it comes to restaurants, books, or even employers. But we're entering a new era where your online reputation may in large part be determined by other people. Assuming this trend takes off.

On one end of this spectrum, we have websites like Checkster, which is more of a reference checking or 360-degree feedback tool. It's what I would consider a "closed" system in its current iteration, because unless you're part of the process (applicant, employer, or reference-giver) you don't interact with it. The information remains relatively private and it's for a specific situation.

In the middle are sites liked LinkedIn, which allow you to "recommend" people. LinkedIn is a bit more open in that you can view someone's profile, but to see someone's recommendations, you need to be connected to them in some way, which is generally tricky for an employer unless they're already very connected. The other problem is the title--recommendations. This precludes other types of, shall we say, more constructive feedback.

On the far end of the spectrum is Unvarnished, which has recently gotten a lot of press. It's the most open system in that people's profiles are readily available (presumably; it's still in beta). Any unvarnished user can add a profile of a person to the site or comment on an already existing one. And it's all anonymous, although reviews can be rated and moderated. Finally, you can "claim" your profile and receive notification of new reviews, comment on ratings, and request reviews from specific people.

One of the big questions about this model is how accurate the information is. Are people just using this as an opportunity to get back at someone? Do they really know the person? To some these concerns are so overwhelming that they can't imagine using such a site. So it might be helpful for us to look at some recent research on a similar site, RateMyProfessors, which shares the open feel of Unvarnished.

You're probably familiar with RateMyProfessors. It's a simple way for students to provide feedback about their teachers. Teachers are rated on things like helpfulness and clarity and can provide comments as well. Those being rated can even provide responses.

Sounds like a way for failing students to rant about their professors, right? Well you might be surprised. In a new study published online, the authors looked at several hundred students and professors at the University of Wisconsin-Eau Claire. Here are some of their results:

1) Ratings were more frequently positive than negative.

2) "Popularity" (or lack thereof) of teacher was not correlated with frequency of feedback.

3) Students are not using the site to "rant" or "rave" as their primary motivation.

4) Those who posted were no different than those who hadn't in terms of GPA, year in school, or learning goal orientation. They were more likely to be male and their program was correlated with likelihood of feedback (e.g., those in the social sciences were more likely than those in the arts and humanities).

These results, if generalizable to other similar sites like Unvarnished, suggest that the results may be more accurate than we fear, and thus more useful. We know that peer reviews have at least moderate validity in terms of predicting performance. But there a still a lot of questions to be answered in terms of how the feedback is structured and how the information will be used by a potential employer.

So...might there be hope for crowdsourcing one's reputation? Or are we headed down a dangerous road? Would this make employers' lives easier--or just more confusing? Are defamation suits a possibility?

As an applicant yourself, here's something else to think about: would you rather your online reputation be determined by what an employer finds out about you while randomly surfing, or would you rather have a site where you can--at least partially--manage it?

Finally, consider this: If such a website became popular and filled with information about applicants...would you look someone up before hiring them?

Saturday, May 01, 2010

Webinar on Internet snooping


Okay, that's not the title. The real title is actually much lengthier but more accurate: "They posted what? Promises and pitfalls of using social networking and other Internet sites to research job candidates." Yours truly will be presenting this webinar--IPAC's first--on June 9th.

I believe Internet snooping is one of the elephants in the room when it comes to personnel selection--most people are doing it, but we don't talk about it. The way to deal with this is to get things out in the open and provide hiring supervisors with some informed guidance rather than pretend they're not doing it or deluding ourselves into thinking that blocking these websites at work takes care of it.

I'll be mainly focusing on two points: (1) why websites like Facebook and LinkedIn hold so much promise when it comes to gathering additional data on candidates, and (2) what the drawbacks are if you're going to do this. The latter includes things like ensuring authenticity, uncovering information you wish you hadn't, and finding the information in the first place.

It's free for IPAC members and $75 for non-members that includes a membership for the rest of the year. More details are here. Hope to "see" you there!

Tuesday, April 27, 2010

Assessments: More than meets the eye


So I quasi-randomly completed the 'personality quiz' over at the LA Times page. I did it just for fun, but it actually did something useful with my results--tailored my news. Although the label it gave me ("hot shot") is questionable, the stories it returned were based on the pictures I selected, things like politics, vacationing in Hawaii, and spending time with family. This led to a couple thoughts:

1) Why aren't more tests like this? Those of us on the professional side of testing often forget that there are tests that people actually enjoy taking. In fact people take "personality tests" all the time, through Facebook or on random websites. It's the kind of thing people pass around via e-mail. When was the last time you looked forward to taking an employment test?

2) I wonder what theory (if any) this is based on? It uses Imagini's VisualDNA technology, but I wasn't able to determine much from their website other than it took over three years to develop. Oh, and that apparently it's used by a number of sites, including match.com and hotels.com, for marketing purposes.

Taking this quiz also made me think not only about the "fun" side of testing but about alternate uses of assessment tools. These measures don't have to be used for selecting in and out. They can be used for many purposes, including some that are obvious (development) and some that perhaps aren't, like placement.

Using assessments for placement is something career counselors do all the time, but it's relatively rare for organizations. It shouldn't be. Imagine the value of putting some of your old-but-still-good assessments on the web and allowing people to take them, get feedback about their results, and receive some information that would allow them to self select in or out of various positions. It's a tool for insight, a realistic job preview, and an efficient way to populate the top of your selection funnel--all at the same time.

But wait, there's more. Imagine if you could populate your applicant tracking system with the results of said assessments. Imagine if, at the end of the assessment(s), the results strongly indicated the individual would be a good fit for a certain type of job. You could store their results for contacting in the future, provide them with additional recruiting material, lead them to relevant vacancies, and/or encourage them to apply.

Aside from some of the bleeding edge video game-type assessments, I haven't seen any selection tests that come close to fun (yes, I know we like to think that assessment centers are "fun" for applicants but we're fooling ourselves). And I don't recall seeing anyone using tests for placement in the way I described.

Have you?

Thursday, April 15, 2010

The perils of testing incumbents


Many of us plan or administer promotional tests on a daily basis. It's a normal way for the organization to figure out if people are ready for the next step--as journey person, lead, supervisor, manager, etc.

What isn't so usual is testing incumbents to determine if they can keep their jobs (drug testing aside). Amtrak, who is poised to take control of Metrolink, Southern California's commuter rail service, ran into fierce opposition recently when they announced that they will require train crews to take and pass two personality tests traditionally used only for screening applicants.

Why now? Amtrak is trying to prevent a repeat of the Chatsworth crash, where a Metrolink engineer crashed head on into a freight train, resulting in 25 dead (including the engineer) and 135 injured. Records revealed the engineer had a history of sending and receiving text messages hundred of times while operating trains (violating safety rules), including seconds before he ran a red led and crashed. Amtrak officials hope to identify "psychological issues", particularly those that manifest themselves during times of stress.

The tests in question are the Applicant Personality Inventory (API) and the Hogan Personality Inventory (HPI). The tests have been used since 2004 (API) and 2002 (HPI) to select among applicants for engineer and conductor positions, and the pass rate for the API is around 80%.

Union leaders have aggressively resisted the idea, claiming that the tests aren't valid or "relevant" measures of a trained and experienced employee's ability to safely operate trains. Instead, they claim the tests are a "witch hunt."

There are several things I find interesting about this situation:

1) Union leaders don't believe the tests are valid for incumbents, but have no problem with using them to select among potential hires. Assuming the personality aspects tested for by these measures are relatively stable, why would they be okay with testing applicants but not members? Methinks the issue is less validity and more membership.

2) If the vast majority of applicants, who are presumably a more heterogeneous group than incumbents, pass these exams, why would the union think that any incumbents would "fail" them? If it is indeed a witch hunt, what evidence do they believe management is relying upon to pre-judge certain individuals?

3) On a related note, given the high pass rates why would Metrolink/Amtrak think that any incumbents will "fail" these exams? (One wonders if the pass point would remain the same) This could be a situation where much political capital is expended with very little utility as a result of the assessments.

Incumbent testing is always a risky business. Even in the best of situations, those that fail or are passed over may harbor feelings of resentment, even anger. HR professionals must treat these selection situations with particular care and plan for how to communicate about the exam, before and after results are announced and used. But as this case demonstrates, when the tests in question are used to determine continued employment in one's current position--and the tests are personality inventories--tempers may flare particularly high.


Hat tip to my friend Warren Bobrow for this story.

Saturday, April 10, 2010

April 2010 TIP and latest EEO Insight

The April 2010 issue of The Industrial/Organizational Psychologist (TIP) is now available here. Let's check out some of what's inside:

- Several great, touching dedications to the late great Frank Landy. I found Rick Jacobs' eulogy particularly moving. My first real job out of graduate school was working for Frank doing research for expert witness testimony. Fabulous experience, amazing individual.

- Joel Wiesen proposes a novel approach to promotional selection in fire department settings.

- A great little article about moving from an I/O position to an HR generalist one. Very timely given all the changes HR departments are experiencing.

- The current status of legal protection for sexual minorities in the workplace

- A great article about the Bridgeport, CT case and why both the media and the city got things wrong.

- A fascinating breakdown of the major activities of I/O consultants v. internal practitioners v. academics. Check out Table 3.


In addition, there's a new issue of EEO Insight with some great content, and one article in particular I'd like to point out: A comprehensive look at how to successfully develop diversity initiatives and testing programs post-Ricci, including the essential elements of a Croson study (see Table 2) and the "strong basis in evidence" standard as applied to several practical examples (Table 3). It's a keeper and starts on page 27.

Some other articles to check out include:

- Strategies for defending and framing the issue of adverse impact in selection

- How to avoid adverse impact when choosing a test

Wednesday, April 07, 2010

Some blogs to follow + FB group

Here are three blogs you have missed:

Shaker Consulting Group (makers of the Virtual Job Tryout) has a new blog out written by my friend Joe Murphy. Recent posts covered the ERE conference, the importance of data-based decision making (can't agree with that enough!), and the recruiting value proposition.

Another is I/O at work, sponsored by HR Catalyst. They do a great job summarizing recent research across a wide spectrum of topics. Recent posts covered evidence-based management, the importance of positive feedback, and what job ads say about organizational culture.

Finally, those of you that subscribe to my shared items know I'm a fan of the blog from Recruitment Directory, an Australian consulting firm. Recent topics include checking out your job site on the iPad, website security, and the Australian HR Tech Report.

On a different note, if Facebook is a big part of your life, you may want to join the HR Tests fanpage. It's a great way to get your news in your feed, learn who else reads HR Tests, and have the ability to comment, without waiting for moderation, on any post.

Saturday, April 03, 2010

ClicFlic offers assessment innovation


A while back I posted about a creative use of technology that Vestas was using for onboarding. At the time I wrote about the potential I saw for the use of such technology for assessment, but actually creating these videos was a bit of a mystery from the customer side. Now I've come across a vendor that allows us to create these tools.

I don't post about specific products very often--usually I focus on research and best practices--but I have made occasional exceptions. When I see a product that I think has the potential to be innovative, highly effective, and highly valid, I want to share the wealth.

Such is the case with ClicFlic. In a nutshell, ClicFlic allows customers to create customized interactive web-based videos that can be used for things like situational judgment tests (SJTs). But we've seen that before, right? What I hadn't seen was the branching ability of ClicFlic.

Historically, video-based testing, whether Internet-enabled or not, presents all candidates with the same content. A situation is presented, and the candidate is provided with either several pre-determined responses or an open-ended response area. But much like traditional computerized adaptive testing (CAT), ClicFlic allows for the creation of branching videos. In other words, what the user sees in the next segment will vary depending on how they respond on the current one.

Although most of the examples you'll see on their website involve customer service or training applications, the technology is easily adaptable to assessment situations, as you can see from this example.

I had an opportunity to speak with Mike Russiello, President and CEO of ClicFlic (and co-founder of Brainbench) and he allowed me to peek "under the hood"--what I saw looked plug-and-play easy. The scripting branches are easy to generate, videos simple to upload, and you can quickly assign points to different responses. The videos are flash-based and you can easily generate the HTML to place it on a webpage.

Want to learn more? Check out the examples on their website--the demo on the front page will give you a good feel for the technology. Here are some others that will give you an idea of the possibilities. For assessment-specific usages, here you can select several different types of items with some characters you may recognize.

Questions? You can learn more about how the tools are built here. You may also run into Mike at SIOP if you have questions. Finally, they're also planning on an upcoming webcast through tmgov.org.

I hope this sparks some interest for you and maybe even some ideas about where this technology could be taken even further (RJPs anyone?).

Wednesday, March 31, 2010

This and that

I follow several journals, several of which aren't specifically devoted to recruitment and selection. But if you believe, as I do, that organizational structure and behavior have implications for what we usually talk about on this blog, I think you might find the following recently published articles interesting. I've also included a couple directly on point that you may have missed:

Got meetings? Turns out they're a key aspect of job satisfaction.

Thinking about work-life balancing measures? Consider the type of employee.

GLBT nondiscrimination policies may impact overall organizational performance.

Wrap your mind around this one: The ability to recognize opportunities may have a genetic component, similar to the personality aspect of openness to experience.

Are formal HR policies bad for morale? This study certainly suggests so. It also suggests that we need to "think small" when it comes to organizational units.

What makes someone "employable"? Willingness to change jobs--yes. Willingness to develop new competencies--not so much.

Interested in presenteeism (people coming to work sick)? Here's a good overview.

Maybe the New London police department wasn't so wacky. Turns out being overeducated negatively impacts job satisfaction--the good news is experience appears to moderate the relationship.

Bothered by the "criterion problem" in measuring the utility of assessments? This study won't make you feel any better, but it does help explain our challenge.

Want to do better on a test? Think positive.

Need more evidence that off-list checks are important? Check this out.

Saturday, March 20, 2010

March 2010 J.A.P.


The March 2010 issue of the Journal of Applied Psychology is out. Let's take a look:

Do women make better leaders? According to a study by Rosette and Tost, it varies with how success is attributed, the level of the position, perceptions of double-standards, and expectations. So the answer? A very solid "it depends."

Who should determine SJT scoring? Motowidlo and Beier suggest in their research study that situational judgment test (SJT) scoring keys based on input from subject matter experts (SMEs) contribute differentially to the prediction of job performance compared to keys based on general knowledge about trait effectiveness. What does this mean? That your ability to predict performance using SJTs depends in part on who is determining the scoring, and getting SME input may boost the effectiveness.

Do Americans work to live or live to work? Based on an analysis from Highhouse, et al., it's looking more and more like the former.

Need more evidence of the value of confirmatory testing? Naquin, et al. performed three experimental studies that demonstrated higher levels of lying when using email compared to pen and paper.

Do you like your leaders proactive? According to research conducted in China by Ning et al., you're not alone.

Finally, a slight correction to an article by Ilies et al. published last July on the relationship between personality and organizational citizenship behaviors.

Wednesday, March 17, 2010

2010 PTC -NC Conference

Last week I was fortunate to attend and present at the 2010 Personnel Testing Council of Northern California (PTC-NC) conference. Several of the presentation slides are now available.

My presentation was a legal update, primarily focusing on last year's big case, Ricci v. DeStefano. While I think the case received a fair share of its publicity simply because Sonia Sotomayor was one of the circuit court judges who ruled for the city, the case itself has some interesting implications for assessment. I gave my two cents last year after the decision.

Some of the points I made during the presentation included:

- Test validation standards as judged by the courts are generally very attainable. Following best practice (i.e., beginning with a thorough job analysis) is a recipe for a defensible process.

- Employers should spend the vast majority of their time before the assessment is given, figuring out what and how to test. Minimal time should be spent after the test making decisions about test usage--you should know that already.

- Employers, and assessment professionals, are expected to be familiar with and consider a wide range of testing mechanisms when planning a selection process. This includes non-cognitive assessments such as situational judgment tests, personality inventories, and biodata measures.

There were many great presentations; I always enjoy hearing what Wayne Cascio has to say, Dale Glaser has a way with statistics, and Deniz Ones and Stephan Dilchert's presentation on personality profiles of leaders was fascinating (they also happen to be very pleasant people to have lunch with). I plan on printing out slide 33 and placing it within reach--it does a great job of pointing out that criterion validity depends greatly on what you're trying to predict!

It's worth reviewing all the slides to get a flavor of what was discussed; there will likely be other presentations added over time.


On a side note, I'd like to acknowledge my readers at Baruch College -- thanks for reading!

Sunday, March 14, 2010

Hiring Right Presentation


Late last year I did a webinar through BCGi titled "Hiring right: How to hire the right person for the job" and the slides are now available.

For many of you it will be review, but I tried to cover a variety of topics relevant for newer professionals as well as hiring supervisors, including:

- accuracy of our perceptions regarding hiring

- reviewing applications and resumes

- types of assessment

- interview questions

- best practices

You can see the PDF version here; content begins on page 5.

Wednesday, March 03, 2010

Personnel Psychology, Spring 2010: SJTs, affect, and job offer timing


The Spring 2010 (v.63, #1) issue of Personnel Psychology is out. Let's look at the highlights:

First out of the gate, a great meta-analysis for anyone interested in situational judgment tests (SJTs; and who isn't?). Christian, et al. looked at 84 studies and found some pretty interesting things:

1) SJTs reported in the literature have been used to measure a variety of things, including leadership skills (37%), some type of composite (33%), interpersonal skills (12.5%), personality tendencies (9.6%), teamwork skills (4.4%) and job knowledge/skills (3%).

2) Criterion-related validity depends--as you might expect--on the match between predictor and performance measure. Conscientiousness measures, for example, predicted task performance much better than managerial performance (rho=.39 and .06 respectively). The highest correlations (albeit based on relatively small samples) were for teamwork skills and personality composites predicting task performance (.50 and .45 respectively).

3) Video-based SJTs tended to have stronger criterion-related validity values compared to paper-based measures. This was particularly true when measuring interpersonal skills (.47 compared to .27).

Second, a small but interesting study by Johnson, et al. on the relationship between trait affect (i.e., being generally disposed to feeling positive or negative emotions) and job performance. Results from 120 matched employee-supervisor pairs from a variety of jobs using both explicit (survey) and implicit (word fragment completion) measures of affect found substantial correlations, particularly between positive affect and performance (in the .50 range), and particularly when using implicit measures.

Something to add to a selection battery, perhaps? Could be perceived negatively by applicants, however, and I can see some questions being raised about the link to medical issues. But the same types of concerns were originally leveled at personality tests and were mitigated by creating measures specifically tied to work behavior. Definitely an area for more research.

Third, check out this study by Becker, et al. on the impact that job offer timing has on acceptance, performance and turnover. The authors found (using data from a Fortune 500 engineering technology company) that for both student and experienced samples, faster offers were associated with higher acceptance rates. Specifically, for experienced candidates, the difference between 2 weeks and 3 weeks taken to make the offer was substantial, whereas for the students 3 weeks versus 4 weeks was important. But, no differences were found in terms of either performance ratings or turnover among employees hired through different offer speeds.

Implication? The study suggests that offer time does impact the likelihood that the offer will be accepted, but viewed broadly this may not have long-term impacts in terms of how employees do on the job. Maybe in cases of good candidate-employer fit, candidates are willing to wait.

Last but not least are the book reviews. Two books are particularly relevant for us, The Structured Interview (Pettersen & Durivage) and Outliers (Gladwell). The first is received very positively and sounds like a great source for anyone wanting more details about the support for and use of structured interviews. The latter is "well worth [a] few evenings" but requires you to overlook the lack of evidence and convenient inferences.

Final notes: those of you interested in multisource performance ratings should check out Hoffman, et al.'s article, which reinforces the impact of having raters from different levels. Chuang and Liao's article also includes a useful measure of a high-performance work system.

Tuesday, March 02, 2010

IPAC Conference + Innovations Award


What are you doing July 18-21? I assume if you enjoy good weather, good company, and--most importantly--great information on state-of-the-art selection practices, you'll be joining me at IPAC's annual conference in beautiful Newport Beach, California.

If you're not only going but have something to present, by all means respond to the call for proposals. It could be a workshop, panel discussion, symposia--pretty much any format you can think of. Don't wait too long, the deadline is this Friday, March 5.

And speaking of the conference, IPAC has announced that nominations for the Innovations in Assessment Award are being accepted from 5/17-6/18. The winner receives not only formal recognition (and bragging rights), but a free pass to the conference.

IPAC's a great group, full of people that are extremely knowledgeable and passionate about using the best selection practices to get organizations the talent they need. Plus, it's the only international (or national) organization I know of devoted exclusively to the topic.

Hope to see you there!

Friday, February 26, 2010

Book review: Strategy-Driven Talent Management


A thought-provoking collection of essays and ideas; but it won't solve all our problems.

The value of a book lies as much with the reader as it does with the content. A book about advanced programming does little good to the person who has problems turning a computer on. A collection of cooking recipes is largely useless to someone who exclusively uses a microwave.

The same is true about business and HR books. Depending upon who you are and where you're at in life, some books may help you, some may be beyond your reach. Such is the case with SIOP's latest entry into its Professional Practice series, Strategy-Driven Talent Management: A Leadership Imperative, edited by Bob Silzer and Ben Dowell.

The book (tome, actually, at nearly 900 pages) is full of thought-provoking pieces from a variety of authors, including some familiar faces such as John Boudreau and Allan Church. There are academics present, but the majority of authors are practitioners in private sector organizations, such as Aon, Ingersoll Rand, HP, Sara Lee, Merck, and Bristol-Myers Squibb.

The book is roughly broken up by major topic area, although the distinction can be hard to maintain. There are chapters on recruitment, executive onboarding, engagement, measurement, and global issues. There's even a 40-page annotated bibliography. But the editors do an admirable job of keeping the topics all related to the broad field of talent management (TM), which they define as, "an integrated set of processes, programs, and cultural norms in an organization designed and implemented to attract, develop, deploy, and retain talent to achieve strategic objectives and meet future business needs" (p. 18).

The book is described as a "comprehensive [collection of] state-of-the-art ideas, best practices, and guidance." It shines on the first two but failed me on the last, although not for lack of trying. The problem is the book is so long, full of so many ideas and case studies, that it's very easy to get lost and not come away with any clear guidance based on the consensus of authors. To some extent this is endemic in any collection of works by separate authors, but it's clearly a collection of "what you might do" rather than a solid prescription for "how to", although some authors do a better job than others.

Another problem is that many authors seem to presume that current TM practices are sub-optimized because they aren't linked to business strategy and results. This may be true if the process is based on non-validated assumptions, but as long as there is a link between job success and specific practices, we're already there. We just haven't made a particularly good link between job success and organizational success, which may explain the attraction to concepts like competencies (mentioned many times in the book).

But my main problem, and this goes for the book as well as the field, is that it treats the concept of talent management as a logical process to be managed. Somewhere in the transition from HR to TM, we lost the H--human. Talent management (and HR) is messy because it involves people. It's political. It changes every day. And you're dealing with emotions, not lines of code. The real challenge--which is discussed but to my mind not driven home--is how to get the talent mindset into the organizational DNA.

There is value to thinking broadly and philosophically about the topic. It helps us plan. But what people really need are concrete suggestions for establishing a self-sustaining high-performance system. In order to do this, we must address the fundamentals (the basic needs of Maslow's hierarchy, if you will), such as:

- HR must learn "the business" and stay close to their customers
- Supervisors must be selected and trained with their talent management role at the forefront
- Success in HR must be defined and measured. It must be communicated, understood, and valued
- Sustained attention to HR success and significant resources must be expended by both HR and line managers

The book does a passable job of presenting these, but you may have to dig for them. The bigger problem is that there seems to be an assumption that what keeps organizations from having a top-notch TM system is a lack of understanding, either of the organizational strategy or best practices in TM, rather than the very real daily troubles that organizations experience, such as:

- Supervisors that hire people they know/like rather than the most qualified person
- People placed into HR with little or no background, interest, or passion for it
- Insufficient resources devoted to TM/HR
- HR managers who are just that--managers--rather than real HR leaders (Avedon and Scholes present a great assessment in Chapter 2 that helps separate these)

Until organizations have these types of "minor"--but real--flaws ironed out, all the charts and good intentions in the world will have very little impact.

Finally, I was also disappointed that there wasn't more in here about evidence-based TM and HR (which may say more about the field than the authors/editors, who acknowledge this lack in Chapter 22). The field desperately needs more research to tie the hard science of assessment with the more anecdotal/consultant practices such as recruiting, retention, and performance management. This will require significantly more research using methods beyond surveys in order to show what works and what doesn't. There are some ties to good research in here, but the hole is significant.


To summarize, the book contains a lot to like, particularly for individuals already schooled in this area looking to optimize their shop, or for graduate students seeking to understand the big picture. But for most HR practitioners (and, I would expect, executives), this book is akin to a collection of recipes for advanced Italian cooking--fabulous for those used to making their own pasta, but beyond the reach of those struggling to make their own sauce.

Thursday, February 25, 2010

Webinar on 21st Century Assessment


Went to a pretty darn good webinar yesterday put on by HCI and featuring Ken Lahti (PreVisor) and Charles Handler (Rocket-Hire). The topic was 21st century assessment.

Some of the topics covered included:

- increased functionality and usability of testing platforms

- increased sophistication of security methods

- off-the-shelf tests and "I/O psychologists in a box"

- integrating assessment with your overall talent strategy

And my two favorites:

- advanced simulations (such as those using video game technology)

- candidate data that follows them

The webinar is going to be re-broadcast several times today and tomorrow, if you have a chance check it out. You can also see a copy of the slides for free if you're an HCI member (which is free).

Free, short, and full of information--that's my kind of training.

Saturday, February 20, 2010

Recruitment v. Assessment, Round 1: Fight!


I'm sure some of you are avid readers of ERE (Electronic Recruiters Exchange), but for those of you that aren't (and don't receive my shared items), there's a lively discussion going on over there regarding the practical value of recruitment versus assessment practices.

It started with Wendell Williams' first post on how to identify a bad test (the second part is also worth reading). The comments begin relatively benignly, debating the strength of various predictors of performance (e.g., P-O fit versus behavioral interviews), but turns into, let's say, a lively debate that includes a discussion of the Gallup 12, the limitation of assessments, the Uniform Guidelines, and a lot more. The most heated exchange occurs between Wendell and Lou Adler, where accusations and sarcasm fly.

Speaking of Lou, he continues to advocate his perspective with his next post on whether increasing interview accuracy increases quality of hire (yes, he's suggesting that's an open question). While the comments following are fewer in number, the debate continues regarding the value of assessment and the evidence used to support it (e.g., Schmidt & Hunter's 1998 piece).

Who said HR is boring?

Saturday, February 13, 2010

Latest IJSA: Emotional intelligence, multiple-choice formats, and lots more

The March 2010 issue of the International Journal of Selection and Assessment (IJSA) is out, and the research covers a wide variety of recruitment and assessment topics as well as being truly international:

Unproctored internet-based testing (UIT) response distortion may be less than we fear (sample included cognitive and personality measures)

What factors are most important to organizations when choosing a test? This study suggests applicant reaction, cost, and diffusion of the test type in the field.

Personality (esp. core self evaluation) is related to the type of work preferred, and hence P-O fit

Career site features may differentially attract men and women

Corporate images do matter when it comes to organizational attractiveness

Who uses job-search websites and how to improve them (the sites, not the people)

Support for performance-based (as opposed to self-report) measures of emotional intelligence

Work samples, interviews, and ability tests perceived best by employees (why? because they work, say the participants)

...and last but definitely not least:

A "2 of 5" multiple-choice format seems superior than traditional "1 of 6" (you just have to make sure you can score them that way!)

Sunday, February 07, 2010

Feds new jobs site is Googlish

The U.S. Government has revamped its jobs page, www.usajobs.gov, and in the process shown everyone else how its done.

Take a look at their old site. Not horrible, but cluttered with lots of features that distracted from the main reason people visit the site: to look for a job.


This picture actually doesn't do it (in)justice; there was additional content below the bar.

Now look at their new website:


This new website is what I would call "Googlish": simple, lots of white space, no scrolling required, and a single search box. The design focuses less on being pretty, and more on being functional. If you're interested in learning more about careers, or if you'd like information related to specific groups, like veterans or those with disabilities, its still there. And there's even more functionality up top in the form of drop-down menus.

Job seekers don't need a magazine ad. They need to quickly and easily find information. And this new website fits the bill.

How does yours compare?

Thursday, February 04, 2010

Lessons from NYC Fire case - part 2

Part 2 of 2

Last time I discussed five important lessons we can take away from recent rulings in the Vulcan v. City of New York case. In this post I'll review the remaining lessons and also discuss the relief order.

----

6) The city failed to provide sufficient evidence that the exam(s) tested for a sufficient number of the critical KSAs. They also failed to explain why they chose not to measure several KSAs identified as critical.

Lesson: the courts do not require employers to measure every single critical KSA. But there is an expectation that employers attempt to measure a sufficient number that represent a significant portion of the job requirements. In this case, that included non-cognitive abilities such as resistance to stress, teamwork, and conscientiousness, that were not measured.

7) The city failed to adequately consider how to measure a significant number of essential KSAs. While some of their concerns were valid (e.g., structured interviews for all applicants would be an operational nightmare), there are many different forms of testing that should have been considered, including situational judgment tests (SJTs) and biodata, which can be used to measure non-cognitive components.

Lesson: triers of fact expect employers to be up on the various assessment methods available and be able to explain why they chose not to use certain ones. This includes tests that are relatively easy to develop (e.g., SJTs) as well as ones that require substantial resources and statistical expertise (e.g., biodata).

8) The city failed to conduct a reading level analysis on the exams to ensure that it was not "pointlessly high." The plaintiff introduced evidence suggesting the reading level was above 12th grade; in addition, it appeared to exceed the reading level of materials at the academy.

Lesson: never forget that every assessment method is in some sense measuring additional KSAs beyond those you intend. For written exams, reading comprehension is always a requirement (barring accommodation). It's quite easy to conduct a reading level analysis (MS Word has it built in) to ensure that the level is reasonable and matches other job-related material.

9) The city failed to show that the cutoff scores (pass points) established for the exams were based on adequate rationale, namely "the necessary qualifications for the job of entry-level firefighter." Instead, the cutoff scores were based on operational need (the number of job openings expected). This is particularly important in multiple-hurdle selection processes such as in this case, where a failure on one exam component precludes an applicant from participating in the rest of the (potentially compensatory) assessment process.

Lesson: ultimately applicants have to pass the test(s) to be considered for employment. Cutoff scores should be established using the expertise of both SMEs and test developers and should be based on the minimum competency levels required upon entry to the job. At a minimum (and I would not rely solely upon this), the scores should be analyzed to identify any logical "break-points."

----

After ruling for the plaintiffs on both the adverse impact and disparate treatment claims, the judge issued a relief order on 1/10/10. In it, he imposes several things, including the following:

1) The city must develop a new testing procedure for entry-level firefighter in conjunction with the relevant parties. Following the development of the test, there will be a hearing to determine if this test should be used rather than the current test (developed in 2007 and not at issue in this litigation).

2) The court shall develop a process by which the approximately 7,400 applicants covered by this case can file a claim for monetary relief.

3) The city will identify 293 black candidates on the eligibility list and offer them priority hiring. (No quotas are being imposed, although the judge leaves this possibility open)

4) Retroactive seniority for those hired.

In addition, several other issues are up for debate, including the appointment of a special master or monitor, standards that will be relied upon in constructing the new exam, and the need for additional relief.

---

So what did we learn from all this? If you follow--fairly closely--best practices when developing and administering exams, you will be on solid ground defending them. If you don't, and your exam has a discriminatory effect, you may be called on it--and it's not a pleasant process. I'll leave you with this quote from the January ruling on disparate treatment:

"The history of the City's efforts to remedy its discriminatory firefighter hiring policies can be summarized as follows: 34 years of intransigence and deliberate indifference, bookeneded by identical judicial declarations that the City's hiring policies are illegal."

Sunday, January 31, 2010

Lessons from the NYC Fire case - part 1

Part 1 of 2

New York City, like the cities of New Haven and Chicago, has a long history of employment discrimination litigation related to its firefighter testing.

Since the 1970s and cases like Guardians, the city has been under scrutiny for its woefully low number of black firefighters.

In 2007 the city found itself faced with another lawsuit over its firefighter hiring practices, and in July of 2009, a U.S. District Court judge found that the city had violated Title VII by administering written exams from 1999-2007 that had high levels of adverse impact. The city marshaled an inadequate defense. In January of 2010, the same judge (Nicholas Garaufis) found the city liable for a pattern and practice of disparate treatment for those same exams. An adverse impact finding, particularly for written exams, and especially for public safety tests, is not earth-shattering. But a finding of disparate treatment in this situation is less common.

This case, while only one example and limited in its impact, has some valuable lessons for test users and sheds some light on how judges look at our field. In particular, I describe below nine points the judge specifically made and what lessons we can draw from them:

1) While the city conducted a job analysis with an "extensive" list of tasks and surveyed incumbents, the city offered "no evidence of 'the relationship of abilities to tasks.'" They conducted a linkage, but the judge found that the SMEs were confused about what they were supposed to do and didn't understand several of the abilities they were rating.

Lesson: simply having subject matter experts (SMEs) link essential tasks and knowledge, skills, and abilities (KSAs) is not sufficient. You need to ensure they understand the statements they are linking as well as how exactly they are supposed to be linking them.

2) In conducting the job analysis, the city inappropriately retained tasks and KSAs that could be learned on the job. It is quite clear (e.g., per the Uniform Guidelines) that only tasks and KSAs that are required upon entry to the job should be identified as critical in terms of exam development.

Lesson: make sure that when you are developing exams based on job analysis results that you focus only on those tasks and KSAs that are required upon entry to the job. This should be determined by your SMEs.

3) The city relied to some extent upon the work of a previous test developer, Dr. Frank Landy (who sadly recently passed away). In addition to a tenuous link between Dr. Landy's work and the current exams, the judge makes it clear that "reliance on the stature of a test-maker cannot stand in for a proper showing of validity." At the same time, the judge emphasizes that exams should be constructed by "testing professionals."

Lesson: tests should be developed by people who know what they're doing. This means HR professionals with the requisite background in test validation and construction in conjunction with job experts. Do not rely solely on previous efforts, particularly when (as in this case) the results of those efforts were either incomplete or not fully relevant to your current situation.

4) The city performed no "sample testing" to ensure that the questions were reliable as well as "comprehensible and unambiguous."

Lesson: few steps in the test development process are as easy--or as valuable--as pilot testing. I have yet to see an exam that didn't benefit from a "trial run" with a group of incumbents. Not only will you catch unintended flaws, you will verify that the exam is doing what you claim it is.

5) There was insufficient evidence that the exams actually measured the (nine cognitive) KSAs the city claimed they intended to measure. Plaintiffs were able to suggest the opposite through analyzing convergent and discriminant validity as well as by conducting a factor analysis.

Lesson: there are two linkages of primary importance in test development. The first was describe in #1. The second is the link between critical KSAs and the exam(s). At the very least, you must be able to show evidence that there is a logical link between the two. When you claim to be measuring cognitive abilities, you incur an additional responsibility, which is gathering statistical evidence that supports this claim.

Next time: more lessons and the relief order.

Friday, January 22, 2010

Jan '10 issue of JAP, plus APA gets stingy

The January 2010 issue of the Journal of Applied Psychology is out, and there are some good articles to take a look at. It just may be more difficult to see them. More on that in a minute.

First, here are some of the titles in this issue:

Emotional intelligence: An integrative meta-analysis and cascading model. A must for anyone interested in EI; posits and supports a cascading model whereby emotion perception-->emotion understanding-->emotion regulation-->job performance.

Time is on my side: Time, general mental ability, human capital, and extrinsic career success. GMA shown to have strong links to two extrinsic measures of career success, income and prestige. (ah, but are smart people happier?)

I won’t let you down… or will I? Core self-evaluations, other-orientation, anticipated guilt and gratitude, and job performance. Core self-evaluations' impact on job performance may depend on how much they focus on others.

Understanding performance ratings: Dynamic performance, attributions, and rating purpose. Performance ratings are influences by a variety of things, including overall performance variance and purpose of the ratings.


Okay, so back to my earlier comment: it appears that APA has restricted viewing abstracts of their journals to registered members (hence the lack of links in this post). On the one hand, no big deal, it appears you can simply register to gain access. On the other hand...why should someone have to do this? This is another unfortunate example of research being restricted (first by charging exorbitant fees for articles, now through personal identification) and contributes to the field being insular.

Granted, APA's not the only one that does this (hey buddy, got $400 for the CRL?) but that doesn't excuse it. Our field benefits from sharing of information, not just among professionals but with the general public. Requiring registration does not further that goal. Thankfully some individual researchers (see the sidebar on the main page) allow access to their work--something we should all be grateful for.

Monday, January 18, 2010

How to get r = 1.0


Recruiters have a variety of measures of their success, often including process outcomes (time-to-fill, number of requisitions filled, etc.).

And although assessment professionals have a variety of success measures, some in common with recruiters (e.g., tenure), there is one measure that stands above all others: job performance.

The "gold standard" of this measurement is to correlate test scores with job performance measures (called criterion-related validation evidence). A correlation of, say, .50 between these two, is considered outstanding. Square that and you have the percentage of behavior explained. So in other words, when we can explain 25% of job performance with assessments, we call that success (and with good reason, because it's a heck of a lot better than 0%).

Why not higher than 25%? What would it take to get r =1.0, in other words a perfect correlation between test scores and performance? Here is a somewhat tongue-in-cheek recipe for achieving this impossible dream:

1. An accurate identification of the top competencies/KSAs required for the job. Qualified subject matter experts reach consensus on a handful of far and away the most important qualities that impact job performance.

2. Perfectly constructed and administered, perfectly reliable and accurate measures of the the top KSAs.

3. Variability among applicants in terms of amount of the relevant KSAs possessed.

4. Test scores combined and weighted appropriately given the job analysis results.

5. Variability in scores for those hired.

6. A clear description of the work to be performed and competencies to be demonstrated so the individuals understand expectations.

7. Perfectly reliable, accurate measures of job performance that capture behaviors one would logically relate to the critical KSAs.

8. A supportive work environment (e.g., high quality supervision, adequate resources) so this doesn't interfere with work performance.

9. Variability in job performance among those hired using the assessments.

10. Elimination of outside factors that may contribute to lower job performance (e.g., family emergencies, medical/psychological changes).

As you can see, some of these are achievable (1, 4, 6), others are challenging and depend on circumstances, but are not impossible to achieve (3, 5, 8, 9) and some are practically impossible (2, 7, 10). I said earlier this was tongue-in-cheek because obviously we'll never have a situation where all of these conditions (as well as ones I'm sure I forgot) are true.

Does this mean we should abandon the correlation between test score(s) and job performance? Absolutely not. It should continue to be one of our "gold standards" for measuring our success as assessment professionals. But we--and our customers--should have our eyes wide open before pressing "compute."

Wednesday, January 13, 2010

Meet the new SIOP...same as the old SIOP

The votes are in, and the new name for the Society for Industrial and Organizational Psychology (SIOP) is...the same.

After over a thousand votes from members, the existing acronym beat The Society for Organizational Psychology (TSOP) by a tally of 51% to 49%--a difference of 15 votes. You can read my comments about this option--and my prediction of the outcome--here.

Why is this non-news, news? Because it's problematic that the main professional, scientific body that devotes itself to researching the psychology of organizations and work (POW!) repeatedly has identity issues. This is in large part because of the word "industrial", which makes it sound like we're all studying factory workers. I am not alone in having people look at me sideways when I attempt to explain our field.

To be perfectly honest, I am reluctant to describe my focus as "psychology", except to others in the same field. It sidetracks the conversation (perhaps due to my insufficient skill). It's much easier to connect with people by saying I'm in Human Resources. This isn't to say that the focus on psychology isn't important, or that others in I/O psychology might not mind using this phrase, or that there isn't some brand value in SIOP. But call me crazy, if you're reluctant to name your field (and attorneys don't count--people know, or think they know, what you do), the profession has a problem.

So, our identity struggle continues. An interesting follow-up study might be to ask SIOP members how they describe their field of work to non-I/O folk and break that down by area of focus. It's a big tent.

Personally, I prefer something that includes Work and Organizational. Mix and match letters as you will.

On a positive note, did you know you can access all of SIOP's quarterly news publication, TIP, here? The January 2010 issue has pieces on integrated performance management, a preview of Lewis v. City of Chicago, and a lot more.

Wednesday, December 30, 2009

Outback settlement contains interesting requirements


You may have heard that Outback Steakhouse, a restaurant chain based in Tampa, Florida, has agreed to settle a gender discrimination lawsuit for $19M. What's interesting about this isn't the size of the settlement, but rather the conditions attached.

Background: The EEOC sued Outback in 2006, claiming it systematically discriminated against its female employees by denying them promotion opportunities to the more lucrative profit-sharing management positions. In addition, they claimed that female employees were denied promotional job assignments such as kitchen management, which were required for employees to be considered for top management positions.

The settlement: Outback agreed to a four-year consent decree and $19M in monetary relief. So far, pretty standard. But there were additional settlement requirements, and here's where it gets interesting. In addition to the monetary relief, Outback has agreed to:

1. Create an online application system for employees interested in management positions. This is the first time I've seen this in a settlement (which isn't to say it hasn't happened) and seems to indicate that the EEOC views this as a more "objective" screening mechanism.

2. Create and hire someone for a newly created "human resources executive" position titled Vice President of People. Again, this is a new one for me.

3. Hire an outside consultant for at least two years who will monitor the online application system to ensure women are being provided equal opportunities for promotion and provide reports to the EEOC every 6 months.

The main thing that strikes me about this settlement is the faith that is being placed in an online application system to somehow ensure equal opportunity. Sure, having a standardized application system may cut down on some of the subjectivity of individual hiring supervisors, but it leaves me wondering:

- What will the screening criteria for management positions be?

- How will the outside consultant define "equal opportunities"?

- How will access to the online system be controlled, and who will be making screening/hiring decisions?

- What happens if there continues to be adverse impact, which you would expect if applicants continue to be screened on experience?

- What will be the duties of the Vice President of People, how will they be hired, and how will they interact with the consultant?

This will be interesting to watch.

Sunday, December 20, 2009

Validity: An elusive (unitary?) concept

What makes a test "valid"? What is the best way to develop a selection system? These are two of the most fundamental questions we try to answer as personnel assessment professionals, yet the answers are strangely elusive.

First of all, let's get two myths out of the way: (1) a test is valid or invalid, and (2) there is a single approach to "validating" a test. It is the conclusions drawn from test results that are ultimately judged on their validity, not simply the instruments themselves. You may have the best test of color vision in the world--that doesn't mean it's useful for hiring computer programmers. And many sources of evidence can be used when making the validity determination; this is the so-called "unitary" view of validity described in references like the APA Standards and the SIOP Principles. Unitary in this case refers to validity being a single, multi-faceted concept, not that psychologists agree on the concept of validity--a point we'll come back to shortly.

Although we can debate test validation concepts ad infinitum, the bottom line is we create tests to do one primary thing: help us determine who will perform the best on the job. The validation concept that most closely matches this goal is criterion-related validity: statistical evidence that test scores predict job performance. So we should gather this evidence to show our tests work, right? Here's where things get complicated.

It's likely that many organizations can't, for various reasons, conduct criterion-related validity studies (although baseline evidence of this would be helpful). Most of the time, it's because they lack the statistical know-how or high quality criterion measures (a 3-point appraisal scale won't do it). So in a strange twist of fate, the evidence we are most interested in is the evidence we are least likely to obtain.

So what are organizations to do? Historically the answer is to study the requirements of the job and select/create exams that target the KSAs/competencies required; this matching of test and job is often referred to as "content validity" evidence. But Kevin Murphy, in a recent article in SIOP's journal Industrial and Organizational Psychology, makes an important point: this is good practice, but not a guarantee that our tests will be predictive of job performance. Why not? For a number of reasons, including poor item writing and applicant frame of reference. Murphy makes a passionate argument that we rely way too heavily on unproven content validation approaches when we should focus more on criterion-related validation evidence. Instead of focusing on job-test match, we should focus on selecting proven, high quality exams.

Not surprisingly, the article is accompanied by 12 separate commentaries that argue with various points he makes. It's also interesting to compare this piece with Charley Sproule's recent IPAC monograph where he makes an impassioned defense of content validity.

A complete discussion of the pros and cons of different forms of validation evidence are obviously beyond a simple blog post. My main issues with Murphy's emphasis on criterion-related validation are threefold. First, as stated above, most organizations likely don't have the expertise to gather criterion-related validation evidence for every selection decision (maybe this is his way of creating a need for more I/O psychologists?). Perhaps "insufficient resources" is a poor excuse, particularly for an issue as important as employment, but it is a reality we face.

Second, even if we were to shift our focus to individual test performance, following a content validation approach for development enhances job relatedness (which Murphy acknowledges). Should your selection system face an adverse impact challenge, the ability to show job relatedness will be essential.

Finally, let's not forget that high test-job match gives candidates a realistic job preview--hardly an unimportant consideration. RJPs help candidates decide whether the job would be a good match for their skills and interests. And no employer that I know of enjoys answering this question from candidates: "What does this have to do with the job?"

The approach advocated by Murphy, taken to its extreme, would result in employers focusing exclusively on the performance of particular exams rather than on their content in relation to the job. This seems unwise from a legal as well as face validity perspective.

In the end, as a practitioner, my concern is more with answering the second question I posed at the beginning of this post: What is the best way to develop a selection system? Given everything we know--technically, legally, psychologically--I return to the same advice I've been giving for years: know the job, select or create good tests that relate to KSAs/competencies required on the job, and base your selection decision on the accumulation of test score evidence.

Should researchers work harder to show that job-test content "works" in terms of predicting job performance? Sure. Should employers take criterion-related validation evidence into consideration and work to collect it whenever possible? Absolutely. Will job-test match guarantee a perfect match between test score and job performance? No. But I would argue this approach will work for the vast majority of organizations.

By the way, if you are interested in learning more about the different ways to conceptualize validity--"content validity" in particular--Murphy's focal article as well as the accompanying commentaries are highly recommended. He acknowledges that he is purposely being provocative, and it certainly worked. It's also obvious that our profession has a ways to go before we all agree on what content validity means.

Last point: the first focal article in this issue--about identifying potential--looks to be good as well. Hopefully I'll get around to posting about it, but if not, check it out.

Sunday, December 13, 2009

R/A Predictions for 2010

With 2010 right around the corner, here are some predictions for what the new year will bring in the area of recruitment and assessment:

1) More personality testing. Year after year personality testing continues to be one of the hottest topics. Look for more research, more online personality testing, and new measurement methods.

2) More boring job ads. Even though we know better, don't expect to see any big leaps in readability for 80% of job ads. Same old job descriptions. Maybe we'll see some pictures. On the plus side, more organizations focus on making their career portals attractive.

3) A slow trickle of research on recruiting. The amount of large-scale, sophisticated research on recruiting methods remains a shadow of that found in the assessment literature. Don't expect this to change.

4) More focus on simulations. 2010 sees more focus on simulations, particularly those delivered on-line, as highly predictive assessments as well as realistic job previews. Oh, and they likely have low adverse impact (research, anyone?).

5) Leadership assessment gets even hotter. With the economy improving and more boomers deciding the time is right to retire, finding and placing the right people in leadership positions becomes an even more important strategic objective.

6) Federal oversight agencies get more aggressive. With more funding and backing from the Obama administration, expect to see the EEOC and OFCCP go after employers with renewed vigor. By the way, have you seen the EEOC's new webpage? It's actually quite well done.

7) More fire departments get sued. In the wake of the Ricci decision, fire dept. candidates feel emboldened when they fail a test or fail to get hired/promoted. Look for departments to try to get out ahead of this one by revamping their selection systems.

8) More age discrimination lawsuits. With so many boomers, expect to see more claims of discrimination, particularly over terminations. Keep words like "energetic" and "fresh" out of your job ads.

9) Automation providers slowly focus on simplicity. Whether we're talking applicant tracking or talent management systems, vendors slowly realize that they need to make their applications simpler to increase usability and buy-in. No, simpler than that. Keep going...

10) Employers get more sophisticated about social networking sites. Many realize that rather than jumping on the latest Twitter-wagon, it's best to figure out where these sites fit with their recruitment/assessment strategy. Watch for more positions whose sole role is managing social media.

11) Online candidate-employer matching continues to be a jumbled mess. Without a clear winner in terms of a provider, job seekers are forced to maintain 400 profiles on different sites and may give up altogether and focus more on social networking. Meanwhile, employers continue to try to figure out how to reach passives; LinkedIn continues to look good here but needs to expand its reach a la Facebook.

12) More employers face the disappointing results of online training and experience questionnaires. Will they go back to the drawing board and try to improve them (hint: don't use the same scale throughout), or abandon them for more valid methods, such as biodata, SJT, and simulations? More research on T&Es is badly needed, even if we are just putting lipstick on a pig.

13) Decentralized HR shops centralize. Centralized ones decentralize. Particularly in the public sector, these decisions unfortunately continue to be made based on budgets rather than best practice. Hiring supervisors wonder why HR still can't get it right.

14) Fortunately, HR continues to professionalize. With much of the historical knowledge walking out the door and the job market improving, HR leaders are forced to re-conceptualize how they recruit and train recruitment and assessment professionals. This is a good thing, as it means more focus on analytical and consultative skills.

Keep up the good work everybody. And Happy Holidays!

Sunday, December 06, 2009

Setting cutoff scores on personality tests


What's the best way to set a cutoff score for a personality test, knowing that some candidates inflate their score? It all depends on your goal. Are you trying to maximize validity or minimize the impact of inflation?

According to a research study by Berry & Sackett published in the Winter '09 issue of Personnel Psychology, if your goal is to maximize validity, your best bet is to wait until applicants have taken the exam, then set your cut-score (e.g., the top two-thirds); this was particularly true when selection ratios are small (i.e., organization is very selective).

If your goal is to minimize the number of deserving applicants who are displaced by "fakers", you're better off establishing the cut point ahead of time, by using a non-applicant derived sample (e.g., job incumbents, research group). The results were generated using a Monte Carlo simulation.

Interestingly, the authors also replicated the work of other researchers who have shown that the impact of faking on the criterion-related validity of personality measures is relatively low. There are a few other very good points made in this article:

- Expert judgment methods of establishing pass points (e.g., Angoff method) may be difficult to use for personality tests since experts may find it difficult to judge individual items. Methods used to select a certain number of applicants or methods based on a criterion-related validity study (both used as variables in this study) are more appropriate for personality tests.

- There is no consensus of how prevalent faking on personality exams is; estimates range from 5-71%. It likely depends on the situation and how motivated test takers are to engage in impression management.

- Some recommend setting a very low cutoff score for personality tests, which would exclude only those likely not suitable for the position (and not faking), while others prefer a more stringent cutoff to maximize utility.

- A reasonable range of d-values for score inflation on personality inventories is .5-1.0 (used in this study).

- There exists very little research on the skewness of faking score increases. A positively-skewed distribution (meaning most people faked a small amount) was used in this study. (I would think this would also vary on the situation)

So bottom line: where--and how--you set your cutoff score on personality inventories depends on whether you want to maximize the predictive validity or minimize the number of deserving applicants that get left out of the process.

Other good reads in this issue:

- Police officer applicants reactions to promotional assessment methods

- The impact of diversity climate on retail store sales

- The construct validity of multisource performance ratings

- Labor market influences on CEO compensation