Q&A with Piers Steel: Part 1

A few weeks ago I wrote about a research article that I think proposes a revolutionary idea: The creation of synthetic validity database that would generate ready-made selection systems that would rival or exceed the results generated through a traditional criterion validation study.

I had the opportunity to connect with one of the articles authors, Piers Steel, Associate Professor of Human Resources and Organizational Dynamics at the University of Calgary. Piers is passionate about the proposal and believes strongly that the science of selection has reached a ceiling. I wanted to dig deeper and get some details, so I posed some follow-up questions to him. Read on for the first set of questions, and I'll post the rest next time:

Q1) What is the typical state of today's selection system--what do we do well, and what don't we?

A1) Here is quote from well a respected selection journal, Personnel Psychology: “Psychological services are being offered for sale in all parts of the United States. Some of these are bona fide services by competent, well-trained people. Others are marketing nothing but glittering generalities having no practical value.... The old Roman saying runs, Caveat emptor--let the buyer beware. This holds for personnel testing devices, especially as regard to personality tests.”

Care to try and date it? It is from the article “The Gullibility of Personnel Managers,” published in 1958. Did you guess a different date? You might of, as the observation is as relevant today as yesterday -- nothing fundamental has changed. Just compare that with this more recent 2005 excerpt from HR Magazine, Personality Counts: “Personality has a long, rich tradition in business assessment,” says David Pfenninger, CEO of the Performance Assessment Network Inc. “It’s safe, logical and time-honored. But there has been a proliferation of pseudo tests on the market: Caveat emptor.”

Selection is typically terrible with good being the exception. The biggest reason is that top-notch selection systems are financially viable only for large companies with a high-volume position. Large companies can justify the $75,000 cost and months to develop and validate and perhaps, if they are lucky, have the in-house expertise to identify a good product. Most other employers don’t the skill to differentiate the good from the bad as both look the same when confronted with nearly identical glossy brochures and slick websites. And then the majority of hires are done with a regular unstructured job interview – it is the only thing employers have the time and resources to implement. Interviews alone are better than nothing but not much better – candidates are typically better at deceiving the interviewer than the interviewer is at revealing the candidate.

The system we have right now can’t even be described as being broken. That implies it once worked or could be fixed. Though ideally we could do good selection, typically, it is next to useless, right up there with graphology, which about a fifth of professional recruiters still use during their selection process. For example, Nick Corcodilos reviews how effective internet job sites are getting people a position. He asks us to consider “is it a fraud?”

Q2) What's keeping us from getting better?

A2) Well, there are a lot of things. First, sales and marketing works, even if the product doesn’t. When you have a technical product and untechnical employer or HR office, you have a lot of room for abuse. I keep hearing calls for more education and that management should care more. You are right they should care more and know more. People should also care and know more about their retirement funds as well. Neither is going to change much.

Second, the unstructured job interview has a lot of “truthiness” to it. Every professional selection expert I know includes a job interview component to the process even when it doesn’t do much, as the employer simply won’t accept the results of the selection system without it. There are some cases where people “have the touch” and are value added but this is the exception. Still, everyone thinks they are gifted, discerning, and thorough. This is the classic competition between clinical and statistical prediction, with evidence massively favoring the superiority of the latter over the former but people still preferring the former over the latter (here are few cites to show I’m not lying, as if you are like everyone else, you won’t believe me: Grove, 2005; Kuncel, Klieger, Connelly, & Ones, 2008).

Third, it just costs too much and takes too much time to do it right. Also, most jobs aren’t really large enough to do any criterion validation.

Q3) What might the future look like if we used the promise of synthetic validity?

A3) Well, to quote an article John Kammeyer-Mueller and I wrote, our selection systems would be "inexpensive, fast, high-quality, legally defensible, and easily administered.” Furthermore, every year they would noticeable improve, just like computers and cars. A person would have their profile taken and updated whenever they want, with initial assessments done online and more involved ones conducted in assessment centers. Once they have the profile, they would get a list of jobs they would likely be good at, ones that they would be likely good at and enjoy, and ones they would be likely good at, enjoy and that are in demand.

Furthermore, using the magic of person-organization fit, you inform them what type of organization they would like to work for. If someone submitted their profile to a job database, every day job positions would come to them automatically, with the likelihood of them succeeding at it. These jobs would come in their morning email if they wanted it. Organizations would also automatically receive appropriate job applicants and a ready built selection system to confirm that the profile submitted by the applicant was accurate.

Essentially, we would efficiently match people to jobs and jobs to people. I would recommend people update their profile as they get older or go through a major life change to improve the accuracy of the system, but even initially it would be far more accurate than anything available today -- a true game changer.

Follow-up: Some might see a contradiction here. You cite an article that bashes internet-based job matching, yet this is what you're suggesting. Would your system be more effective or simply supplement traditional recruiting methods (e.g., referrals)?

A: Yup, we can do better. The internet is just a delivery mechanism and no matter how high-speed and video enabled, it is just delivering the same crap. This would provide any attempt to match people to jobs or jobs to people with the highest possible predictiveness.


Next time: Q&A Part 2

References:
Grove, W. M. (2005). Clinical versus statistical prediction: The contribution of Paul E. Meehl. Journal of Clinical Psychology, 61(10), 1233-1243. doi: 10.1002/jclp.20179

Kuncel, N. R., Klieger, D., Connelly, B., & Ones, D. S. (2008, April). Mechanical versus clinical data combination in I/O psychology. In I. H. Kwaske (Chair), Individual Assessment: Does the research support the practice? Symposium conducted at the annual meeting of the Society for Industrial and Organizational Psychology, San Francisco, CA.

Stagner, R. (1958). The Gullibility of Personnel Managers. Personnel Psychology, 11(3), 347-352.

How to hire an attorney

What's the best way for an organization to hire an attorney with little job experience? What should they look for? LSAT scores? Law school grades? Interviewing ability? A multi-year project that issued its final report in 2008 gives us some guidance. And while the study focused on ways law schools should select among applicants, it's also instructive for the hiring process. (By the way, individuals looking for personal representation may find the following interesting as well.)

Recall that the formalization of the "accomplishment record" approach occurred in 1984 with a publication by Leaetta Hough. She showed, using a sample of attorneys, that scores using this behavioral consistency technique correlated with job performance but not with aptitude tests or grades, and showed smaller ethnic and gender differences.

But in my (limited) experience, many hiring processes for attorneys have consisted of a resume/application, writing sample, and interview. Is that the best way to predict how well someone will perform on the job?

Assessment research would strongly point to cognitive ability tests being high predictors of performance for cognitively complex jobs. This is at least part of the logic of hurdles like the Law School Admissions Test (LSAT), a very cognitively-loaded assessment. When you're at the point of hire, however, LSAT scores are relatively pointless. Applicants have--at the very least--been through law school, and may have previous experience (such as an internship) you can use to determine their qualifications.

So what we appear to have at the point of hire is a mish-mash of assessment tools, relying heavily on un-proven filters (e.g., resume review) followed by a measure of questionable value (the writing sample) and the interview, which in many cases isn't conducted in a structured way that would maximize validity.

So what should we do to improve the selection of attorneys (besides using better interviews)? Some research done by a psychology professor and law school dean at UC Berkeley may offer some answers.

The investigators took a multi-phase approach to the study. The first part resulted in 26 factors of lawyer effectiveness--things like analysis and reasoning, writing, and integrity/honesty. In the second phase they identified several off-the-shelf assessments they wanted to investigate for usefulness, and they developed three new assessments--a situational judgment test (SJT), a biodata measure (BIO), and other measures, including optimism and a measure of emotional intelligence (facial recognition). In the final phase, they administered the assessments online to over 1,000 current and former law students and looked at the relationship between predictors and job performance (N for that part of about 700, using self, peer, and supervisor ratings).

Okay, so enough with the preamble--what did they find?

1) LSAT scores and undergraduate GPA (UGPA) predicted only a few of the 26 performance factors, mainly ones that overlapped with LSAT factors such as analysis and reasoning, and rarely higher than r=.1. Results using first-year law school GPA (1L GPA) were similar.

2) The scores from the BIO, SJT, and several scales of the Hogan Personality Inventory predicted many more dimensions of job performance compared to LSAT scores, UGPA, and 1L GPA.

3) The correlations between BIO and SJT and job performance were substantially higher-- in the .2-3 range compared to LSAT, UGPA, and 1L GPA. The BIO measure was particularly effective in predicting a large number of performance dimensions using multiple rating sources.

3) In general, there were no race and gender subdifferences on the new predictors.

These results strongly suggest that when it comes to hiring attorneys with limited work experience, organizations would be well advised to use professionally developed assessments, such as biodata measures, situational judgment tests, and personality inventories, rather than rely exclusively on "quick and dirty" measures such as grades and LSAT scores. Yet another proof of the rule that the more time spent developing a measure, the better the results.


On a final note, several years back I did a small exploratory study looking at the correlation between law school quality and job performance. I found two small to moderate results: law school quality was positively correlated with job knowledge, but negatively correlated with "relationships with people."

References:
Here is the project homepage.
You can see an executive summary of the final report here.
A listing of the reports and biographies is here.
The final report is here.

September mega research update

I'm behind in posting updates of the September journals. Really behind. So instead of posting a series describing the detailed contents of each issue, I'm going to give you links to what's come out in the last month or so and let you explore. But I'll try to hit some high points:

Journal of Applied Psychology, v95(5)

Highlights include this piece from Ramesh and Gelfand who studied call center employees in the U.S. and India and found that while person-job fit predicted turnover in the U.S., person-organization fit predicted turnover in India.


Human Performance, v23(4)

This issue has some good stuff, including Converse et al.'s piece on different forced-choice formats for reducing faking on personality measures, Perry et al.'s piece on better predicting task performance using the achievement facet of conscientiousness, and Zimmerman et al.'s article on observer ratings of performance.


Journal of Business and Psychology, v25(3)

Another issue filled with lots of good stuff, but I'm almost 100% positive abstract views are session based, so use the title link for article on response rates in organizational research (full text here), the importance of using multiple recruiting activities, and the importance of communicating benefit information in job ads.


Journal of Applied Social Psychology, v40(9)

The article to check out here is by Proost et al. and deals with different self-promotion techniques during an interview and their effect on interviewer judgments.


Journal of Occupational and Organizational Psychology, v83(3)

Check this issue out for articles on organizational attraction, communication apprehension in assessment centers, and the impact of interviewer affectivity on ratings.

Every once in a while, an idea comes along...

Once in a while a research article comes along that revolutionizes or galvanizes the field of personnel assessment. Barrick & Mount's 1991 meta-analysis of personality testing. Schmidt & Hunter's 1998 meta-analysis of selection methods. Sometimes a publication is immediately recognized for its importance. Sometimes the impact of the study or article isn't recognized until years after its publication.

The September 2010 issue of Industrial and Organizational Psychology contains an article that I believe has the potential to have a resounding, critical impact for years to come. Will it? Only time will tell.

The article in question is by Johnson, et al. and is on its face a summary of the concept of synthetic validation and champions its use. As a refresher, synthetic validation is the process of inferring or estimating validity based on the relationship between components of a job and tests of the KSAs needed to perform those components. It differs from traditional criterion-related validation in that the statistics generated are not based on a local study of the relationship between test scores and job performance. Studies have shown that estimates based on synthetic validity closely correspond to local validation studies as well as meta-analytic VG estimates. Hence it has the potential to be as useful as criterion-related validation in generating estimates of, for example, cost savings, without requiring the organization to actually gather sometimes elusive data.

But the impact of the article, if I'm right, will not be felt based on its summary of the concept, but on what it proposes: a giant database containing performance ratings, scores from selection tests, and job analysis information. This database has the potential to radically change how tests are developed and administered. I'll let the authors explain:

"Once the synthetic validity system is fully operational, new selection systems will be significantly easier to create than with a traditional validation approach. It would take approximately 1-2 hours in total; employers or trained job analysts just need to describe the target job using the job analysis questionnaire. After this point, the synthetic validity algorithms take over and automatically generate a ready-made full selection system, more accurately than can be achieved with most traditional criterion-related validation studies."

Sound like a mission to Mars? Maybe. But the authors are incredibly optimistic about the chances for such a system, and it appears that it is already in the beginning stages of development. The commentaries following this focal article are generally very positive about the idea, some authors even committing resources to the project. The authors respond by suggesting that SIOP initiate the database and link it to O*NET. They point out, correctly, that this project has the potential to radically improve the macro-level efficiency of matching jobs to people; imagine how much more productive a society would be if the people with the right skills were systematically matched with jobs requiring those skills.

So as you can probably tell, I think this is pretty exciting, and I'm looking forward to seeing where it goes.

I should mention there is another focal article and subsequent commentaries in this issue of IOP, but it's (in my humble opinion) not nearly as significant. Ryan & Ford provide an interesting discussion of the ongoing identity crisis being experienced by the field of I/O psychology, demonstrated most recently by the practically tie vote over SIOP's name. I found two things of particular interest: first, the fact that they come out of the gate using the term "organizational psychology" which deserves only a footnote (a fact pointed out by several commentary authors). Second, they take an interesting approach to presenting several possible futures for the field, from the strengthening of historic identity to "identicide."

Finally, I want to make sure everyone knows about the published results of a major task force that looked at adverse impact. It too has the potential to have a significant impact on the study and legal judgment of this sticky (and persistent) issue.

Personnel Psychology, August 2010

The August, 2010 issue of Personnel Psychology came out a while ago, so I'm overdue in taking a look at some of the content:

Greguras and Diefendorff write about their study of how "proactive personality" predicts work and life outcomes. Using data from 165 employees and their supervisors across three time periods, the authors found that proactive individuals were more likely to set and attain goals, which itself predicted psychological need satisfaction. It was the latter that then predicted job performance and OCBs as well as life satisfaction.

Speaking of personality, next is an interesting study by Ferris et al. that attempts to clarify the relationship between self-esteem and job performance. Using multisource ratings across two samples of working adults, the authors found that the importance participants placed on work performance to their self-esteem moderated this relationship. In other words, this suggests that whether self-esteem predicts job performance depends on the extent to which people's self-esteem exists outside of their performance. Interesting.

Lang et al. describe the results of a relative importance analysis of GMA compared to seven narrower cognitive abilities (using Thurstone's primary mental abilities). Using meta-analysis data, the authors found that while GMA accounted for between 10 and 28% of the variance in job performance, it was not consistently the strongest predictor. Add this study to a number of previous ones suggesting that one solution to the validity-adverse impact dilemma may be in part to use narrower cognitive abilities (e.g., verbal comprehension, reasoning).

Last but definitely not least, Johnson and Carter write about a large study of synthetic validity (a topic Johnson writes more about in the August issue of IOP). For those that need a reminder, synthetic validity is the process of inferring validity rather than directly analyzing predictor-criteria relationships. After analyzing a fairly large sample, the authors found that synthetic validity coefficients were very close to traditional validity coefficients--in fact within the bounds of sampling error for all eleven job families studied. Validity coefficients were highest when both predictors and criterion measures were weighted appropriately.

So what the heck does that mean? Essentially this provides support for employers (or researchers) who lack the resources to conduct a full-blown criterion validation study but are looking for either (a) a logical way to create selection processes that do a good job predicting performance, or (b) support for said tests. Good stuff.

Catbert tackles HR initiatives

In honor of Labor Day in the U.S., let's take a humor break from research and high-tech developments. In case you're not a regular Dilbert reader, Catbert (Evil Director of Human Resources) has recently gotten involved in three popular HR initiatives, with varying levels of success:

Workforce skill assessment ("strengths" fans take note)

Internal promotions

Employee surveys

September 2010 IJSA (those considering SHRM certification, read on)

The September issue of the International Journal of Selection and Assessment (IJSA) is out with a boatload of content. Let's check out some of the highlights:

First up, a piece by Gentry, et al. that has implications for self-rating instruments. The authors studied self-observer ratings among managers in Southern Asia and Confucian Asia and found an important difference: the discrepancy between the ratings was greater in Southern Asia. Specifically, the difference appears in self-ratings rather than observer ratings, indicating differences in how managers in the different areas perceived themselves. Implication? Differences in self ratings may be due to cultural differences in addition to things like personality and instrument type.

The second article is a fascinating one by Saul Fine in which the author analyzed differences in integrity test scores across 27 countries. Fine found two important things: first, there are significant differences in test scores across countries. Second, test results were significantly correlated (r= -.48) with country-level measures of corruption as well as several aspects of Hofstede's cultural dimensions.

Next, an article by De Corte, et al. that describes a method for creating Pareto-optimal selection systems that balance validity, adverse impact, and predictor constraints. This article continues the quest for balancing utility and subgroup differences. A link to the article is here but it wasn't functional at the time I wrote this; hopefully it will be soon.

Next, in an article that SHRM will probably place on their homepage if they haven't already, Lester et al. studied alumni from three U.S. universities to analyze the relationship between attainment of the Professional in Human Resources (PHR) certification offered by SHRM and early career success. Results? Those with a PHR were significantly more likely to obtain a job in HR (versus another field) BUT possession was not associated with starting salary or early career promotions. I'll let you decide if you think it's worth the time (and expense).

If you need another reason to focus on work samples and structured interviews, here ya go. Anderson, et al. provide us with the results of a meta-analysis of applicant reactions to selection instruments. Drawing from data from 17 countries, the authors found results similar to what we've seen in the past: work samples and interviews were most preferred, while honesty testing, personal contacts, and graphology were the least preferred. In the middle (favorably evaluated) were resumes, cognitive tests, references, biodata, and personality inventories.

Fans of biodata and personality testing may find the article by Sisco & Reilly reassuring. Using results from over 700 participants, the authors found that the factor structures of a personality inventory and biodata measure were not significantly impacted by social desirability at the item level. Implication? The measures seemed to hold together and retain at least an aspect of their construct validity even in the face of items that beg inflation.

Speaking of personality tests, Whetzel et al. investigated the linearity of the relationship between the OPQ and job performance. Results? Very little departure from linearity and where present the departure was small. This suggests that utility gains may be obtained across the spectrum of personality test results.

Are you overloading your assessment center raters? Melchers et al. present the results of a study that strongly suggests that if you are using group discussions as an assessment tool, you need to be sensitive to the number of participants that raters are simultaneously observing.

There are other articles in here you may be interested in, including ones on organizational attractiveness, range shrinkage in cognitive ability test scores, and staffing services related to innovation.

The personality echo

Psychologists have known for a while about something called perceiver effects, which refer to general tendencies to judge others in a particular way. For example you may tend to see people as generally self-serving or selfless, open-minded or closed minded, etc.

It turns out that these perceiver effects say something about you. For example, one of the ways Machiavellianism is measured is by asking whether you generally see a lack of sincerity or integrity in others. In a sense, the judgments you make about others echo back and, when interpreted properly, can say something about your personality.

In the July 2010 issue of the Journal of Personality and Social Psychology, Wood, et al. describe the results of several studies of this phenomenon. Most of the previous studies have used an "assumed similarity" paradigm, where the researchers have attempted to confirm that how one views themselves is assumed to transfer to how one views others.

This study, on the other hand, made no such assumptions. The researchers were interested in what impact self-ratings of personality had on perceptions, regardless of whether it was the same trait. The primary relationship they looked at was the correlation between how one scored on a personality inventory and how they tended to rate others.

In three different studies of college students, the strongest trend was related to agreeableness: those that rated others high in agreeableness tended to rate themselves high in the same trait (r's of .19 to .29 depending on the sample).

The second highest was conscientiousness, and in the same direction, but for different traits: those that rated others high in conscientiousness tended to rate themselves high in agreeableness (r's varied from .10 to 25). Rating others high in openness was also associated with higher self-rating scores of agreeableness (r's from .12 to .27).

Interestingly, in the third study the authors also identified moderate positive correlations between perceiving others in a positive light and several individual characteristics, including self-rated agreeableness, fit with peers, and organizational goals, and negative correlations with need for power, social dominance orientation, and depression.

So what does this mean? Essentially this study suggests that if someone (say, a job applicant) tends to describe others in a positive light--specifically, as agreeable, conscientious, and open--there is a significant chance that they themselves will rate highly on agreeableness. Anyone that's ever interviewed someone who makes negative stray marks about previous co-workers likely has intuited this.

There are several important caveats:

1) The effect sizes (correlations) were modest

2) The sample was restricted to university students

3) There are likely important differences between jobs, impacting both the value of agreeableness as well as how these attitudes are formed.

By the way, an in press version is available here.

Generational differences in work values: Fact or fiction?

There's been a lot written over the years about so-called "generational differences" among different age groups--e.g., Baby Boomers, Gen X, and GenMe (i.e., Gen Y, Millennials, Net Generation). Various authors have claimed important differences, such as GenMe valuing altruistic employers and social experiences much more than, say, Boomers. Take a look at the business section of your local bookstore and you're sure to find examples.

The problem isn't with the issue--if there truly are differences, they would have important implications for attracting and retaining different segments of the workforce. The problem is with the data.

Turns out most of the previous research is either qualitative (anecdotes) or based on cross-sectional studies done at a single point of time. The problem with these studies is they make it impossible to separate generational differences from career stage differences. In other words, younger applicants/employees may indeed differ from older ones at any point in time--but that could be purely due to factors that impact one's age, not factors related to being born or experiencing a particular point in time.

Luckily for us, in the September issue of the Journal of Management, Twenge, et al. report the results of a longitudinal study that allows us to answer these generational questions with some authority.

The authors used data collected from a nationally representative sample of over 16,000 U.S. high school seniors taken in 1976, 1991, and 2006 (from the Monitoring The Future project).

The results may surprise you. Let's look at each of the work values studied by the authors:

Leisure (e.g., vacation, work-life balance): This became progressively more important over the generations, with GenMe valuing it the most. The difference between GenMe and Boomers was the largest reported in the study (d>.50).

Intrinsic (e.g., interesting and challenging work): While GenY did not differ significantly from Boomers, GenMe were significantly less likely to value this compared to either GenY or Boomers.

Altruistic (e.g., ability to help others and society): While it's commonly reported that GenMe values this more highly than previous generations, results did not support this. No significant differences were found between the three groups.

Social (e.g., job gives feeling of belonging and being connected): This is another area where some have suggested that with the skyrocketing success of sites like Facebook, the younger generations more highly value--indeed, insist on--a workplace that allows social interaction. The results? Not so much. In fact, GenMe placed less value on this compared to both GenY and Boomers.

Extrinsic (e.g., job pays highly or is prestigious). This is an interesting example of non-linearity. Turns out this value peaked with GenY. While GenMe valued this more than Boomers, the difference was more pronounced between Boomers and GenY.

Among the items with the biggest differences were:

- GenMe valuing having 2+ weeks of vacation compared to Boomers
- Boomers valuing a job that allows you to make friends compared to GenMe
- GenY valuing having a job with prestige/status compared to Boomers
- GenMe reporting that work is just a way to make a living compared to Boomers
- GenX valuing being able to participate in decision making compared to Boomers

Overall, intrinsic reward items had the highest means across the generations, with a job that is "interesting" having the highest item mean. Altruistic and social values were also valued more highly, with extrinsic rewards having lower mean values and leisure rewards having the lowest.

The authors summarize the results by saying the data suggest "small to moderate generational differences." If you aren't surprised, kudos to your observational skills. At the very least this is important data to consider when evaluating your recruiting and retention efforts. And it certainly calls into question some of the conclusions being drawn in the popular press.

By the way, a full version of the article is available (at least it was at the time I published this) here.

Ignore Facebook

Somewhere there is a team--or an individual--that has been tasked with figuring out "how to use this Facebook thing" to assist with recruitment and assessment. Management doesn't know what they want--they may not even know what Facebook is--but they're convinced the organization needs to use it.

For most organizations this is a complete waste of time.

Yeah, yeah, I know a study just came out from Nielsen showing that Americans spend nearly a quarter of their online time on social networking or blogs (an odd combination if you ask me), up 43% from a year ago, and I know Facebook recently announced they surprised 500 million users worldwide.

But just because something's popular doesn't mean it has direct application in all walks of life. Just because the Kindle is hot doesn't mean you should use it as a cutting board.

This is a classic mis-direction. It's like focusing on re-painting the guest room because brown is the new blue, while meanwhile the rest of your house falls apart.

Don't get me wrong, I'm all about innovation in our field, and longtime readers know I'm no hater of online developments. But we should be spending our most valuable commodity (time) where it counts.

What we should be doing is learning from this shift in online behavior and applying those lessons to our existing efforts. What the heck does that mean? Well, let's look at what people are using sites like Facebook for:

1. Sharing their life. Perhaps surprisingly, a lot of people like sharing their lives publicly, and for many, status updates are the 2010 version of the family letter. Think about the extent to which you allow applicants to share themselves. Does your application process consist only of a resume upload, or do you allow them to upload cool things they've done? Do candidates have the ability to customize their job search based on their background, not your jobs?

2. Connecting with people. There at least three direct applications here: (1) think about the extent to which your career portal allows applicants to contact a real human being; (2) think about the possibility of starting an internal network (e.g., Yammer) to capture all that knowledge sharing, bench strength, and informal connection goodness; and (3) consider publishing employee profiles online so potential candidates know who they'd be working with.

3. Playing games. I've written several times over the years about the benefits of interactive and entertaining recruitment and assessment applications. Video and online games aren't just for nerds anymore (see Farmville, the Wii). Think about how you can make your career portal not only attractive and useful--but FUN. Or at the very least engaging.

Maybe the biggest lesson here: your reputation is out there more than ever. If you don't think your employees are updating/blogging/tweeting about their jobs, you're mistaken. And you may have the best sourcers and recruiters in the world, but if your reputation as an employer stinks, it's all over the web and you've just made their job 10x harder. Know how your employees feel. Work hard on improving their work life. Reward good performance and deal with lack thereof.

Don't let the hype about Facebook get to you. Take your time and identify what you really need to be focusing on to increase your talent pipeline.

Disagree? Think we should be spending more, not less, time with these technologies? Bring it. (ya know, with the comments feature)

Human Performance, v23 (3)

Three interesting articles in the most recent issue of Human Performance:

Kanar, et al. studied the impact of organizational reputation on applicant attraction with a sample of college-level job seekers and found that negative information had a greater impact than positive information on attraction to the organization as well as recall, and this effect persisted for one week. Implication? Recognize that negative information about your organization may be processed differently than positive information, and thus not easily balanced. Also reinforces keeping an eye out for negative remarks and addressing them.

Second, a fascinating study by Kell, et al. on how Big 5 personality factors differentially predict various aspects of performance within a single job. Specifically, the authors found (as judged by raters) that emotionally stable and conscientious actions were more effective in task situations and open and agreeable actions were more effective in interpersonal situations. Implications? Not only does it seem that different assessment types predict different general aspects of performance (i.e., personality measures usually are better at predicting contextual performance), there appears to be prediction differences within the same job such that there is value in considering each facet of the Big 5 and its relationship to different aspects of performance.

Lastly, if you're looking for a way to predict performance in jobs that require "multi-tasking", you'll be interested in Poposki & Oswald's description of the development of the Multitasking Preference Inventory. In addition to the development, the authors describe a study of its convergent and discriminant validity as well as initial criterion-related validity. I found it interesting that this is not an ability test but rather a preference inventory, which actually makes sense since the brain doesn't actually focus on multiple things very well!

Learning from orchestra hiring

Lessons come from all sorts of places. Recently an article in the New York Times described the surprising number of vacancies nationwide in major symphony orchestras, including the N.Y. Philharmonic, which will have an unprecedented 12 vacancies next season (12% of its "workforce").

Here are some things that stood out from the article:

1) The customers (audience) may not notice. Top seats do not go vacant because assistant principals step up or substitutes are hired.

2) Economic problems have impacted an incredibly wide breadth of industries. Some positions are left vacant because it is cheaper to do so.

3) Leaders sometimes defer the decision. Some conductors have passed on filling positions to give their successor an opportunity to make the pick.

4) The selection process is resource intensive and difficult to coordinate. This is primarily due to the difficulty in getting the conductor and a committee of players (let's call them SMEs) together to do the judging.

5) Sometimes no decision is made. Even after the months of advertising and auditioning a suitable person isn't found. Kudos to those organizations with the wisdom to pass, assuming a valid selection process.

6) Which raises the next point--the selection process is suspect at best. Why? Well, aside from the historical gender discrimination, the process used to select musicians relies upon a single hour-long work sample test. What if the person is having a bad day? Perhaps more importantly, the most important criteria--symphony performance--relies upon all the players working together.

7) On a related point, applicants aren't being judged solely on technical proficiency. Committee members also judge them on fit, or as one described it, they look for "thinking, thoughtful musicians who are the whole package". What does that mean? It's hard to say, since, as stated in the article, "orchestra officials and musicians are loath to discuss the auditioning process in detail." Here's a question: why? Afraid that the "correct answers" will get out there--or afraid of the critical eye that may be focused on them?

Hat tip.

July 2010 J.A.P.

A new round of journals is out, so let's start with the June issue of the Journal of Applied Psychology.

First up, Schleicher et al. looked at whether there were demographic differences in how much candidate scores improved upon retesting. Turns out there were several. Whites showed larger improvements than Blacks or Hispanics on several assessments, particularly on written tests. Women and applicants under 40 showed greater improvements than men and applicants 40+. Implications? In some situations allowing applicants to retest may exacerbate adverse impact.

Next, an important piece by Aguinis et al. (that you can read here) about test bias. This follows on the heels of the June IOP articles on the same topic and seems to represent a resurgence of interest in a topic that seemed dormant. In this article the authors report the results of a very large Monte Carlo simulation (billions and billions of data points) where they found that if bias is measured using slope-based techniques, it's likely to go undetected, and intercept-based bias favoring minority group members is likely to be found when in fact it does not exist. This study, combined with points made in the IOP article suggest that some of the "established" conclusions regarding test bias may not be as solid as we thought.

Third, for those of you interested in differential functioning (of items or scales), you should check out the piece by Adam Meade where he presents a taxonomy of potential differential functioning effect sizes and also describes a software program created for computing the indices and graphing differential functioning.

Next, a piece by Wang et al. on locus of control. Importantly, they found that when locus of control (LOC) is specific to work-related issues, there are stronger correlations between LOC and work-related criteria such as job satisfaction and commitment. Similarly, when LOC is defined more broadly to include non-work issues, there are some stronger correlations with non-work criteria such as life satisfaction. Implications? Much like research on personality items, specifying a work-related context would seem to increase the predictive power of LOC measures.

Last but not least, an important article on counterproductive work behavior (CWB) and organizational citizenship behavior (OCB) by Spector, et al. CWB and OCB seem like they should be opposites of each other--one demonstrated by disengaged, unhappy workers, the other by engaged, happy ones--right? Not so fast. The authors report the results of an experiment that suggest that the concepts are unrelated and do not necessarily have opposite relationships with other variables. The authors also recommend that when measuring these behaviors, frequency of performance be used rather than level of agreement.

Unvarnished Unwrapped

A few months ago I mentioned a website called Unvarnished, which was getting a lot of mixed press. The basic concept is (as they describe it) Yelp mixed with LinkedIn. You provide anonymous reviews of people you've worked with. Call it a social resume, call it a web-based reference, I call it fascinating. And something that anyone interested in recruitment and assessment should pay attention to.

I had a chance recently to test out the site, and then I met with the co-founder, Peter Kazanjy. I'm still not sure which direction this will go, but I think you'll agree after reading what follows that the concept merits our attention.

The Test Drive
First, the test drive, which started with an invitation through Facebook from Peter. After spending some time on the site, I'm more optimistic about Unvarnished in some respects, more cautious in others.

Why optimism? The site hits it out of the ballpark on two accounts: it's simple and fast (at least someone learns from Google). Simple and fast is good, because one of the biggest challenges will be building a large community and making reviews easy helps immensely.

Even better, the ratings are relevant. This isn't a popularity contest, it's an honest attempt to provide a useful description of someone's performance. While it's unlikely the rating scales were developed after reading a Personnel Psychology meta-analysis, I was pleased to discover that they pass the smell test and some even have benchmarks.

The ratings consist of an overall performance rating (5-point, anchors described), what job you are rating the person in, four 10-point scales that are described but not anchored (skill, relationships, productivity, integrity), and an open-ended strengths/areas for improvement box. That's it. It's like a super-basic reference check form that takes all of about a minute. You can see what this process looks like below.

Your Unvarnished homepage is fed by your network and generates suggestions for your review (PMYK--People You Might Know). You can review people at any time (even people who haven't claimed a profile) or request reviews using your Facebook contacts. The open comments section is limited to 500 characters to encourage people to review and move on.

The site was developed to be heavily reliant on algorithms. A reviewer's reputation is based in part on the pattern of reviews they have generated as well as how their reviews have been rated. Recommendations for reviews are backed by similar math. A smaller (but important) feature is a profanity filter, which may allay some concerns regarding people looking to settle a score.

Speaking of Facebook, one of my major concerns is it relies HEAVILY on Facebook (not unlike Quora). At least in its current iteration, your identity is verified through having a Facebook account, you invite people through your Facebook contacts, and invitations are posted to Facebook. This is both a potentially good thing (e.g., cuts down on spammers) as well as a bad thing (e.g., not everyone wants to use their Facebook profile in the service of another website). It also begs the question of what would happen if Facebook went belly up. Read on to see what Peter had to say about this.

The interview

I happened to be in San Francisco (interestingly enough to go the Exploratorium) and had an opportunity to talk with Peter about the product and his company. After talking to him for about 10 minutes, one thing became abundantly clear: this guy has thought a great deal about online reputation management. I'd love to get him and Bob Hogan together.

Background
The impetus for the site came from a couple directions. One was a previous job where he was continually surprised that competitors (e.g., a certain company we'll call Bicrosoft) would recruit away the lesser talented employees. Why? His theory is they lacked important information--namely the reputation people had within the organization. (One could argue that they should have done better reference checks, but we all know how easy/productive those can be)

On the other end of the spectrum, he saw excellence not being properly recognized and questioned whether upper management really knew what their talent looked like (a point not lost, btw, on purveyors of performance management software with their 9-box grids and helicopter views).

His ah-ha moment (or one of them) came when he realized that if social ratings can work for things like books and software, couldn't they work for people? If he could develop a site that aggregated high quality information about people's performance, talent decisions would be higher quality as well as more fair (it's hard to argue with that goal).

Peter also believes there is an important employer segment not being served by existing background/reference checking processes. Employers that hire hourly workers rely largely on criminal/credit checks. Those hiring for executive-level positions often rely on high cost search firms. But for employers hiring large number of employees in the middle, there isn't really a good option.

Such a site would have three primary users: those being reviewed, those providing reviews, and those using the information (e.g., employers checking out candidates and vice-versa). And quite commonly a single individual could be in all three roles at various times. The site would have to accommodate all three perspectives.

Concerns/criticism
So what about my concerns? When it comes to the reliance on Facebook, Peter pointed out that it's a good bet Facebook will be around for a while, but the site is not being built to rely solely upon it. It has--or will have--the ability to use contact information from sites like Gmail and LinkedIn. I still have a concern about forcing people through Facebook, so it will be interesting to see whether this impacts the ability to generate reviews.

Concern from others has focused on the potential for abuse. But Peter made several important points. First, this isn't like an online newspaper comment space--those are anonymous with no repercussions for inaccuracy. In Unvarnished your reputation would suffer and your reviews become less valid (assuming your reviews are themselves reviewed). Second, this information is largely out there (or potentially at least) in the form of places like Twitter. But it's not in a central location that can be easily managed, and it's not objective. By guarding invites and relying on anonymity, the goal is to make legitimate reviewers feel safe in leaving honest feedback (whether it's an A+ or a D-) without worrying about the interpersonal implications.

What about stickiness--why would someone want to keep coming back? They're working on several (re)engagement initiatives. One idea is to provide people with periodic updates letting them know when people in their network have been updated (let's hope it doesn't turn into those updates from LinkedIn that are easy deleted). They're also working on the ability to follow an individual based on your "gestures" (e.g., you've reviewed them).

Future directions
The team is considering adding several features. One is the ability for reviewers to better define their relationship--e.g., describing not only the organization they worked together in but their relationship. This would factor into behind-the-scenes algorithms but would not be published.

They're also discussing allowing people to identify themselves, but this raises all of the associated issues such as accuracy, thoroughness, and feeling a need for reciprocity.

Another is allowing access to premium features (e.g., at a cost) such as making trusted reviews more obvious, something that "super users" such as recruiters would likely be willing to pay for.

In terms of opening up access, they're in no big rush to expand access beyond Facebook invites. While this may hinder their growth, it helps keep the data quality high, and they're willing (smartly, I think) to make this trade off.

Conclusion
Currently the company is focused on acquiring the talent it needs to succeed (check out the way they recently advertised for new engineers). One of Peter's primary concerns is that the community evolve in the right direction. Right now it's somewhat of a "love fest" with lots of positive reviews. The site will gain in usefulness when reviews are a combination of pros and cons.

Given what we know about performance ratings, it will be interesting to see if the existing invitation and rating process is sufficient to generate that depth. It also remains to be seen whether some type of incentive will need to be given to generate reviewers.

My overriding concern continues to be the size and diversity of the user group (right now primarily filled with Silicon Valley IT folk). Accuracy, something other writers have been obsessed with, is less of a concern for me after kicking the tires and talking with Peter. But we'll see how the promise and concerns ebb and flow with the user base as well as changes to the service.

At the very least, I hope you'll agree with me that this website is a fascinating development and one we should watch. After all, you might want to use Unvarnished to provide feedback on someone. Or research a potential boss. Or research an applicant. Or...you may be the applicant.

Three on EI

A couple of posts ago I wrote about the most recent issue of IOP and the focal article on emotional intelligence (EI).

Now there's another meta-analysis out by O'Boyle et al. in JOB which may add some support to fans of EI. Here are the main findings:

1) Corrected correlations of between .24 and .30 with job performance.

2) The three "streams" of measures (ability, self- or peer-report, and "mixed models") correlated differently with cognitive ability and personality measures.

3) Perhaps most interestingly, self-/peer-report and "mixed models" had the most incremental validity beyond cognitive ability and personality.

More evidence that these measures--whatever they're measuring--are correlated with job performance, and they seem to be picking up something new. But do we know what's being measured? And what aspect of job performance is being predicted (contextual rather than task performance seems likely)? And what about other considerations, such as face validity/applicant reaction?

For those of you into EI, here are a couple articles that you may have missed:

Mayer et al.'s overview in the 2008 issue of Annual Review of Psychology (PDF; full text)

Cote and Miners' 2006 piece in Administrative Science quarterly (PDF; full text).

June '10 IOP Part II: "Test bias"

In Part I of this two-part post, I described the first focal article in the June 2010 issue of Industrial and Organizational Psychology (IOP), devoted to emotional intelligence. In this post, I will describe the second focal article, which focuses on test bias.

The focal article is written by Meade & Tonidandel and has at its core one primary argument. In their words:

"...the most commonly used technique for evaluating whether a test is biased, the regression-based approach outlined by Cleary (1968), is itself flawed and fails to tell us what we really want to know."

The Cleary method of detecting test bias is conducted by regressing a criterion (e.g., job performance) on test scores along with a dichotomous demographic grouping variable, such as sex or race, and looking at the interaction. If there is a significant difference by group variable (e.g., different slope, different intercepts), this suggests the test may be "biased". This situation is called differential prediction or predictive bias. The authors contrast this with "internal methods" of examining tests for bias, such as differential item and test functioning and confirmatory factor analysis which do not rely upon any external criterion data.

Meade & Tonidandel state that while the Cleary method has become the de facto method for evaluating bias, there are several significant flaws with the approach. Most critically, the presence of differential prediction does not necessarily mean the test itself is the cause (which we would call measurement bias). Other potential causes are:

- bias in the criterion
- reliability of the test
- omitted variables (i.e., important predictors are left out of the regression)

Another important limitation of the Cleary method is the susceptibility of slope difference tests to low power--this can result in findings of no slope differences due to small samples rather than a true absence. In addition, because of the type I error rate of the intercept test, one is likely to conclude that intercept differences are present when none truly exist.

Because of these limitations, the authors recommend the following general steps when investigating a test:

1. Conduct internal analyses examining differential functioning of items and the test.
2. Examine both test and criterion scores for significant group mean differences before conducting regression analyses.
3. Compute d effect size estimates for group mean differences for both the test and the criterion.

The authors present several scenarios where tests may be "biased" as conceived in the traditional sense but may or may not be "fair"--an important distinction. For example, one group may have higher performance scores, but there is no group difference in test scores. Use of the predictor may result in one group being hired at greater or lesser rates than they "should be", but our judgment of the fairness of this requires consideration of organizational and societal goals (e.g., affirmative action, maximum efficiency) rather than simply an analysis of the tests.

The authors close with several overall recommendations:
1) Stop using the term test bias (too many interpretations, confounds different concepts).
2) Always examine both measurement bias and differential prediction.
3) Do not assume a test is unusable if differential prediction is indicated.

There are several commentary articles that follow the focal one. The authors of these pieces make several points, ranging from criticisms of techniques and specific statements to suggestions for further analyzing the issue. Some of the better comments/questions include:

- How likely is it that the average practitioner is aware of these issues and further is able to conduct these analyses? (a point the authors agree with in part)

- This approaches advocated mainly work with multi-item tests; things get more complicated when we're using several tests or an overall rating.

- It may not be helpful to create a "recipe" of recommendations (as listed above); rather we should acknowledge that each selection scenario is a different context.

- We are still ignoring the (probably more important) issue of why a test may or may be "biased". An important consideration is the nature of group membership in demographic categories (including multiple group memberships).

Meade & Tonidandel provide a response to the commentaries and acknowledge several valid points raised but end this article with the same proposition that they started the focal article with:

"For 30 years, the Cleary model has served as the dominant guidelines for analyses of test bias in our field. We believe that these should be revisited."

June '10 IOP Part I: Emotional Intelligence

The July 2010 issue of Industrial and Organizational Psychology has two very good focal articles with, as always, several thought-provoking commentaries following each. This post will review the first article and I'll cover the next in the following post.

The first focal article, by Cary Cherniss, attempts to provide some clarity to the debate over emotional intelligence (EI). In it, Cherniss makes several key arguments, including:

- While there is disagreement over the best way to measure EI, there is considerable agreement over its definition. Cherniss adopts Mayer, et al.'s definition of "the ability to perceive and express emotions, assimilate emotion in thought, understand and reason with emotion, and regulate emotion in the self and others."

- EI can be distinguished from emotional and social competence (ESC), which explicitly links to superior performance.

- Among the major extant models, the Mayer-Salovey-Caruso model (MSCEIT) represents EI, while the others (Bar-On; Boyatzis & Goleman; Petrides, et al.) primarily consist of ESC aspects. Importantly, this does not make the Mayer, et al. model "superior", just more easily classified as an ability measure.

- There are significant problems with all of the major measures of EI/ESC, such as convergent/discriminant validity and inflation (depending on the measure). According to Cherniss, "it is difficult at this point to reach any firm conclusions--pro or con--about the quality of the most popular tests of EI and ESC."

- Emerging measures, such as those involving multiple ratings (e.g., Genos EI), appear promising, although they may be more complex and expensive than performance tests or self-report inventories.

- Other, even more creative, measures hold even more promise. This includes video and simulation tests. This point was echoed in several commentaries. I've posted a lot about the promise of computer-based simulation testing, and EI may be one of the areas where this type of measurement holds particular promise.

- Context is key. I think this is one of Cherniss' most important points: the importance of EI likely depends greatly on the situation--i.e., the job and the "emotional labor" required to perform successfully. This point was also echoed in several commentaries. EI may be particularly important for jobs that require a lot of social interaction and influence, team performance, and in jobs that involve a high level of stress.

- Several studies have shown modest correlations between EI/ESC measures and job performance, but there are issues with many of them (e.g., student samples, questionable criteria). Newer meta-analyses (e.g., Joseph & Newman, 2010) suggest "mixed-model EI" measures may hold more promise in terms of adding incremental validity.

The commentaries provide input on a whole host of points which would be difficult to summarize here (I will say I enjoyed Kaplan et al.'s the most). Needless to say there is still an enormous amount of disagreement regarding how EI is conceptualized, measured, and its overall importance. Then again, as Cherniss and others point out, EI as a concept is in its infancy and this type of debate is both healthy and expected.

Perhaps most importantly, users of tests should exhibit particular caution when choosing to use a purported measure of EI as the scientific community has not reached anything close to a consensus on the appropriate measurement method. Of course pinpointing the exact moment when that occurs will be--as with all measures--a challenge.

Research update

Here are a few research articles you may have missed. Just links this time because...well, I'm really enjoying World Cup.

McGrath, et al.: Evidence for response bias as a source of error variance in applied assessment.
(May Psych. Bulletin)

Hulleman, et al.: A meta-analytic review of achievement goal measures: Different labels for the same constructs or different constructs with similar labels? (May Psych Bulletin)

Thornton & Potemra: Utility of assessment center for promotion of police sergeants. (March Public Personnel Management)

Bright: Why age matters in the work preferences of public employees: A comparison of three age-related explanations (March Public Personnel Management)

Thomas, et al.: Employee proactivity in organizations: A comparative meta-analysis of emergent proactive constructs (June JOOP)

Bjørkelo, et al.: Predicting proactive behaviour at work: Exploring the role of personality as an antecedent of whistleblowing behaviour (June JOOP)

Zibarras & Woods: A survey of UK selection practices across different organization sizes and industry sectors (June JOOP)

What we can learn from the baseball draft

Longtime Major League Baseball commentator Peter Gammons was recently interviewed on National Public Radio. What does baseball have to do with recruitment and selection? Quite a bit, actually, but in this case the conversation was even more relevant because he was discussing the accuracy of the baseball draft.

I think you will agree with me after reading/listening to his observations that the draft has several lessons for all kinds of employers:

- More information is better. As scouts have been able to gather and crunch more data about prospects, the accuracy of predicting how a pick will fare in the big leagues has increased. We know from assessment research that more measures are better (up to a point) in predicting performance.

- The type of information matters. Scouts used to focus on relatively narrow measures such as running speed. Today they consider a whole host of factors. Similarly, modern assessment professionals consider a wide range of measures appropriate to the job.

- Personality matters. While lots of data about skills is important in prediction, personality/psychological factors also play a big role in determining success. Personality has also been one of the hottest topics in employment testing over the last 20-30 years.

- It's rare for a single candidate to shine head and shoulders over the rest. Making a final selection is usually a challenge.

- Assessment is imperfect. Even with all the information in the world, a host of other factors influence whether someone will be successful, including which team the person is on, their role, and how they interact with other teammates. This also means "low scoring" candidates can--and do--turn into superstars (Albert Pujols of the St. Louis Cardinals was a 13th-round draft pick).

- Ability to learn is important. We know from assessment research that cognitive ability shows the highest correlation with subsequent job performance (esp. for complex jobs), and many have suggested that it is the learning ability component of cognitive ability that matters most.

- Good recruitment and assessment requires resources. Organizations that take talent management seriously are willing to put their money where their mouth is and devote resources to sourcers, recruiters, and thorough assessment procedures.

By the way, 2009's first overall draft pick, Stephen Strasburg, had an excellent debut last night for the Washington Nationals, striking out 14.

May '10 J.A.P.

Summer journal madness continues with the May issue of the Journal of Applied Psychology. It's a diverse issue, check it out:

- Taras et al. conducted a very large meta-analysis of the association between Hofstede's cultural value dimensions (e.g., power distance, masculinity, individualism) and a wide variety of individual outcomes. One interesting finding is the stronger relationship between these values and emotions (organizational commitment, OCBs, etc.) compared to job performance.

- Are high performers more likely to stay or leave? In a study of over 12,000 employees in the insurance industry over a 3-year period, Nyberg found the answer was: it depends. Specifically, it depends on the labor market and pay growth.

- Think g (cognitive ability) is just related to job performance? In a (albeit small) study by Judge, et al., it turns out it was also related to physical and economic well-being. Maybe their next study will address my personal hypothesis: g is related to choice of car.

- A study by Lievens, et al. (in press version here) found with a sample of 192 incumbents from 64 occupations that 25% of the variance in competency ratings (like you might find in a job analysis) was due to the nature of the rater's job, such as level of complexity. Not surprisingly, the greatest consensus was reached for jobs that involved a lot of equipment or contact with the public.

- Self-efficacy (i.e.., confidence) has been proposed as an important predictor of job performance. In a study by Schmidt & DeShon, the authors found that this relationship depends on the ambiguity present in the situation--in situations high in ambiguity, self-efficacy was negatively related to job performance; in situations low in ambiguity, the opposite was true.

- Finally, for anyone citing Ilies, et al.'s 2009 study of the relationship between personality and OCB, there have been a couple corrections.

Lewis case emphasizes need for valid tests

On Monday, May 24, the U.S. Supreme Court ruled in Lewis v. City of Chicago that plaintiffs filing an adverse impact discrimination claim under Title VII of the Civil Rights Act have 300 days to file from each time the test is used to fill a position--not just 300 days from when the test was administered.

For the most part, this mainly impacts employers that run an exam and use the results to fill positions for several years. In this case it was a firefighter entry-level exam, and my guess is it will mostly be public sector agencies and large employers that should pay particular attention to the ruling.

Why? Well for one it means more potential adverse impact lawsuits. If you were counting on being safe after 300 days from the exam, that is no longer the case. Second, it emphasizes the need to follow professional guidelines when developing an exam. Employers can successfully defend against an adverse impact case by showing that the selection practice is "job related for the position in question and consistent with business necessity..." (and that no alternatives with similar validity with less adverse impact were available). This means your exams need to be developed and interpreted by people who know what they're doing.

The City claimed that evidence related to an employer's business necessity defense might be unavailable by the time the lawsuit is brought--the court was not swayed. This means you'll want to hang on to your exam development records for at least a year beyond the last time you use the results to fill a position.

Another point worth noting: this case boils down to the validity of a cut score used by the City, which they themselves admitted wasn't supportable. Proper exam development and interpretation includes setting a job-related pass point based on subject matter input and/or statistical evidence that it is linked to job performance.

The Lewis ruling doesn't fundamentally change what we should be doing. It just emphasizes that we need to do it right.

Click here for a quick overview of the facts of the case.

June 2010 IJSA

The summer journal season continues with the June 2010 issue of the International Journal of Selection and Assessment. Take a deep breath, there's a lot of stuff packed into this issue:

- Roth et al. provide evidence that women outperformed men on work sample exams that involved social skills, writing skills, or a broad array of KSAs. To the extent that an employer is trying to avoid discriminating against female applicants, this provides support for work sample usage.

- In a study of managers in Taiwan, Tsai et al. show that the most effective way an applicant can make up for a slip in an interview is to apologize (vs. attempting to justify or use an excuse).

- Jackson et al. strive to add some clarity on task-based assessment centers

- Blickle & Schnitzler provide evidence of the construct and criterion-related validity of the political skill inventory

- Colarelli et al. studied how racial prototypicality and affirmative action policies impact hiring decisions. Results of a resume review indicated more jobs were awarded to black candidates as racial prototypicality and affirmative action policy strength increased, but stronger AA policies decreased the percentage of minority hires attributed to higher qualifications.

- In my personal favorite article of the issue, Karl et al. found in a study of U.S. and German students that those low on conscientiousness (especially), agreeableness, and emotional stability were more likely to post "Facebook Faux Pas". This provides some support for employers who screen out applicants based on inappropriate social networking posts. I'll talk more about this in my upcoming webinar.

- Denis, et al. provide support for the NEO PI-R's ability to predict job performance in two French-Canadian samples.

- Bilgiç and Acarlar report results of a study of Turkish students and perceptions of various selection instruments. Interviews were rated most highly and there were some differences in terms of privacy perceptions depending on the goal orientation of the student.

- Trying to figure out how to hire better direct support professionals (e.g., those providing long-term residential care or care to those with disabilities)? Robson, et al. describe the development of a composite predictor composed of various measures (e.g., agreeableness, numerical ability) that predicted performance, satisfaction, and turnover.

- Ahmetoglu et al. provide support for using the Fundamental Interpersonal Relationship Orientations-Behaviour (FIRO-B) to predict leadership capability.

- Ispas et al. describe results of a study that showed support for a nonverbal cognitive ability measure (the GAMA) in predicting job performance in two samples.

- Last but not least, in another win for context-specific assessments, Pace & Brannick show how a measure of openness to experience tailored to specific work outpredicted the comparable general NEO PI-R scale. IMHO this is how personality measures will eventually become more prominent and accepted as pre-hire assessments.

IPAC conference to feature Campbell, McDaniel, Highhouse, and more

Those of you on the fence about attending the 2010 International Personnel Assessment Council (IPAC) conference on July 18-21 may be interested to know that a preliminary schedule has been released that reveals some great speakers and topics. For example:

- David Campbell's provocatively titled opening session, The Use of Picture Postcards for Exploring Diversity Issues Such as Bias and Prejudices, or "How Can We Keep Our Grandchildren From Going to War With Each Other?"

- Not to be outdone, Michael McDaniel kicks things off Tuesday morning with Abolish the Uniform Guidelines.

- Scott Highhouse closes things up Wednesday with A Critical Look at Holistic Assessment

- Great pre-conference workshops on everything from job analysis to fairness

- Wonderfully diverse concurrent sessions on topics such as public service motivation, leadership coaching, simulations, engagement, online testing, charging for exams, test transportability, cross-cultural personality assessment, measuring workforce gaps, adverse impact analysis, faking and lie detection, and succession planning. And that's just a sample!

Staying current on assessment through professional education is one of the commandments of our field. I hope you'll be joining your friends and colleagues in Newport Beach. Early bird registration ends June 1st.