Lessons from the NYC Fire case - part 1

Part 1 of 2

New York City, like the cities of New Haven and Chicago, has a long history of employment discrimination litigation related to its firefighter testing.

Since the 1970s and cases like Guardians, the city has been under scrutiny for its woefully low number of black firefighters.

In 2007 the city found itself faced with another lawsuit over its firefighter hiring practices, and in July of 2009, a U.S. District Court judge found that the city had violated Title VII by administering written exams from 1999-2007 that had high levels of adverse impact. The city marshaled an inadequate defense. In January of 2010, the same judge (Nicholas Garaufis) found the city liable for a pattern and practice of disparate treatment for those same exams. An adverse impact finding, particularly for written exams, and especially for public safety tests, is not earth-shattering. But a finding of disparate treatment in this situation is less common.

This case, while only one example and limited in its impact, has some valuable lessons for test users and sheds some light on how judges look at our field. In particular, I describe below nine points the judge specifically made and what lessons we can draw from them:

1) While the city conducted a job analysis with an "extensive" list of tasks and surveyed incumbents, the city offered "no evidence of 'the relationship of abilities to tasks.'" They conducted a linkage, but the judge found that the SMEs were confused about what they were supposed to do and didn't understand several of the abilities they were rating.

Lesson: simply having subject matter experts (SMEs) link essential tasks and knowledge, skills, and abilities (KSAs) is not sufficient. You need to ensure they understand the statements they are linking as well as how exactly they are supposed to be linking them.

2) In conducting the job analysis, the city inappropriately retained tasks and KSAs that could be learned on the job. It is quite clear (e.g., per the Uniform Guidelines) that only tasks and KSAs that are required upon entry to the job should be identified as critical in terms of exam development.

Lesson: make sure that when you are developing exams based on job analysis results that you focus only on those tasks and KSAs that are required upon entry to the job. This should be determined by your SMEs.

3) The city relied to some extent upon the work of a previous test developer, Dr. Frank Landy (who sadly recently passed away). In addition to a tenuous link between Dr. Landy's work and the current exams, the judge makes it clear that "reliance on the stature of a test-maker cannot stand in for a proper showing of validity." At the same time, the judge emphasizes that exams should be constructed by "testing professionals."

Lesson: tests should be developed by people who know what they're doing. This means HR professionals with the requisite background in test validation and construction in conjunction with job experts. Do not rely solely on previous efforts, particularly when (as in this case) the results of those efforts were either incomplete or not fully relevant to your current situation.

4) The city performed no "sample testing" to ensure that the questions were reliable as well as "comprehensible and unambiguous."

Lesson: few steps in the test development process are as easy--or as valuable--as pilot testing. I have yet to see an exam that didn't benefit from a "trial run" with a group of incumbents. Not only will you catch unintended flaws, you will verify that the exam is doing what you claim it is.

5) There was insufficient evidence that the exams actually measured the (nine cognitive) KSAs the city claimed they intended to measure. Plaintiffs were able to suggest the opposite through analyzing convergent and discriminant validity as well as by conducting a factor analysis.

Lesson: there are two linkages of primary importance in test development. The first was describe in #1. The second is the link between critical KSAs and the exam(s). At the very least, you must be able to show evidence that there is a logical link between the two. When you claim to be measuring cognitive abilities, you incur an additional responsibility, which is gathering statistical evidence that supports this claim.

Next time: more lessons and the relief order.

Jan '10 issue of JAP, plus APA gets stingy

The January 2010 issue of the Journal of Applied Psychology is out, and there are some good articles to take a look at. It just may be more difficult to see them. More on that in a minute.

First, here are some of the titles in this issue:

Emotional intelligence: An integrative meta-analysis and cascading model. A must for anyone interested in EI; posits and supports a cascading model whereby emotion perception-->emotion understanding-->emotion regulation-->job performance.

Time is on my side: Time, general mental ability, human capital, and extrinsic career success. GMA shown to have strong links to two extrinsic measures of career success, income and prestige. (ah, but are smart people happier?)

I won’t let you down… or will I? Core self-evaluations, other-orientation, anticipated guilt and gratitude, and job performance. Core self-evaluations' impact on job performance may depend on how much they focus on others.

Understanding performance ratings: Dynamic performance, attributions, and rating purpose. Performance ratings are influences by a variety of things, including overall performance variance and purpose of the ratings.


Okay, so back to my earlier comment: it appears that APA has restricted viewing abstracts of their journals to registered members (hence the lack of links in this post). On the one hand, no big deal, it appears you can simply register to gain access. On the other hand...why should someone have to do this? This is another unfortunate example of research being restricted (first by charging exorbitant fees for articles, now through personal identification) and contributes to the field being insular.

Granted, APA's not the only one that does this (hey buddy, got $400 for the CRL?) but that doesn't excuse it. Our field benefits from sharing of information, not just among professionals but with the general public. Requiring registration does not further that goal. Thankfully some individual researchers (see the sidebar on the main page) allow access to their work--something we should all be grateful for.

How to get r = 1.0

Recruiters have a variety of measures of their success, often including process outcomes (time-to-fill, number of requisitions filled, etc.).

And although assessment professionals have a variety of success measures, some in common with recruiters (e.g., tenure), there is one measure that stands above all others: job performance.

The "gold standard" of this measurement is to correlate test scores with job performance measures (called criterion-related validation evidence). A correlation of, say, .50 between these two, is considered outstanding. Square that and you have the percentage of behavior explained. So in other words, when we can explain 25% of job performance with assessments, we call that success (and with good reason, because it's a heck of a lot better than 0%).

Why not higher than 25%? What would it take to get r =1.0, in other words a perfect correlation between test scores and performance? Here is a somewhat tongue-in-cheek recipe for achieving this impossible dream:

1. An accurate identification of the top competencies/KSAs required for the job. Qualified subject matter experts reach consensus on a handful of far and away the most important qualities that impact job performance.

2. Perfectly constructed and administered, perfectly reliable and accurate measures of the the top KSAs.

3. Variability among applicants in terms of amount of the relevant KSAs possessed.

4. Test scores combined and weighted appropriately given the job analysis results.

5. Variability in scores for those hired.

6. A clear description of the work to be performed and competencies to be demonstrated so the individuals understand expectations.

7. Perfectly reliable, accurate measures of job performance that capture behaviors one would logically relate to the critical KSAs.

8. A supportive work environment (e.g., high quality supervision, adequate resources) so this doesn't interfere with work performance.

9. Variability in job performance among those hired using the assessments.

10. Elimination of outside factors that may contribute to lower job performance (e.g., family emergencies, medical/psychological changes).

As you can see, some of these are achievable (1, 4, 6), others are challenging and depend on circumstances, but are not impossible to achieve (3, 5, 8, 9) and some are practically impossible (2, 7, 10). I said earlier this was tongue-in-cheek because obviously we'll never have a situation where all of these conditions (as well as ones I'm sure I forgot) are true.

Does this mean we should abandon the correlation between test score(s) and job performance? Absolutely not. It should continue to be one of our "gold standards" for measuring our success as assessment professionals. But we--and our customers--should have our eyes wide open before pressing "compute."

Meet the new SIOP...same as the old SIOP

The votes are in, and the new name for the Society for Industrial and Organizational Psychology (SIOP) is...the same.

After over a thousand votes from members, the existing acronym beat The Society for Organizational Psychology (TSOP) by a tally of 51% to 49%--a difference of 15 votes. You can read my comments about this option--and my prediction of the outcome--here.

Why is this non-news, news? Because it's problematic that the main professional, scientific body that devotes itself to researching the psychology of organizations and work (POW!) repeatedly has identity issues. This is in large part because of the word "industrial", which makes it sound like we're all studying factory workers. I am not alone in having people look at me sideways when I attempt to explain our field.

To be perfectly honest, I am reluctant to describe my focus as "psychology", except to others in the same field. It sidetracks the conversation (perhaps due to my insufficient skill). It's much easier to connect with people by saying I'm in Human Resources. This isn't to say that the focus on psychology isn't important, or that others in I/O psychology might not mind using this phrase, or that there isn't some brand value in SIOP. But call me crazy, if you're reluctant to name your field (and attorneys don't count--people know, or think they know, what you do), the profession has a problem.

So, our identity struggle continues. An interesting follow-up study might be to ask SIOP members how they describe their field of work to non-I/O folk and break that down by area of focus. It's a big tent.

Personally, I prefer something that includes Work and Organizational. Mix and match letters as you will.

On a positive note, did you know you can access all of SIOP's quarterly news publication, TIP, here? The January 2010 issue has pieces on integrated performance management, a preview of Lewis v. City of Chicago, and a lot more.

Outback settlement contains interesting requirements

You may have heard that Outback Steakhouse, a restaurant chain based in Tampa, Florida, has agreed to settle a gender discrimination lawsuit for $19M. What's interesting about this isn't the size of the settlement, but rather the conditions attached.

Background: The EEOC sued Outback in 2006, claiming it systematically discriminated against its female employees by denying them promotion opportunities to the more lucrative profit-sharing management positions. In addition, they claimed that female employees were denied promotional job assignments such as kitchen management, which were required for employees to be considered for top management positions.

The settlement: Outback agreed to a four-year consent decree and $19M in monetary relief. So far, pretty standard. But there were additional settlement requirements, and here's where it gets interesting. In addition to the monetary relief, Outback has agreed to:

1. Create an online application system for employees interested in management positions. This is the first time I've seen this in a settlement (which isn't to say it hasn't happened) and seems to indicate that the EEOC views this as a more "objective" screening mechanism.

2. Create and hire someone for a newly created "human resources executive" position titled Vice President of People. Again, this is a new one for me.

3. Hire an outside consultant for at least two years who will monitor the online application system to ensure women are being provided equal opportunities for promotion and provide reports to the EEOC every 6 months.

The main thing that strikes me about this settlement is the faith that is being placed in an online application system to somehow ensure equal opportunity. Sure, having a standardized application system may cut down on some of the subjectivity of individual hiring supervisors, but it leaves me wondering:

- What will the screening criteria for management positions be?

- How will the outside consultant define "equal opportunities"?

- How will access to the online system be controlled, and who will be making screening/hiring decisions?

- What happens if there continues to be adverse impact, which you would expect if applicants continue to be screened on experience?

- What will be the duties of the Vice President of People, how will they be hired, and how will they interact with the consultant?

This will be interesting to watch.

Validity: An elusive (unitary?) concept

What makes a test "valid"? What is the best way to develop a selection system? These are two of the most fundamental questions we try to answer as personnel assessment professionals, yet the answers are strangely elusive.

First of all, let's get two myths out of the way: (1) a test is valid or invalid, and (2) there is a single approach to "validating" a test. It is the conclusions drawn from test results that are ultimately judged on their validity, not simply the instruments themselves. You may have the best test of color vision in the world--that doesn't mean it's useful for hiring computer programmers. And many sources of evidence can be used when making the validity determination; this is the so-called "unitary" view of validity described in references like the APA Standards and the SIOP Principles. Unitary in this case refers to validity being a single, multi-faceted concept, not that psychologists agree on the concept of validity--a point we'll come back to shortly.

Although we can debate test validation concepts ad infinitum, the bottom line is we create tests to do one primary thing: help us determine who will perform the best on the job. The validation concept that most closely matches this goal is criterion-related validity: statistical evidence that test scores predict job performance. So we should gather this evidence to show our tests work, right? Here's where things get complicated.

It's likely that many organizations can't, for various reasons, conduct criterion-related validity studies (although baseline evidence of this would be helpful). Most of the time, it's because they lack the statistical know-how or high quality criterion measures (a 3-point appraisal scale won't do it). So in a strange twist of fate, the evidence we are most interested in is the evidence we are least likely to obtain.

So what are organizations to do? Historically the answer is to study the requirements of the job and select/create exams that target the KSAs/competencies required; this matching of test and job is often referred to as "content validity" evidence. But Kevin Murphy, in a recent article in SIOP's journal Industrial and Organizational Psychology, makes an important point: this is good practice, but not a guarantee that our tests will be predictive of job performance. Why not? For a number of reasons, including poor item writing and applicant frame of reference. Murphy makes a passionate argument that we rely way too heavily on unproven content validation approaches when we should focus more on criterion-related validation evidence. Instead of focusing on job-test match, we should focus on selecting proven, high quality exams.

Not surprisingly, the article is accompanied by 12 separate commentaries that argue with various points he makes. It's also interesting to compare this piece with Charley Sproule's recent IPAC monograph where he makes an impassioned defense of content validity.

A complete discussion of the pros and cons of different forms of validation evidence are obviously beyond a simple blog post. My main issues with Murphy's emphasis on criterion-related validation are threefold. First, as stated above, most organizations likely don't have the expertise to gather criterion-related validation evidence for every selection decision (maybe this is his way of creating a need for more I/O psychologists?). Perhaps "insufficient resources" is a poor excuse, particularly for an issue as important as employment, but it is a reality we face.

Second, even if we were to shift our focus to individual test performance, following a content validation approach for development enhances job relatedness (which Murphy acknowledges). Should your selection system face an adverse impact challenge, the ability to show job relatedness will be essential.

Finally, let's not forget that high test-job match gives candidates a realistic job preview--hardly an unimportant consideration. RJPs help candidates decide whether the job would be a good match for their skills and interests. And no employer that I know of enjoys answering this question from candidates: "What does this have to do with the job?"

The approach advocated by Murphy, taken to its extreme, would result in employers focusing exclusively on the performance of particular exams rather than on their content in relation to the job. This seems unwise from a legal as well as face validity perspective.

In the end, as a practitioner, my concern is more with answering the second question I posed at the beginning of this post: What is the best way to develop a selection system? Given everything we know--technically, legally, psychologically--I return to the same advice I've been giving for years: know the job, select or create good tests that relate to KSAs/competencies required on the job, and base your selection decision on the accumulation of test score evidence.

Should researchers work harder to show that job-test content "works" in terms of predicting job performance? Sure. Should employers take criterion-related validation evidence into consideration and work to collect it whenever possible? Absolutely. Will job-test match guarantee a perfect match between test score and job performance? No. But I would argue this approach will work for the vast majority of organizations.

By the way, if you are interested in learning more about the different ways to conceptualize validity--"content validity" in particular--Murphy's focal article as well as the accompanying commentaries are highly recommended. He acknowledges that he is purposely being provocative, and it certainly worked. It's also obvious that our profession has a ways to go before we all agree on what content validity means.

Last point: the first focal article in this issue--about identifying potential--looks to be good as well. Hopefully I'll get around to posting about it, but if not, check it out.

R/A Predictions for 2010

With 2010 right around the corner, here are some predictions for what the new year will bring in the area of recruitment and assessment:

1) More personality testing. Year after year personality testing continues to be one of the hottest topics. Look for more research, more online personality testing, and new measurement methods.

2) More boring job ads. Even though we know better, don't expect to see any big leaps in readability for 80% of job ads. Same old job descriptions. Maybe we'll see some pictures. On the plus side, more organizations focus on making their career portals attractive.

3) A slow trickle of research on recruiting. The amount of large-scale, sophisticated research on recruiting methods remains a shadow of that found in the assessment literature. Don't expect this to change.

4) More focus on simulations. 2010 sees more focus on simulations, particularly those delivered on-line, as highly predictive assessments as well as realistic job previews. Oh, and they likely have low adverse impact (research, anyone?).

5) Leadership assessment gets even hotter. With the economy improving and more boomers deciding the time is right to retire, finding and placing the right people in leadership positions becomes an even more important strategic objective.

6) Federal oversight agencies get more aggressive. With more funding and backing from the Obama administration, expect to see the EEOC and OFCCP go after employers with renewed vigor. By the way, have you seen the EEOC's new webpage? It's actually quite well done.

7) More fire departments get sued. In the wake of the Ricci decision, fire dept. candidates feel emboldened when they fail a test or fail to get hired/promoted. Look for departments to try to get out ahead of this one by revamping their selection systems.

8) More age discrimination lawsuits. With so many boomers, expect to see more claims of discrimination, particularly over terminations. Keep words like "energetic" and "fresh" out of your job ads.

9) Automation providers slowly focus on simplicity. Whether we're talking applicant tracking or talent management systems, vendors slowly realize that they need to make their applications simpler to increase usability and buy-in. No, simpler than that. Keep going...

10) Employers get more sophisticated about social networking sites. Many realize that rather than jumping on the latest Twitter-wagon, it's best to figure out where these sites fit with their recruitment/assessment strategy. Watch for more positions whose sole role is managing social media.

11) Online candidate-employer matching continues to be a jumbled mess. Without a clear winner in terms of a provider, job seekers are forced to maintain 400 profiles on different sites and may give up altogether and focus more on social networking. Meanwhile, employers continue to try to figure out how to reach passives; LinkedIn continues to look good here but needs to expand its reach a la Facebook.

12) More employers face the disappointing results of online training and experience questionnaires. Will they go back to the drawing board and try to improve them (hint: don't use the same scale throughout), or abandon them for more valid methods, such as biodata, SJT, and simulations? More research on T&Es is badly needed, even if we are just putting lipstick on a pig.

13) Decentralized HR shops centralize. Centralized ones decentralize. Particularly in the public sector, these decisions unfortunately continue to be made based on budgets rather than best practice. Hiring supervisors wonder why HR still can't get it right.

14) Fortunately, HR continues to professionalize. With much of the historical knowledge walking out the door and the job market improving, HR leaders are forced to re-conceptualize how they recruit and train recruitment and assessment professionals. This is a good thing, as it means more focus on analytical and consultative skills.

Keep up the good work everybody. And Happy Holidays!

Setting cutoff scores on personality tests

What's the best way to set a cutoff score for a personality test, knowing that some candidates inflate their score? It all depends on your goal. Are you trying to maximize validity or minimize the impact of inflation?

According to a research study by Berry & Sackett published in the Winter '09 issue of Personnel Psychology, if your goal is to maximize validity, your best bet is to wait until applicants have taken the exam, then set your cut-score (e.g., the top two-thirds); this was particularly true when selection ratios are small (i.e., organization is very selective).

If your goal is to minimize the number of deserving applicants who are displaced by "fakers", you're better off establishing the cut point ahead of time, by using a non-applicant derived sample (e.g., job incumbents, research group). The results were generated using a Monte Carlo simulation.

Interestingly, the authors also replicated the work of other researchers who have shown that the impact of faking on the criterion-related validity of personality measures is relatively low. There are a few other very good points made in this article:

- Expert judgment methods of establishing pass points (e.g., Angoff method) may be difficult to use for personality tests since experts may find it difficult to judge individual items. Methods used to select a certain number of applicants or methods based on a criterion-related validity study (both used as variables in this study) are more appropriate for personality tests.

- There is no consensus of how prevalent faking on personality exams is; estimates range from 5-71%. It likely depends on the situation and how motivated test takers are to engage in impression management.

- Some recommend setting a very low cutoff score for personality tests, which would exclude only those likely not suitable for the position (and not faking), while others prefer a more stringent cutoff to maximize utility.

- A reasonable range of d-values for score inflation on personality inventories is .5-1.0 (used in this study).

- There exists very little research on the skewness of faking score increases. A positively-skewed distribution (meaning most people faked a small amount) was used in this study. (I would think this would also vary on the situation)

So bottom line: where--and how--you set your cutoff score on personality inventories depends on whether you want to maximize the predictive validity or minimize the number of deserving applicants that get left out of the process.

Other good reads in this issue:

- Police officer applicants reactions to promotional assessment methods

- The impact of diversity climate on retail store sales

- The construct validity of multisource performance ratings

- Labor market influences on CEO compensation

Want better prediction? Gather more data.

That's the bottom line from a study in the November 2009 issue of the Journal of Applied Psychology.

Oh & Berry looked at how adding personality ratings from peers and supervisors added incremental validity to self-ratings using a five-factor model measure. What were the results? Increases of 50-74% in operational validity across personality facets. They also looked at differential prediction of task and contextual performance (unfortunately those results weren't reported in the abstract). Bottom line? If you're using a personality assessment for promotions, strongly consider gathering data from co-workers.

Speaking of self-presentation, in the same issue Barrick et al. report the results of a meta-analysis of how self-presentation tactics (e.g., appearance, non-verbal behavior) impact interview ratings and later job performance. Results? "What you see in the interview may not be what you get on the job and...the unstructured interview is particularly impacted by these self-presentation tactics." An important reminder of how who the candidate seems to be impacts your assessment, and another reason to collect multiple sources of data.

There are a number of other great articles in this issue, such as:

How Major League Baseball CEO personalities impact important outcomes (like, um, winning).

How SJT and biodata measures add to the prediction of college student performance.

How personality scale validities change over time among a group of medical students.

Differences among letters of recommendation in academia between genders.

HR: C-level or AAA?

Just when we thought we were C-level, Dogbert reminds us how old-school CEOs view HR.

p.s. I think I like CPO better than HCO.

Explain away

Of all the low-hanging fruit in recruitment and selection, perhaps none are easier to implement than explaining your process. It's shocking how few selection processes are fully (and coherently) explained, not just in terms of getting from point A to point B, but why the points exist in the first place.

Turns out explaining the process to candidates matters. A lot. And while we already knew that applicant perceptions were important, a recent meta-analysis by Truxillo, et al. published in the International Journal of Selection and Testing clarifies the impact that explanations have. Specifically, explanations were related to:

- Fairness perceptions (important in their own right)

- Perceptions of the hiring organization

- Test-taking motivation

- Performance on cognitive ability tests

Furthermore, fairness effects were greater when paired with personality tests rather than cognitive ability exams.

What does all this mean? It's pretty simple, really--communicate, communicate, communicate. Explain in clear terms to all applicants what the full selection process is, and why. Imagine something like this:

"Thank you for your interest in applying for the Blog Reader position. The selection process will consist of the following:

Step 1. You submit your application and work sample by December 5.

Step 2. Your application is reviewed to ensure you meet our minimum qualifications found on the job posting.

Step 3. If so, your work sample is scored by internal subject matter experts. It will be judged on relevance to the position, complexity, and contribution to the profession. The top scorers move on to Step 4.

Step 4. You are contacted by Human Resources to set up an interview. The interview will last approximately 2 hours and will take place at our San Jose campus. They will take place in mid-January.

Step 5. You interview with the hiring supervisor and 2-3 potential co-workers. You will be asked a series of questions designed to measure your knowledge of Blogs. You will also be asked to complete a writing sample which will be judged for style as well as content. Top performers in these steps will be asked for a final interview with the head of the Blogger Division.

Should you fail to proceed to any Step, you will be contacted and told why."

That wasn't so hard, was it?

For more about applicant perspectives in selection, check out the entire December '09 issue.

And while I'm on the subject of research, check out the latest issue of the International Journal of Testing for good stuff on test compromise, DIF, and P-O fit. Oh, and don't forget about the entire issue devoted to test adaptation.

Last but not least, Practical Assessment, Research, and Evaluation (PARE) has put out several things lately worth reading, check 'em out.

What employers can learn from Twilight

Not since Harry Potter have I seen such obsessed fans. The buzz started several months ago in anticipation. And November 20th is almost here.

Don't know what happens on November 20th? Ask your daughter. Or granddaughter. Or, heck, pretty much any woman between the age of 16 and 30. That's the day that New Moon, the second installment in the wildly popular Twilight series, hits theaters. Why should employers care about this, other than anticipating that certain staff members will be out of the office that day? Read on.

Like Harry Potter, the Twilight books (written by Stephenie Meyer) are enormously popular, and the movies are too. Also like Harry Potter, fans range the demographic spectrum, although the most rapid fans seem to be women (not surprising given the protagonist and the love triangle she's in the middle of).

Most employers would kill to have the kind of brand devotion that Twilight fans have. If Twilight was an employer, they would be competing with Google for top talent. So what can we learn from this phenomenon that can help us with branding our organization?

1. People like good stories. Twilight was a phenomenon way before Robert Pattinson and Taylor Lautner (the male lead actors). Whether you're in banking, IT, public utilities, or a flower shop, you have stories to tell about how your organization has impacted others--or how your employees have impacted each other. Are you telling these stories, or letting stories be told about you?

2. People still read. Related to #1, the Twilight phenomenon, like Harry Potter, began with the books. There's a lot of hype about video these days, but given something interesting, people have no problem spending time reading it. What does your recruitment material look like--is it entertaining? Educational? Would you read it even if you weren't interested in a job there?

3. People like fantasy. There's an awful lot of reality out there right now--the recession, H1N1, wars--and people like to take a mental break. Don't be afraid to break out of the mold and try telling a story that takes people away from their day-to-day lives.

4. People like contests and "sides". One of the biggest, possibly the biggest, dramas within the world of Twilight is the competition between the two main lead male characters. Fans identify themselves as being on "Team Edward" or "Team Jacob". This isn't something you see employers do very often, and it requires a bit of built-in loyalty, but it's something that can engage fans even more.

5. People like being fans. There's something primal about being part of a group of people who share the same interest. If you give people something to be a fan of, they'll enjoy connecting with others who share their passion. This is what the Facebook fan pages are all about. Google has 300k+ Facebook fans. Twilight has over 4 million.

6. You can brand almost anything. Branding is about more than your website, or your recruitment fliers. It can become part of everything your organization does, if it's strong enough. And it's more than just a logo, it's about the organization's philosophy and accomplishments. Look around and you can probably see Twilight branded on almost everything. I'm surprised they don't have Twilight adhesive bandages. Oh wait, they do.

As November 20th approaches, be prepared for a media onslaught about Twilight. Whether you're a fan or not, use it as an opportunity to think about how your organization could garner that kind of excitement. After all, that's what leads high potentials to want to apply.

New Job Simulations Report

The U.S. Merit Systems Protections Board (MSPB) just released a great, easily digestible, report on job simulations.

The report includes several things, such as:

- Job simulations defined and advantages/disadvantages

- Types of job simulations (SJT, work samples, etc.) and concrete examples

- Benchmark data on satisfaction with candidate quality as well as how federal agencies currently use simulations (just don't look at GPA compared to job knowledge tests in Figure 2)

- Survey data on why simulations aren't used more often in the federal government (time and expertise, sadly, were the top reasons)

- A 5-step strategy for using job simulations

- References to support the use of simulations (and good selection in general)

A great, free resource for anyone wanting to learn more about one of the best selection mechanisms you can use. And particularly relevant as more and more organizations move to using training and experience (T&E) questionnaires as their first (quick but not particularly valid) hurdle.

Personality tests: Situation matters

Is a personality test the right selection mechanism for your needs? In trying to answer that question, an important consideration is: Does the job allow for the expression of personality facets?

There's a concept in psychology called situation strength. It refers to how the "strength" of a situation impacts the display of personality via behavior, and it's something important to remember when using measures of personality to predict job performance.

A situation's "strength" refers to the environment under which personality aspects are displayed; think of it like a rulebook. If Job A is described perfectly in exacting detail with very little room for deviation ("Place container A over part B..."), does one's personality really matter when it comes to successfully performing the job? Or thought of another way, if the rulebook repeatedly emphasizes using an aspect of personality (e.g., extraversion), how will people without the ability to express that behavior consistently fare?

If you're a software programmer, it probably matters little in the grand scheme of things how extraverted you are; analytical ability is likely much more important. Similarly, if you work in a call center and follow a script, differences in openness to experience probably don't mean much; extraversion is likely more important.

But what about something like conscientiousness--could the strength of a situation impact the relationship between this aspect of personality and job behavior? That's the question that Meyer and colleagues set out to answer, and an answer is published in a recent issue of the Journal of Organizational Behavior.

The way they went about trying to solve this puzzle was to conduct a meta-analysis at the occupation (rather than job) level. What did they find? A few things:

1) Uncorrected correlations between conscientiousness and performance varied widely between .06 and .23.

2) Correlations appear slightly stronger when using overall performance as the criterion rather than task performance (not surprising given previous research).

3) Stronger correlations were found in "weak" occupations.

What does this mean? Well, for one it reinforces the fact that the answer to "Do personality tests work?" varies greatly depending on what the job is and how you measure performance. But perhaps more interestingly, it suggests that at an occupation level we can expect that for jobs that come with built in rules regarding behavior ("strong" occupations), measures of personality aspects such as conscientiousness may not predict performance as well as for jobs with more flexibility ("weak" occupations).

So the next time you're thinking about using a personality inventory for selection purposes, consider: To what extent will incumbents be allowed to express their personality?

Is recruiting using SNS discriminatory?

I keep reading/hearing about how recruiting using social networking sites (SNS) opens employers up to discrimination lawsuits because of who uses the sites. For the most part, this just plain isn't true.

A recent Pew study is the latest to show that when it comes to using SNS like Facebook, MySpace, and LinkedIn, you really should have one primary demographic concern when it comes to ensuring a diverse candidate pool: age.

Not gender, at least not in traditional sense. While four years ago SNS users tilted slightly male (55%), the balance has essentially flipped today (54% female).

Not race, there simply do not appear to be generalizable differences in racial groups when it comes to these sites (in fact I've seen some data that suggest the user base on these sites is more diverse)--but things change, and this may vary with particular sites, so keep an eye on this one.

But when it comes to age, SNS users are disproportionately younger than the overall Internet population. In the words of the Pew report, "[this] doesn't mean that more older adults aren't flocking to SNS--they are--but younger adults are ALSO flocking to the sites, so the overall representation of the age cohorts in the SNS user population has actually gotten younger."

One demographic difference I don't see a whole lot about: disability status. Are individuals with disabilities more/less likely to use SNS? I think that's an important question we need to address if we're truly trying to diversity our candidate pools.

Myths about assessment

Despite plenty of evidence and commentary otherwise, several myths persist about personnel assessment:

1) Some tests are "objective", others are "subjective." This is a myth reinforced by no less than the U.S. Supreme Court on a regular basis. The reality is even the choice to use certain selection methods is a judgment. Sure, certain methods involve more ongoing judgment, but a multiple-choice test can be highly "subjective" and an interview highly "objective" depending on how they are made and used.

2) Only certain selection methods are legally considered "tests" and therefore vulnerable to legal scrutiny. Wrong. Anything you do to narrow down your candidate pool is technically fair game. This includes how you advertise, screen, and interview.

3) Good hiring is an art more than a science. Actually we have decades of research showing the opposite. Human judgment is full of flaws. Combine this with the fact that most people think they are experts, and you have a perfect storm of personal overconfidence. The time and effort spent creating standardized instruments targeting competencies relevant for a particular position will be well spent.

4) Even the best assessments can predict only a small fraction of job performance. It's true it's a fraction, but it's not small. Research indicates that close to 40% of the variation in performance can be predicted with assessments. That's nothing to sneeze at when you consider all the other things that impact performance (organizational climate, quality of supervision, reward structures, team composition, role clarity, resources, mood, etc.).

5) Good assessment will solve all your people problems. Yes, this is sort of the flip side of the above. Consultants like to pretend that with the right assessment instrument every person you hire will be the most productive, friendly, team-oriented person ever. The reality is that performance depends not only on what someone brings to the job, but leadership, organizational norms...all that stuff I mentioned in #4.

Do we have all the answers when it comes to hiring the right person? Nope. Is there enough best practice out there so that any hiring supervisor should be able to get the expertise they need to do significantly better than chance? Yep.

Latest EEO Insight

EEO Insight is quickly becoming a great resource for anyone interested in issues related broadly to equal employment opportunity. And this isn't just affirmative action plans--it includes anyone interested in recruitment and assessment.

In the latest issue (v1, #3), you'll read about:

- Alternatives to RIFs such as wage freezes and job sharing and the EEO implications

- Analyzing layoff decisions for statistical evidence of adverse impact

- Using multiple regression to detect race and gender differences in compensation

- Ricci in retrospect and lessons learned

- Reaching out to veterans and individuals with disabilities

- Results of the EEO best practices survey and (very good) recommendations

By the way, if you're interested in EEO issues and you're not already reading OFCCP Blog Spot, I highly recommend starting.

SIOP Name Change: Will They or Won't They?

The Society for Industrial and Organizational Psychology (SIOP) is considering a name change. For those that don't know, SIOP is a division of the American Psychological Association and has thousands of members devoted to research into a variety of phenomena, including recruitment and assessment.

SIOP was established in 1982 and a significant number of members have been asking on and off for years whether the name continues to accurately describe what they do. The problem is mainly with the "Industrial" part--it's just not a word that gets as much attention as it used to (remember when everything was "industrial strength"?).

The name change is something they've tossed around for years, but beginning in October they're going to survey their members and ask them to consider three alternatives:

1. The Society for Organizational Psychology (TSOP) - name's okay but continues implication that focus is on cleaning our offices; acronym sounds like a rapper; oh, and URL is taken.

2. Society for Work Psychology (SWP) - a little bland but definitely simpler; how do you say the acronym? Is it like "swap"? Or maybe "swip"? Oh, and restricts the field to "work", which is a little narrow.

3. Society for Work and Organizational Psychology (SWOP) - the most complete name in terms of description; acronym easy to use but has some interesting brethren. Too bad we couldn't come up with SWOT (ya know, as in strengths, weaknesses, opportunities, threats). Oh, and URL is taken. Sort of.

The winner between these will take on SIOP for the final determination.

My bet? Members will split among the three, none will receive a passionate endorsement and it will lose against SIOP.

Any takers?

Screening on personality: Legal loophole or pothole?

In a recent post over at ERE, the author mentions a website that provides employers with the ability to screen candidates based on a measure of personality that applicants complete online.

My reading of this article prompted the following internal debate:

Me #1: Boy oh boy, is that ever a bad idea. The Uniform Guidelines clearly state (Q&A #75) that a measure of a trait or construct cannot be validated based on content validity, which is what most employers are likely to rely on in this situation.

Me #2: Ah yes, but because personality tests typically result in much less adverse impact than traditional cognitive tests, are the Uniform Guidelines even likely to come into play?

Me #1: Maybe not, but you never know until you go through a selection process, so why take the risk?

Me #2: What about in cases where the numbers being screened are so small that adverse impact analysis is likely to be wonky? (that's a technical term)

Me #1: Well that may well be different, but you're missing the point.

Me #2: What is the point?

Me #1: That employers should use caution before screening based on constructs such as personality. They need to take validation seriously.

Me #2: But isn't it good that they're using an instrument that's at least based on an evidence-based theory of personality (the Big 5)?

Me #1: Absolutely, and props to them. But it is still incumbent on employers to realize the legal risks as well as the implications of using a self-report personality screen as a first hurdle.

Me #2: Fine, but aren't you being a little hypocritical? Haven't you said one of your goals is a giant database where applicant information can be matched with employer needs?

Me #1: True, but I was thinking more along the lines of verifiable skills testing, not self-report inventories.

Me #2: Actually in that post you specifically refer to Big 5 assessments.

Me #1: Hey! This isn't about me. This is about warning employers to make sure they know what they're doing when they screen based on personality measures.

Me #2: Aren't you making an awful lot of assumptions about the website's process without having actually used it or talked to the owners?

Me #1: Well, yes, but I'm a blogger. That's what we do.

Me #2: Ugh.

2009 IPMA/IPAC Conference Material

Just got back from Nashville after having attended the 2009 IPMA-HR/IPAC conference. Great information, great people, and really enjoyed Nashville.

For me, the conference was largely about one thing: leadership. Makes sense in times like these, people are focusing on how leaders can help guide organizations through rough waters. People like Bob Hogan made great arguments for the importance of leadership and how far we have to go in doing a good job selecting them.

If you weren't able to attend, you can see many of the presentations here. Here's a sampling of topics:

- The selection interview

- Assessment centers for supervisors and managers

- Employee engagement

- Online testing

Can't wait for next year's IPAC conference in Newport Beach!