Test validity: A fairy tale

Once upon a time there was a field called Industrial and Organizational Psychology.  Its researchers and practitioners dealt with a myriad of magical issues ranging from individual differences and behavior to organizational structures.

Within this field, there was a specialty called Personnel Psychology.  It dealt with narrower--but no less mysterious--issues such as defining and designing jobs and, most relevant for us, finding and hiring the right people.  The I/O psychologists and HR professionals that quested for these answers often found themselves on dangerous missions like battling Monsters of Doubt (i.e., first-line supervisors).

These adventurers had two main weapons at their disposal when fighting these monsters: the Carrot of Truth and the Stick of Pain.

When invoked, the Carrot of Truth, fashioned deep in the Mines of Correlation, caused monsters to realize that hiring the right people was the right thing to do for their realm.  It increased productivity and morale, customer satisfaction, and organizational flexibility.  It also allowed supervisors to spend more time leading and less time dealing with gremlins (i.e., employees with performance problems).

The Stick of Pain used an opposing form of magic but was sometimes equally effective.  It attempted to slay the monsters using a peculiar power called The Law.  The Law frightened monsters because it meant they could experience emotional pain and suffering, and--more importantly--fewer bags of gold.

For a long time, Personnel adventurers used both of these weapons to slay all kinds of monsters, on high mountains and in dungeons.  But over time, the adventurers discovered something: the Stick of Pain was becoming less and less effective.

It wasn't that the Stick was powerless.  It's just that its magic didn't seem to frighten the monsters as much.  The monsters saw their gold piling up and didn't feel the sting of suffering as they once had.  And they started developing an addiction to carrots all on their own.

So there came a day where the adventurers and the monsters met on the battlefield and came to an agreement.  No longer would the adventurer wield the Stick of Pain.  And in return, the monsters pledged to respect the Carrot of Truth.  They forged an eternal partnership and lived happily ever after.

The End.


Okay, so I've taken a little artistic license with my blog post today.  But hopefully you see where I'm going.

Back in the old days (ya know, like the 80s), employers were faced with a foreboding world of testing, with the Civil Rights Act and cases like Griggs vs. Duke Power looming large over their assessment programs.  I/O psychologists were brought in to help organizations navigate the complicated world of employment testing, which required an appreciation of statistics and the law alike.  Large awards and settlements brought C-level attention, and regulatory agencies like the EEOC and DOJ served in ongoing oversight roles, requiring employers to clean up their act with procedural requirements that could be burdensome.

Nowadays, I/O psychologists are as likely to be valued for their ability to crunch "big data" to detect employee behavior trends as their ability to conduct thorough job analyses (not that the two are mutually exclusive).  Lawsuits regarding testing are infrequent compared to issues like wages and hours, harassment, and terminations.  The selection cases that do come up are as likely to involve disabilities as adverse impact due to cognitive loading.

Sure, we have the occasional big case that gets attention.  But the bottom line is over the years the "stick" has become much less effective as an argument for sound assessment than the "carrot."

Smart employers like Google have started crunching the numbers and realized the true business value of defining the right competencies for jobs.  They're doing so not because they're afraid of litigation, but because they see more clearly the direct line between best practices in selection that we've been preaching for years--i.e., focusing on valid assessment results--and the bottom line.


So where does that leave the stick (i.e., fear of lawsuits)?  Is it time to put it away along with phrenology and T&Es (woops, that slipped in)?

I don't think so.  Organizations will always be subject to legal scrutiny when their selection processes have adverse impact and the right person talks to the right attorney.  Personnel psychologists and HR professionals should always have a healthy respect for the legal climate we operate in, and not forget that "job related and consistent with business necessity" isn't fictional gibberish.

But what it does mean is that because organizations are paying attention to their assessments, they are more likely to yield valid results and be more free of illegal bias.  That means management's quest and the selection professional's quest are more likely to converge, with a lot more cooperation.

And hopefully a lot fewer monsters.

Facebook fallout continues

The fallout from earlier reports of employers asking applicants for their Facebook passwords continues. Obviously a nerve was struck.

Today, U.S. Senators Richard Blumenthal and Charles E. Schumer formally asked the Department of Justice (DOJ) and Equal Employment Opportunity Commission (EEOC) to launch an investigation into whether this practice violates federal laws.

From the press release:

"Blumenthal and Schumer argued that this disturbing practice represents a grave intrusion into personal privacy that could set a dangerous precedent for personal privacy and online privacy, make it more difficult for Americans to get jobs, and expose employers to discrimination claims"

"'With few exceptions, employers do not have the need or the right to demand access to applicants’ private, password-protected information.'”

"'In an age where more and more of our personal information – and our private social interactions – are online, it is vital that all individuals be allowed to determine for themselves what personal information they want to make public and protect personal information from their would-be employers. This is especially important during the job-seeking process, when all the power is on one side of the fence.'"

"In their letter to the Justice Department, Blumenthal and Schumer pointed out that two courts have found that when supervisors request employee login credentials, and access otherwise private information with those credentials, that those supervisors may be subject to civil liability. Although those two cases involved current employees, the courts’ reasoning does not clearly distinguish between employees and applicants."

"Blumenthal and Schumer also announced that they are currently drafting legislation that would seek to fill any gaps in federal law that allow employers to require personal login information from prospective employees to be considered for a job."


In related news, on today's Talk of the Nation show, they discussed this issue with a reporter from Wired magazine and an HR consultant.


There are a lot of issues here, ranging from online privacy to public reputation to discrimination, but one that I think deserves more attention is how employers can legitimately get the type of information they're seeking. Again, we're not talking about a background check for, say, a peace officer position, we're talking about your run-of-the-mill clerical job. Basically employers are hungry for any information like displays of poor judgment, a negative attitude about their employer, duplicity in their application, etc.

How might an employer get this type of information without resorting to asking for applicant passwords? It's pretty simple actually, we go back to the basics such as:

1) Reference checks; highly under-used and maligned, with many organizations unaware of technological advances made in this area that make it more likely they'll get the information they need.

2) Work sample/performance tests that simulate actual job tasks. These can be very effective in determining how an applicant will respond in an actual situation (i.e., where things like judgment are important).

3) Situational judgment tests: a lower fidelity version of a performance test that nonetheless can be very effective at assessing candidate's knowledge of, and propensity to engage in, appropriate behavior in various situations.

4) Personality inventories: made to measure things like conscientiousness, openness to experience, and extraversion, which may all be good or bad things depending on the needs of the position.

Bottom line: there are other--better--forms of assessment out there that have been around for a long time and when done well, do the trick. No need to ask for someone's online diary.


As a reminder, for those of you that are IPAC members, I gave a webinar about this topic about a year and a half ago where I gave an overview of the technology as well as a summary of many of the major challenges inherent in this practice. The recording is available in the Members Only area.

Final research update of 2011

Welcome to the last HR Tests research update of 2011! This year has been filled with research developments, building on previous thinking as well as venturing out into new areas. Let's see what the end of the year brings us (hint: a lot)...

We'll start with the Winter 2011 issue of Personnel Psychology:

- If you're looking for executives who excel at strategic thinking, you'll want to pay attention to not only their cognitive ability but their accumulated work history, according to Dragoni, et al.

- I tend to think of realistic job previews (RJPs) as occurring pre-hire, but research by Earnest, et al. suggests that an effective technique is to conduct an oral and written RJP post-hire--and be as honest as possible.

- Meta-analyses are relied upon heavily as summaries of large bodies of research. Roth, et al. point out ways we can make them better, particularly with respect to estimates of validity and group differences.

- One of the downsides of cognitive ability tests is they're not always perceived very well by applicants. Sumanth & Cable show how this perception is influenced by the status of the organization as well as individual status.

- Next, a fascinating description of a scale to measure the idea of calling by Dobrow and Tosti-Kharas. What I found most interesting was how the measure was associated with different criteria for different domains (e.g., art, music, management). You can read a draft version here.


Let's turn now to the January issue of JOB:

- Political skill is a hot topic, and Gentry, et al. demonstrate how the perception of promotability related to this skill varies between bosses, peers, and direct reports. Long story short: it differs, and also depends on behavior (attitude will only take you so far, right?). You can read the full version (at least right now!) here.

- Anyone that's worked with (or has in their family) engineers or scientists knows that they often share some strong traits. This suggests leading these groups that engage in creative activity may require specific attributes, which is what Robledo, et al. set out to describe.


On to the January issue of JOM:

- Core self-evaluation (CSE) is another hot topic, and Chang, et al. provide an important review of 15 years of research, including the meta-analytic support for CSE predicting in-role and extra-role performance.

- Boswell, et al. provide an integrative review of the concept of job search across different situations (e.g., following job loss, while employed).

- One area that deserves more attention is selection into the highest positions within organizations. Withers, et al. provide a review of the process of selecting a director for a board.


Let's not forget v41(2) of Personnel Review:

- Hoglund delves into the topic of talent management as a strategic HRM practice. A fascinating topic that reinforces the importance of HRM as an influence over employee perceptions and behavior.

- The wording of job ads can have important impacts on applicant perceptions and behavior. De Cooman and Pepermans analyze the differences between for-profit and non-profit job ads, and show how only a fraction of the information potentially relevant for job-person match is published.

- Another topic that deserves additional attention is the motivation to expatriate. Altman and Baruch describe results of a qualitative study that may be useful to organizations thinking about attracting and selecting for positions that require this substantial move.


Don't forget this one:

- Johnson, et al. with an important reminder that when looking at the issue of discrimination, using single categories to define groups is probably not the best strategy.


Whew! And last but certainly not least, in the December issue of IOP, Michael McDaniel and colleagues present an argument (similar to ones made elsewhere) that the Uniform Guidelines are outdated and, worse than that, a detriment to the field of selection. The commentaries are many and range from support to passionate disagreement, with a healthy dose of caution (and dare I say...intransigence?) thrown in. Worth a read, particularly for those following the Guidelines literally and those engaged in related litigation. You can read a draft version here.


I hope everyone has a great New Years; here's to a wonderful 2012!

One last time: It's all subjective

While reading news about a court decision recently I was struck again by how the U.S. Court system continues to make a false and largely unhelpful distinction between "objective" and "subjective" assessment processes, and they're certainly not the only ones. Presumably this is meant to highlight how some are based on judgment while others are free from it.

One last time: it's all subjective.

I challenge you to come up with a single aspect of the personnel assessment process that is not based on judgment. Not only that, "degree of judgment" is not a useful metric in determining the validity of a selection process--for many reasons including but not limited to whose judgment is being used.

Here is a sample of assessment components that are based on judgment:

- how to conceptualize the job(s) to be filled

- how to study the job(s)

- how to recruit for the job(s)

- which subject matter experts to use

- how to select the KSAs to measure

- how to measure those KSAs

- how items are created or products are selected

- how to set a pass point, if used

- how to administer the assessments

- how to score the assessments

- how to combine assessments

- how to make a final selection decision

- what type of feedback to give the candidates

And this is just the tip of the iceberg. The entire process is made up, like all complex decisions, of smaller decisions that can impact the selection process (what software do I use to analyze the data? Is this one KSA or two?).

So what WOULD be helpful in describing how an assessment process is developed and administered? I can think of a few yardsticks:

1. Extent to which processes and decisions are based on evidence. To what extent is the assessment and selection process based on methods that have been shown scientifically to have significant utility?

2. Degree of structure. To what extent are the assessment methods pre-determined? How flexible are the processes during the assessment?

3. Multi-dimensionality. How many KSAs are being measured? Is it sufficient to predict performance?

4. Measurement method. How many assessment methods are being used? Do they make sense given the targeted KSAs?

5. Transparency. Is the entire process understandable and documented?

I'm not inventing the wheel here. I'm not even reinventing it. I'm pointing out how most cars have four of them. It's obvious stuff. But it's amazing to me that some continue to perpetuate artificial distinctions that fail to point out the truly important differences between sound selection and junk.

Recipe for losing a lawsuit

Ingredients

One large, diverse candidate pool
One cognitive ability OR physical agility test
One protective services job (police or fire if in season, otherwise corrections)

Optional: large, aggressive employee union
Optional: history of litigation


Instructions

1. Begin by deciding what type of exam--cognitive ability or physical agility--you feel like giving; don't worry about performing a job analysis first as these are time consuming and boring. If you must, select a small sample of current employees (preferably the poor performers) and provide minimal instruction. Don't worry about whether they are "true" job experts, and whatever you do, don't link tasks to KSAs--everyone hates doing it.

2. Select what you will be measuring. Base your decision on what you feel like, or whatever's easiest. Usually this is just doing what you did last time.

3. Have untrained analysts prepare the exams. Because anyone can do hiring, select whoever has time on their hands. Optional: search the Internet for a test that catches your eye. Rule of thumb is one question per content area (if you have more than one question, you're wasting applicant time).

4. Make sure the reading level of the exam is graduate school-level. After all, isn't reading an important part of any job? And don't you want the best?

5. Next, choose weighting of your exam components either randomly or based on gut feeling. When in doubt, place the largest weight on the test that is most related to cognitive ability.

6. Select a pass point. It should either be: (a) 70 percent; (b) based on administrative convenience; or (c) chosen at random.

7. Administer the exam, preferably with limited advertising. If you must advertise, give applicants a very short amount of time to prepare--after all, this isn't grade school. Do not pre-test the exam--if you've followed these instructions, it should be fine.

8. Score the exam--if you can, avoid "right/wrong" questions and go with ones where you can personally judge the quality of the answer. Don't worry about a boring "benchmark"--you know a good response when you see it.

9. Keep all scoring results and details regarding the process to yourself. Candidates don't need to know (and won't understand).

10. Make final selection decisions. Do not administer yet another test before making your selection, or if you must because of boring rules, make it an unstructured interview. Ask lots of questions like, "If you were a book, what would your title be?" If you have any women or minorities, ALWAYS ask questions about their ability to perform the job.

11. Do not document any of this process. Everyone involved will be with the organization for a long time, and people have really good memories.

Above all: have fun! After all, it's just people's livelihood.


A good example of how this type of thing plays out (albeit in a less horrific manner) is the recent decision in Easterling v. Connecticut Dept. of Corrections.

And for those of you that want to read more about how the law applies to selection, look for an upcoming IPAC monograph written by yours truly!

Jan TIP gems: HRO, UIT, SIOP, and VII

Yes, my goal was to create a blog post title using words no longer than four letters.

Anyway, for those non-SIOP'ers out there, or SIOP'ers that may have missed 'em, there were some gems in the latest issue of TIP:

How I/O can shape the practice of strategic human resources outsourcing (HRO)

A great little study on perceptions of various ways of mitigating cheating on unproctored Internet testing (UIT)

The difference between academics and practitioners in terms of what topics are valued at the SIOP conference (e.g., the latter were more interested in job analysis, staffing, and strategic HR)

Last but not least, a point, counter-point on whether the addition of sex as a protected category under Title VII was a joke

Supreme Court upholds NASA background checks

Yesterday, in a unanimous 8-0 decision*, the U.S. Supreme Court handed federal employers a victory when they upheld a background check process used by the National Aeronautics and Space Administration (NASA).

The checks, which have been standard for millions of federal employees since the 1950's, were challenged by a group of contract employees at the Jet Propulsion Laboratory (JPL) in California who previously had not been required to undergo them. The employees claimed the questions asked during the background checks infringed on their "constitutional right to information privacy" rights by asking about things like drug treatment.

In their decision, the court held that "the Government has an interest in conducting basic background checks in order to ensure the security of its facilities and to employ a competent, reliable workforce to carry out the people’s business" and that the challenged questions were "reasonable, employment-related inquiries that further the Government’s interests in managing its internal operations." It also held that there was no meaningful difference between contract employees and civil service employees.

With respect to questions regarding drug use, the court pointed out that the government used follow-up questions regarding treatment as a "mitigating" factor to separate out drug users from those who are taking steps to address and overcome their problems.

They gave the federal government a low hurdle to jump when it comes to business necessity, rejecting the argument that "the Government has a constitutional burden to demonstrate that its employment background questions are “necessary” or the least restrictive means of furthering its interests."

They also touched on reference checks, writing "Asking an applicant’s designated references broad questions about job suitability is an appropriate tool for separating strong candidates from weak ones. The reasonableness of such questions is illustrated by their pervasiveness in the public and private sectors."

Importantly, the court pointed out several times that concerns regarding the collection of personal information by the government were mitigated by the substantial controls in place to protect such information.

You can read the decision here. You can also see the challenged forms here and here.

*Justice Kagan took no part in the decision

Lewis case emphasizes need for valid tests

On Monday, May 24, the U.S. Supreme Court ruled in Lewis v. City of Chicago that plaintiffs filing an adverse impact discrimination claim under Title VII of the Civil Rights Act have 300 days to file from each time the test is used to fill a position--not just 300 days from when the test was administered.

For the most part, this mainly impacts employers that run an exam and use the results to fill positions for several years. In this case it was a firefighter entry-level exam, and my guess is it will mostly be public sector agencies and large employers that should pay particular attention to the ruling.

Why? Well for one it means more potential adverse impact lawsuits. If you were counting on being safe after 300 days from the exam, that is no longer the case. Second, it emphasizes the need to follow professional guidelines when developing an exam. Employers can successfully defend against an adverse impact case by showing that the selection practice is "job related for the position in question and consistent with business necessity..." (and that no alternatives with similar validity with less adverse impact were available). This means your exams need to be developed and interpreted by people who know what they're doing.

The City claimed that evidence related to an employer's business necessity defense might be unavailable by the time the lawsuit is brought--the court was not swayed. This means you'll want to hang on to your exam development records for at least a year beyond the last time you use the results to fill a position.

Another point worth noting: this case boils down to the validity of a cut score used by the City, which they themselves admitted wasn't supportable. Proper exam development and interpretation includes setting a job-related pass point based on subject matter input and/or statistical evidence that it is linked to job performance.

The Lewis ruling doesn't fundamentally change what we should be doing. It just emphasizes that we need to do it right.

Click here for a quick overview of the facts of the case.

IPAC conference to feature Campbell, McDaniel, Highhouse, and more

Those of you on the fence about attending the 2010 International Personnel Assessment Council (IPAC) conference on July 18-21 may be interested to know that a preliminary schedule has been released that reveals some great speakers and topics. For example:

- David Campbell's provocatively titled opening session, The Use of Picture Postcards for Exploring Diversity Issues Such as Bias and Prejudices, or "How Can We Keep Our Grandchildren From Going to War With Each Other?"

- Not to be outdone, Michael McDaniel kicks things off Tuesday morning with Abolish the Uniform Guidelines.

- Scott Highhouse closes things up Wednesday with A Critical Look at Holistic Assessment

- Great pre-conference workshops on everything from job analysis to fairness

- Wonderfully diverse concurrent sessions on topics such as public service motivation, leadership coaching, simulations, engagement, online testing, charging for exams, test transportability, cross-cultural personality assessment, measuring workforce gaps, adverse impact analysis, faking and lie detection, and succession planning. And that's just a sample!

Staying current on assessment through professional education is one of the commandments of our field. I hope you'll be joining your friends and colleagues in Newport Beach. Early bird registration ends June 1st.

April 2010 TIP and latest EEO Insight

The April 2010 issue of The Industrial/Organizational Psychologist (TIP) is now available here. Let's check out some of what's inside:

- Several great, touching dedications to the late great Frank Landy. I found Rick Jacobs' eulogy particularly moving. My first real job out of graduate school was working for Frank doing research for expert witness testimony. Fabulous experience, amazing individual.

- Joel Wiesen proposes a novel approach to promotional selection in fire department settings.

- A great little article about moving from an I/O position to an HR generalist one. Very timely given all the changes HR departments are experiencing.

- The current status of legal protection for sexual minorities in the workplace

- A great article about the Bridgeport, CT case and why both the media and the city got things wrong.

- A fascinating breakdown of the major activities of I/O consultants v. internal practitioners v. academics. Check out Table 3.


In addition, there's a new issue of EEO Insight with some great content, and one article in particular I'd like to point out: A comprehensive look at how to successfully develop diversity initiatives and testing programs post-Ricci, including the essential elements of a Croson study (see Table 2) and the "strong basis in evidence" standard as applied to several practical examples (Table 3). It's a keeper and starts on page 27.

Some other articles to check out include:

- Strategies for defending and framing the issue of adverse impact in selection

- How to avoid adverse impact when choosing a test

2010 PTC -NC Conference

Last week I was fortunate to attend and present at the 2010 Personnel Testing Council of Northern California (PTC-NC) conference. Several of the presentation slides are now available.

My presentation was a legal update, primarily focusing on last year's big case, Ricci v. DeStefano. While I think the case received a fair share of its publicity simply because Sonia Sotomayor was one of the circuit court judges who ruled for the city, the case itself has some interesting implications for assessment. I gave my two cents last year after the decision.

Some of the points I made during the presentation included:

- Test validation standards as judged by the courts are generally very attainable. Following best practice (i.e., beginning with a thorough job analysis) is a recipe for a defensible process.

- Employers should spend the vast majority of their time before the assessment is given, figuring out what and how to test. Minimal time should be spent after the test making decisions about test usage--you should know that already.

- Employers, and assessment professionals, are expected to be familiar with and consider a wide range of testing mechanisms when planning a selection process. This includes non-cognitive assessments such as situational judgment tests, personality inventories, and biodata measures.

There were many great presentations; I always enjoy hearing what Wayne Cascio has to say, Dale Glaser has a way with statistics, and Deniz Ones and Stephan Dilchert's presentation on personality profiles of leaders was fascinating (they also happen to be very pleasant people to have lunch with). I plan on printing out slide 33 and placing it within reach--it does a great job of pointing out that criterion validity depends greatly on what you're trying to predict!

It's worth reviewing all the slides to get a flavor of what was discussed; there will likely be other presentations added over time.


On a side note, I'd like to acknowledge my readers at Baruch College -- thanks for reading!

Lessons from NYC Fire case - part 2

Part 2 of 2

Last time I discussed five important lessons we can take away from recent rulings in the Vulcan v. City of New York case. In this post I'll review the remaining lessons and also discuss the relief order.

----

6) The city failed to provide sufficient evidence that the exam(s) tested for a sufficient number of the critical KSAs. They also failed to explain why they chose not to measure several KSAs identified as critical.

Lesson: the courts do not require employers to measure every single critical KSA. But there is an expectation that employers attempt to measure a sufficient number that represent a significant portion of the job requirements. In this case, that included non-cognitive abilities such as resistance to stress, teamwork, and conscientiousness, that were not measured.

7) The city failed to adequately consider how to measure a significant number of essential KSAs. While some of their concerns were valid (e.g., structured interviews for all applicants would be an operational nightmare), there are many different forms of testing that should have been considered, including situational judgment tests (SJTs) and biodata, which can be used to measure non-cognitive components.

Lesson: triers of fact expect employers to be up on the various assessment methods available and be able to explain why they chose not to use certain ones. This includes tests that are relatively easy to develop (e.g., SJTs) as well as ones that require substantial resources and statistical expertise (e.g., biodata).

8) The city failed to conduct a reading level analysis on the exams to ensure that it was not "pointlessly high." The plaintiff introduced evidence suggesting the reading level was above 12th grade; in addition, it appeared to exceed the reading level of materials at the academy.

Lesson: never forget that every assessment method is in some sense measuring additional KSAs beyond those you intend. For written exams, reading comprehension is always a requirement (barring accommodation). It's quite easy to conduct a reading level analysis (MS Word has it built in) to ensure that the level is reasonable and matches other job-related material.

9) The city failed to show that the cutoff scores (pass points) established for the exams were based on adequate rationale, namely "the necessary qualifications for the job of entry-level firefighter." Instead, the cutoff scores were based on operational need (the number of job openings expected). This is particularly important in multiple-hurdle selection processes such as in this case, where a failure on one exam component precludes an applicant from participating in the rest of the (potentially compensatory) assessment process.

Lesson: ultimately applicants have to pass the test(s) to be considered for employment. Cutoff scores should be established using the expertise of both SMEs and test developers and should be based on the minimum competency levels required upon entry to the job. At a minimum (and I would not rely solely upon this), the scores should be analyzed to identify any logical "break-points."

----

After ruling for the plaintiffs on both the adverse impact and disparate treatment claims, the judge issued a relief order on 1/10/10. In it, he imposes several things, including the following:

1) The city must develop a new testing procedure for entry-level firefighter in conjunction with the relevant parties. Following the development of the test, there will be a hearing to determine if this test should be used rather than the current test (developed in 2007 and not at issue in this litigation).

2) The court shall develop a process by which the approximately 7,400 applicants covered by this case can file a claim for monetary relief.

3) The city will identify 293 black candidates on the eligibility list and offer them priority hiring. (No quotas are being imposed, although the judge leaves this possibility open)

4) Retroactive seniority for those hired.

In addition, several other issues are up for debate, including the appointment of a special master or monitor, standards that will be relied upon in constructing the new exam, and the need for additional relief.

---

So what did we learn from all this? If you follow--fairly closely--best practices when developing and administering exams, you will be on solid ground defending them. If you don't, and your exam has a discriminatory effect, you may be called on it--and it's not a pleasant process. I'll leave you with this quote from the January ruling on disparate treatment:

"The history of the City's efforts to remedy its discriminatory firefighter hiring policies can be summarized as follows: 34 years of intransigence and deliberate indifference, bookeneded by identical judicial declarations that the City's hiring policies are illegal."

Lessons from the NYC Fire case - part 1

Part 1 of 2

New York City, like the cities of New Haven and Chicago, has a long history of employment discrimination litigation related to its firefighter testing.

Since the 1970s and cases like Guardians, the city has been under scrutiny for its woefully low number of black firefighters.

In 2007 the city found itself faced with another lawsuit over its firefighter hiring practices, and in July of 2009, a U.S. District Court judge found that the city had violated Title VII by administering written exams from 1999-2007 that had high levels of adverse impact. The city marshaled an inadequate defense. In January of 2010, the same judge (Nicholas Garaufis) found the city liable for a pattern and practice of disparate treatment for those same exams. An adverse impact finding, particularly for written exams, and especially for public safety tests, is not earth-shattering. But a finding of disparate treatment in this situation is less common.

This case, while only one example and limited in its impact, has some valuable lessons for test users and sheds some light on how judges look at our field. In particular, I describe below nine points the judge specifically made and what lessons we can draw from them:

1) While the city conducted a job analysis with an "extensive" list of tasks and surveyed incumbents, the city offered "no evidence of 'the relationship of abilities to tasks.'" They conducted a linkage, but the judge found that the SMEs were confused about what they were supposed to do and didn't understand several of the abilities they were rating.

Lesson: simply having subject matter experts (SMEs) link essential tasks and knowledge, skills, and abilities (KSAs) is not sufficient. You need to ensure they understand the statements they are linking as well as how exactly they are supposed to be linking them.

2) In conducting the job analysis, the city inappropriately retained tasks and KSAs that could be learned on the job. It is quite clear (e.g., per the Uniform Guidelines) that only tasks and KSAs that are required upon entry to the job should be identified as critical in terms of exam development.

Lesson: make sure that when you are developing exams based on job analysis results that you focus only on those tasks and KSAs that are required upon entry to the job. This should be determined by your SMEs.

3) The city relied to some extent upon the work of a previous test developer, Dr. Frank Landy (who sadly recently passed away). In addition to a tenuous link between Dr. Landy's work and the current exams, the judge makes it clear that "reliance on the stature of a test-maker cannot stand in for a proper showing of validity." At the same time, the judge emphasizes that exams should be constructed by "testing professionals."

Lesson: tests should be developed by people who know what they're doing. This means HR professionals with the requisite background in test validation and construction in conjunction with job experts. Do not rely solely on previous efforts, particularly when (as in this case) the results of those efforts were either incomplete or not fully relevant to your current situation.

4) The city performed no "sample testing" to ensure that the questions were reliable as well as "comprehensible and unambiguous."

Lesson: few steps in the test development process are as easy--or as valuable--as pilot testing. I have yet to see an exam that didn't benefit from a "trial run" with a group of incumbents. Not only will you catch unintended flaws, you will verify that the exam is doing what you claim it is.

5) There was insufficient evidence that the exams actually measured the (nine cognitive) KSAs the city claimed they intended to measure. Plaintiffs were able to suggest the opposite through analyzing convergent and discriminant validity as well as by conducting a factor analysis.

Lesson: there are two linkages of primary importance in test development. The first was describe in #1. The second is the link between critical KSAs and the exam(s). At the very least, you must be able to show evidence that there is a logical link between the two. When you claim to be measuring cognitive abilities, you incur an additional responsibility, which is gathering statistical evidence that supports this claim.

Next time: more lessons and the relief order.

Outback settlement contains interesting requirements

You may have heard that Outback Steakhouse, a restaurant chain based in Tampa, Florida, has agreed to settle a gender discrimination lawsuit for $19M. What's interesting about this isn't the size of the settlement, but rather the conditions attached.

Background: The EEOC sued Outback in 2006, claiming it systematically discriminated against its female employees by denying them promotion opportunities to the more lucrative profit-sharing management positions. In addition, they claimed that female employees were denied promotional job assignments such as kitchen management, which were required for employees to be considered for top management positions.

The settlement: Outback agreed to a four-year consent decree and $19M in monetary relief. So far, pretty standard. But there were additional settlement requirements, and here's where it gets interesting. In addition to the monetary relief, Outback has agreed to:

1. Create an online application system for employees interested in management positions. This is the first time I've seen this in a settlement (which isn't to say it hasn't happened) and seems to indicate that the EEOC views this as a more "objective" screening mechanism.

2. Create and hire someone for a newly created "human resources executive" position titled Vice President of People. Again, this is a new one for me.

3. Hire an outside consultant for at least two years who will monitor the online application system to ensure women are being provided equal opportunities for promotion and provide reports to the EEOC every 6 months.

The main thing that strikes me about this settlement is the faith that is being placed in an online application system to somehow ensure equal opportunity. Sure, having a standardized application system may cut down on some of the subjectivity of individual hiring supervisors, but it leaves me wondering:

- What will the screening criteria for management positions be?

- How will the outside consultant define "equal opportunities"?

- How will access to the online system be controlled, and who will be making screening/hiring decisions?

- What happens if there continues to be adverse impact, which you would expect if applicants continue to be screened on experience?

- What will be the duties of the Vice President of People, how will they be hired, and how will they interact with the consultant?

This will be interesting to watch.

Is recruiting using SNS discriminatory?

I keep reading/hearing about how recruiting using social networking sites (SNS) opens employers up to discrimination lawsuits because of who uses the sites. For the most part, this just plain isn't true.

A recent Pew study is the latest to show that when it comes to using SNS like Facebook, MySpace, and LinkedIn, you really should have one primary demographic concern when it comes to ensuring a diverse candidate pool: age.

Not gender, at least not in traditional sense. While four years ago SNS users tilted slightly male (55%), the balance has essentially flipped today (54% female).

Not race, there simply do not appear to be generalizable differences in racial groups when it comes to these sites (in fact I've seen some data that suggest the user base on these sites is more diverse)--but things change, and this may vary with particular sites, so keep an eye on this one.

But when it comes to age, SNS users are disproportionately younger than the overall Internet population. In the words of the Pew report, "[this] doesn't mean that more older adults aren't flocking to SNS--they are--but younger adults are ALSO flocking to the sites, so the overall representation of the age cohorts in the SNS user population has actually gotten younger."

One demographic difference I don't see a whole lot about: disability status. Are individuals with disabilities more/less likely to use SNS? I think that's an important question we need to address if we're truly trying to diversity our candidate pools.

Latest EEO Insight

EEO Insight is quickly becoming a great resource for anyone interested in issues related broadly to equal employment opportunity. And this isn't just affirmative action plans--it includes anyone interested in recruitment and assessment.

In the latest issue (v1, #3), you'll read about:

- Alternatives to RIFs such as wage freezes and job sharing and the EEO implications

- Analyzing layoff decisions for statistical evidence of adverse impact

- Using multiple regression to detect race and gender differences in compensation

- Ricci in retrospect and lessons learned

- Reaching out to veterans and individuals with disabilities

- Results of the EEO best practices survey and (very good) recommendations

By the way, if you're interested in EEO issues and you're not already reading OFCCP Blog Spot, I highly recommend starting.

Screening on personality: Legal loophole or pothole?

In a recent post over at ERE, the author mentions a website that provides employers with the ability to screen candidates based on a measure of personality that applicants complete online.

My reading of this article prompted the following internal debate:

Me #1: Boy oh boy, is that ever a bad idea. The Uniform Guidelines clearly state (Q&A #75) that a measure of a trait or construct cannot be validated based on content validity, which is what most employers are likely to rely on in this situation.

Me #2: Ah yes, but because personality tests typically result in much less adverse impact than traditional cognitive tests, are the Uniform Guidelines even likely to come into play?

Me #1: Maybe not, but you never know until you go through a selection process, so why take the risk?

Me #2: What about in cases where the numbers being screened are so small that adverse impact analysis is likely to be wonky? (that's a technical term)

Me #1: Well that may well be different, but you're missing the point.

Me #2: What is the point?

Me #1: That employers should use caution before screening based on constructs such as personality. They need to take validation seriously.

Me #2: But isn't it good that they're using an instrument that's at least based on an evidence-based theory of personality (the Big 5)?

Me #1: Absolutely, and props to them. But it is still incumbent on employers to realize the legal risks as well as the implications of using a self-report personality screen as a first hurdle.

Me #2: Fine, but aren't you being a little hypocritical? Haven't you said one of your goals is a giant database where applicant information can be matched with employer needs?

Me #1: True, but I was thinking more along the lines of verifiable skills testing, not self-report inventories.

Me #2: Actually in that post you specifically refer to Big 5 assessments.

Me #1: Hey! This isn't about me. This is about warning employers to make sure they know what they're doing when they screen based on personality measures.

Me #2: Aren't you making an awful lot of assumptions about the website's process without having actually used it or talked to the owners?

Me #1: Well, yes, but I'm a blogger. That's what we do.

Me #2: Ugh.

Ricci presentation

On August 13th I gave a presentation at a PTC-NC luncheon about the Ricci decision. We had a great discussion about the implications (which remain to be seen) and dissected several passages from the decision.

One of the questions that came up had to do with the pass point. I didn't know the answer at the time, but looking back at the case it turns out that the City charter mandated a 70% pass point for these exams. Which is funny, because I made a joke about how 70% is the magic cutoff score given its ubiquity, particularly in the public sector.

In a perfect testing world would the pass point be based on an analysis of the minimum competency level required for the job? Yep. Did the 5-member majority in this case care? Nope.

You can view the (mostly visible) slides below.

Research update

Here are some research studies that you may have missed:

Predicting applicants' job pursuit behavior from their selection expectations: the mediating role of the theory of planned behavior

Complexity in the relationship among the subdimensions of extraversion and job performance in managerial occupations

The use of personality test norms in work settings: Effects of sample size and relevance

Networking as a job search behaviour: A social network perspective

Bonuses:

Avue recently published a white paper titled "The state of the federal workforce: A case for insourcing" that, while focusing on the federal government, has some good information and tips for all of us.

Can't get enough analysis of the Ricci decision? Check out this webinar on 8/14.

Reminder: Want more frequent updates? You can follow my shared items here.

Ricci webcast on August 12

The Ricci v. DeStefano decision continues to generate a lot of interest. To help sort it all out, the Personnel Testing Council of Metropolitan Washington, D.C. (PTC-MW) will host Dr. James Outtz, renowned I/O psychologist and co-author of an amicus brief in the case, on August 12.

Not in D.C.? Not a problem. The luncheon presentation will be webcast at an extremely low price. Check out the website for details.

Coincidentally, a much less well known individual (yours truly) will also be presenting on the Ricci decision at PTC-Northern California (PTC-NC) at their August 13th luncheon.

By the way, check out some great commentary about the decision by several SIOP members here. I find it fascinating that SIOP came out strongly against the validity of the exam, to which the majority of the Supreme Court responded, "yawn."