How good are you at test accommodation?

How good is your organization at accommodating individuals with disabilities when test-time rolls around?

A recent article in Diversity Executive magazine highlights the work of Certiport, a software certification outfit, and the different test accommodations they offer, including:

- Voice recognition software

- Test assistants/surrogates

- Separate, larger rooms

- Extended test times

The article also points out some disturbing facts, like the 70% unemployment rate of individuals with disabilities and the fact that 2 out of 3 of these unemployed individuals would like to work.

Makes me wonder (for the millionth time) about the true nature of the looming "talent shortage"...

Rocket-hire releases 5th annual survey results

I don't usually link to a single article, but this is an exception. Every year, Rocket-Hire and our friend Dr. Charles Handler release the results of an annual survey on the usage of online screening and assessment tools. This year marks the fifth survey and it's got some interesting results based on responses from 141 recruiting and hiring professionals:

1. Many, if not most, organizations are using an ATS. For large organizations (5,000+ employees), usage was 100%.

2. Surprisingly, many organizations are not using online qualification prescreening. Only 47% ask about basic qualifications while only 24% ask about technical skills or certifications. Why surprising? Because it's one thing ATSs tend to be decent at.

3. Use of personality measures as an assessment tool jumped in the last five years from 21% to 59% (which certainly matches the interest I'm seeing). In fact, they were the most popular assessment tool reported, followed closely by skills/knowledge tests. Least used? Simulations and online interviews. I see lots of potential in the former as our technology improves.

4. Assessment tools in general are more widely used than they were five years ago. "Fit" measures went from 29% to 40%, cognitive ability tests from 26% to 41%, and skills/knowledge tests leaped from 12% to 56%. This is good news indeed (assuming the tests are good!).

5. Unfortunately only 27% of users of prescreening tools and 36% of assessment users collect metrics to measure their success. Those that did were much more likely to find these measures effective. Hey, you can't know if you don't measure!

There's a lot more in the article (e.g., take a look at biodata usage). By the way, full results and analysis will be in the May issue of the Journal of Corporate Recruiting Leadership.

Applied Workforce Planning: Air Traffic Controllers

There's plenty of disagreement over whether the aging of the U.S. workforce will indeed spark a wave of simultaneous retirements and thus a scramble to replace employees. Personally, I think it's going to depend on a lot of factors--the housing market, health care costs, and technological advancements to name a few.

But there are instances that demonstrate what it could look like if a significant portion of the U.S. workforce retires. One of those instances is the air traffic controllers who work for the Federal Aviation Administration (FAA) and were the subject of a recent piece on NPR. Let's look at the situation and the implications:

- In the summer of 1981, President Ronald Reagan replaced 12,000 striking air traffic controllers all at once. When you make such a large number of hires, in the same classification, with a lot of them probably being around the same age, in public sector (where folks often stick around for a long time), you better be thinking about workforce planning.

- Thirty percent more controllers retired last year than the Federal Aviation Administration (FAA) predicted. That's significant, and whether it's due to insufficient planning or not, it provides an example of the scope of the problem we're facing.

- Last year the FAA imposed a new labor contract on the controllers which lowered pay for new hires, froze pay for those with longevity, and placed new restrictions on the working environment. Not a great way to attract new candidates into a field where you need a high number of replacements in a short period of time.

- According to the union, employees feel stretched and burned out, which may lead to a serious accident. Not the type of publicity that demonstrates an outstanding employer brand.


Lessons? Like politics, all workforce planning is local. And yes, it's an inexact science. But sometimes the numbers just hit you over the head and demand attention. Do you have a situation like this in your organization? If so, what are you doing about it?

You don't know you

In a recent interview with Gallup Management Journal, Cornell psychologist David Dunning talked about why people aren't very good at judging themselves. Why is this important? Because it has a great deal to do with how we recruit and assess applicants.

A big part of the reason we're so bad at accurately judging ourselves is due to our self-serving bias--our tendency to take credit for successes but blame outside factors (other people, equipment) for our failures. This helps our ego out--if we were always blaming ourselves for failures and attributing success to outside factors, we wouldn't be very happy campers. But it has the downside of oftentimes blinding us to the real reason why things happen.

Dr. Dunning covers a wide range of topics in the interview, including gender differences, when overconfidence may be a good thing, employee training, providing feedback, and the serious implications of this phenomenon (e.g., think about doctors judging their skills as being better than they are).

As I said, this is important because a great deal of recruitment and selection is about self-assessment--a prime example is the growing movement toward online training and experience (T&E) questionnaires made easier with the spread of ATS products. Many of these questionnaires are chock full of questions that (no joke) aren't much different than: "How great are you at X?"

But it's not just about T&Es. People make judgments about themselves when deciding what jobs to apply for in the first place. They describe themselves in certain ways during job interviews (when the motivation to make yourself look good is even stronger).

What can we do about it? Simply put, verify, verify, verify. If someone claims to be the greatest Java programmer on the planet, make them show you. If they claim to be a great orator, make an oral presentation part of the hiring process. Then talk to folks that know their work to establish a history of competence. Don't take someone's word at face value because (a) they may be trying to snow you, but more subtly (b) they may not know themselves.

By the way, Dr. Dunning is co-author of one of my all-time favorite articles, "Flawed self-assessment: Implications for health, education, and the workplace" which describes how the inability of people to judge themselves accurately can result in very serious problems.

Last thing: if you're not already familiar with it, check out the fundamental attribution error, which is one of the other big things our brain is constantly doing. It has huge implications for how we judge others.

Rigorous assessment pays off

It's great when the mainstream press gets assessment right. It doesn't happen a lot, so I want to make sure to point out a good example.

Ellen Simon (AP) devoted a recent article to employers that, even in a tight labor market, put job applicants through the paces.

Some of my favorite bits from the article:

- Employers that recognize their employees are an integral part of their brand. If your employees are unhappy, not trained, or otherwise a bad fit, customers (and potential applicants) notice.

- This quote from Rackspace Managed Hosting CEO Lanham Napier: "We'd rather miss a good one than hire a bad one." Without getting into Type I versus Type II errors, let me just say that Mr. Napier demonstrates the wisdom of someone who's seen what a bad hire can (or can't) do. (Check out their refreshingly simple career portal)

- The fact that Rackspace interviews last ALL DAY. Yep, all day. In this age of "I only have 30 minutes for the interview", that's darn refreshing.

- The wonderful use of realistic job preview videos by Lindblad Expeditions that show employees cleaning toilets and washing dishes. Says Kris Thompson, VP of HR, "If they get on board and say, 'This is not what I expected,' then shame on us." Check out how their online preview video combines push with pull.

I don't agree with everything in the article--I'm not a big fan of the idea of secretly judging people on their waiting room behavior--but all in all some great examples here to recognize.

(by the way, the HBR article Simon cites, called "Fool vs. Jerk: Whom Would you Hire?" is here.

October '07 TIP: Alternatives and Title VII

You legal buffs out there know that under Title VII of the Civil Rights Act of 1964 (as amended in 1991) there exists a "burden shifting" framework that lays out how an employment discrimination case (hypothetically) proceeds:

1 - The plaintiff must show that the employer is using a particular employment practice (e.g., a selection test) that results in disparate (or adverse) impact against a legally protected group; if successful,

2 - The employer must show that the practice was/is job related for the position in question and consistent with business necessity; if successful,

3 - The plaintiff must show that there is an alternative employment practice (e.g., a different type of test) that would serve the employer's needs, be equally valid, and have less adverse impact and the employer refuses to adopt it. The classic case is plaintiffs suing over a written knowledge test and suggesting a personality or performance test should have been used.

You may also know that plaintiffs rarely win employment lawsuits (for many reasons, but one of which is employers are getting better at #2 above), and there seems to be a shift toward the third prong of the case--showing that there are alternative testing mechanisms out there that are equally as valid and with less adverse impact.

The October issue of the Industrial-Organizational Psychologist (TIP) contains two articles (both by individuals who have served as expert witnesses in discrimination cases) that touch on this subject and are worth a read:

Slippery slope of "alternatives" altering the topography of employment testing? by James Sharf

and

Less adverse alternatives: Making progress and avoiding red herrings by James Outtz

Also in this issue, a great analysis of the recent U.S. Supreme Court ruling in Parents v. Seattle School District by Art Gutman and Eric Dunleavy that reviews in detail the current status of affirmative action.

Links a go-go for October 1, 2007

Good reading for October 1, 2007

The new affirmative action (about schools, but lessons for employers)

2007 ILG National Conference Highlights

Don't automatically dismiss people that been fired

Court rules EEOC may proceed with discrimination case against L.A. Weight Loss

Visa and using credit scores in the hiring process

Hiring supervisors and leaders (the #1 problem of most organizations, IMHO)

Deloitte demonstrates just how creative recruiting can be

How many names does it take to get to a hire?

Who does The Gap think it is? Monster?

Are individuals liable for employment discrimination?

A common question I hear from supervisors and HR professionals is: "Am I personally liable for employment discrimination when I make a hiring decision?"

This recent article deals with a California Supreme Court decision but covers the answer to this question generally.*


* Short answer: it's rare (except for Section 1981 or 1983 claims** and failing to verify employment eligibility***) but you may be named anyway as a tactic on the part of the plaintiff.



** Which can be particularly nasty since there is no cap on damages and no administrative requirement (like filing with the EEOC). On the other hand it is more difficult for plaintiffs to prevail in these cases, and it's only relevant in cases of disparate treatment.



*** Okay, this might be nastier because you could face jail time. Don't forget those I-9s!

August ACN

The August, 2007 issue of Assessment Council News is out and Dr. Mike Aamodt provides his usual great writing, this time in article titled, "A Test! A Test! My Kingdom for a Valid Test!" where he goes over what you need to look for when selecting a commercially available test...in two easy steps!

Some of my favorite quotes:

"Previously, [the] clients had their supervisors create their own tests, and we advised them that this was not a good idea." (I just like the idea of saying that to clients, aside from the fact that it's true 99% of the time)

"Creating a reliable, valid, and fair measure of a competency is difficult, time consuming, frustrating, costly, and just about any other negative adjective you can conjure up. Think of the frustration that accompanies building or remodeling a home and you will have the appropriate picture." (So it ISN'T a coincidence that I enjoy testing and home remodeling. Whew.)

"...it is essential to remember that no test is valid across all jobs and that criterion validity is established by occupation, and depending on who you talk (argue) with, perhaps by individual location." (Just don't tell this to Schmidt and Hunter.)

More info about ACN, including links to past issues, here.

And by the way...major kudos to Dr. Aamodt for offering so much of his work online. This is rare and to be commended.

Great July 2007 Issues of Merit

The U.S. Merit Systems Protections Board (MSPB) puts out a great newsletter focused on staffing called Issues of Merit.

The July 2007 edition has some great stuff in it, including:

- Risks inherent with using self-assessment for high-stakes decisions, such as hiring (hint: people are horrible at it)

- Tips for workforce planning

- How to write good questions

- Analyzing entry hires into the federal workforce

- An introduction to work sample tests

Good stuff!

Is USAJobs enough?

Check out this article that came out recently on GovernmentExecutive.com. It's about how the federal government may need to branch out and start using other advertising venues besides USAJobs.gov, which it relies on heavily.

Some individuals quoted in the article, which happens to include a manager at CareerBuilder, point out that:

- Opportunities are not automatically posted on other career sites, like CareerBuilder, Monster, and HotJobs.

- Job openings are not "typically" searchable through search engines like Google. (Although look what happens when I search for an engineering job with the federal government).

- You can't expect people to automatically look for jobs on USAjobs.

The Office of Personnel Management (OPM), the fed's HR shop, fires back:

- USAJobs gets 8 million hits a month. This compares to CareerBuilder's 1.2 million searches a month for government jobs.

- USAJobs is well known and marketing efforts have been ramped up (e.g., last year's television commercials, which unfortunately didn't work with my version of Firefox).

So who wins the argument? I don't think the feds need to panic just yet. But it can't hurt them to investigate other posting opportunities, particularly given how much traffic the heavy hitters like Monster and CareerBuilder get compared to USAJobs:

By the way, don't overlook the comments on that page; in some ways they are more telling than the article. Readers point out that the application process is overly complicated--to the point that one of the readers makes his/her living guiding people through the process (reminds me of a guy that does the same thing for the State of California). My bet is the application process is equally, if not more, important than how the feds are marketing their opportunities.

I would also be willing to bet that it isn't just the feds that have this issue. As more organizations implement automated application and screening programs, they risk falling in love with the technology at the expense of the user experience. I may love the look of your job, but if it takes me 2 hours to apply, well...I may just look elsewhere.

B-school requires PPT slides for admission

So apparently Chicago's Graduate School of Business is going to require four pages of PowerPoint-like slides as part of its admission process this fall.

According to school reps, this will allow students to "show off a creative side that might not reveal itself in test scores, recommendations and even essays." Another rationale given by the school is that students will have to master this type of software before entering the business world.

One problem I see here is the vast majority of applicants will already know PowerPoint--if you get through high school and college without using it, I'm betting you're the rare applicant.

The larger problem here is the same problem employers face with supplemental questionnaires and work samples--namely, who did it? In high-stakes situations like school admissions and job applications, people are known to take, shall we say, less than ethical routes to increase their chances.

The benefit of something like GPA or the GMAT is identity verification--you can be virtually assured (as long as you can validate the numbers) that the person who's applying is the one that took that test.

With things like previous work samples, resumes, and now this PowerPoint idea, you have no idea who actually created the product. So you make an admissions or hiring decision based on an assumption. Do you validate that they actually created these documents? Probably not. Even if you wanted to, how would you do it?

It might not even matter, since this may be more of a way to add excitement to application reviews and to simply get more applicants, which the school acknowledges. There seems to be a trend among organizations to implement projects that aren't so much concerned with valid predictions of performance but with simply attracting attention. This will likely get even more blatant as organizations struggle to keep their staffing levels up in the coming years.

But we should keep this in mind: gimmicks may attract some applicants, but do they turn others away? What about highly qualified individuals who think, "Well that's silly." That's why the best solutions will pique interest while being as close as possible to the actual job (or school) environment. How about asking applicants to give a presentation as part of their interview--now that's a skill they'll need. Plus, absent any Mission Impossible-style disguises, you can be pretty sure the person in front of you is who they claim to be.

Links a go-go for July 27, 2007

Good reading for July 27, 2007:

47% of U.S. adults have high-speed Internet connections at home

Is someone trash talkin' your organization?

June U.S. search engine rankings and career development sites rankings

Think about making your own talent scouts

Good news for public sector and non-profit organizations

Can you consider an applicant's accent when making a hiring decision?

Here comes the OFCCP

Links a go-go

Good reading for July 17, 2007:

What the Seattle School District case means for employer diversity plans (decision here)

EEOC gets a slap on the hand for its use of press releases in litigation

Responding to SSA "no match" letters: A how-to guide

Jamie over at selectionmatters.com talks July TIP and world domination

EEOC and Walgreens settle race discrimination lawsuit

Wendell Williams with advice we wish everyone would listen to

How not to conduct a reference check

New blog to watch

Michael Harris, previously of EASI-HR Blog, has started a new blog titled HRMplus.

Dr. Harris is a professor at the University of Missouri-St. Louis where he teaches HRM. He has also served as an expert witness on discrimination issues, as a trainer, and a consultant. He presented at the most recent IPMAAC conference on Disparate Impact and Employment Testing: A Legal Update.

Check it out!

Men's Wearhouse passes on background checks

A recent Business 2.0 cover story, titled "Ripping up the Rules of Management", mentions that George Zimmer, the founder of clothier Men's Wearhouse, has a policy that no employee or interviewee will ever undergo a criminal background check.

Seem risky? Well it doesn't appear to be hurting them. In fact, the company loses only .4% of revenue to theft, much less than typical for big retailers (1.5%). Says Zimmer:

"I don't trust the U.S. justice system to get it right...I'd rather make my own decisions, and I believe in giving people a second chance."

This policy is particularly interesting given efforts by various jurisdictions to limit criminal history checks in employment screening as well as the EEOC's renewed focus on criminal history checks as part of its new E-RACE Initiative.

Assessment in the Mainstream Press

Boy, folks sure have personality tests on the brain these days.

This article on Forbes.com, titled "Surviving the personality test" discusses how organizations are using personality tests these days and also touches on the issue of "cheating."

Tip of the hat for...

- Talking to real experts in personnel selection, such as Robert Hogan and Murray Barrick. Too often journalists seem like they just pick the first name in the yellow pages.

- Pointing out that using a good personality inventory in conjunction with an interview can increase predictive power.

- Emphasizing the importance of personality inventories being standardized (and the fact that interviews often aren't).

- Reminding everyone that characteristics required of one job aren't required of all jobs. It's a scary moment when a hiring manager wants to use the same assessment, in the same way, for every job.

- Encouraging people to be honest, as the point here is to match what the person can do (and wants to do) with what the organization needs. Not that this will necessary work.


Wag of the finger for...

- Saying that personality tests are "not trying to discern whether you're an extrovert or an introvert." They sure as heck ARE if you're hiring for a job where those traits are associated with success (e.g., long haul trucker, salesperson).

- Claiming that personality tests can't be cheated on. Sure they can. We know they can. It's not always easy to do so, but it doesn't take a rocket scientist to realize that questions like "do you like to go to parties?" are probably getting at extroversion and that's probably important for jobs like a restaurant host/hostess. But the more important point is it doesn't seem to matter all that much.


All things considered, not a bad article. I give it a B+.

Anxiety Impacts Test Scores

One of the (many) news articles that came out of this years' AAAS meeting has to do with anxiety's impact on test performance.

Specifically, a panel titled "Interplay of Emotion and Cognition: Implications for Learning and High-Stakes Testing" included a presentation by Dr. Mark Ashcraft on "How Math Anxiety Compromises Performance: The Role of Working Memory."

Dr. Ashcraft (note the article has this misspelled "Ashcroft") noted that math anxiety takes up working memory that is particularly important for complex math problems.

I'm not quite sure how "math anxiety" differs from plain 'ol test anxiety, but it looks like Ashcraft has written quite a bit about it.

Frankly, this finding is not particularly surprising given all we know about stress' impact on the body, but it does serve to highlight that there are many things that can introduce error variance into someone's test score.

So what can we do about it? The research suggests that orientation/training programs can help with reducing people's anxiety. There are also simple steps you can take as part of the testing process, such as:

- Letting people know exactly what the testing process will be like

- Being available for questions about the process

- For interviews, doing things like offering water, pre-exposing some (or all) of the questions, and telling the candidate to take their time

- Using multiple assessment methods so if a candidate "blows it" on one method they can shine elsewhere

On a related note, although taking these steps can help, they probably won't decrease any differences typically observed between different groups (e.g., men/women, White/Black).

Speaking of group differences, one of the other presenters on the AAAS panel was Dr. Joshua Aronson (of Steele & Aronson fame), who presented on "Stereotype Threat and Sex Differences in Math Performance: New Findings"--looks like he's done some research lately on this topic.

When Hiring a Surgeon, Should We Ask If They Play Video Games?

You may have seen this news article today, about a study published in the most recent issue of the Archives of Surgery. The study found a strong relationship between video game experience/skill and performance on a surgical skills test among a sample of 33 residents and attending physicians at Beth Israel Medical Center in New York.

How did the authors come to this conclusion? They had the participants play some games, fill out a survey about their game playing experience, and take a simulation test (interestingly called "Top Gun").

Correlation, causation, whatever

The authors note that "video game skill correlate[d] with laparoscopic surgical skills." But here's where everything goes to heck and those of us who have taught statistics hang our head in shame. One of the fundamental lessons in statistics is that correlation is not causation--just because two things relate to one another doesn't mean one CAUSES the other. My favorite example of this is the positive correlation between ice cream cone purchases and crime rate. Does this mean ice cream cone purchasing causes crime? Or, um, maybe there's a third variable (like heat)?

The news outlets, feeding off the study authors, picked this up and ran with it, suggesting that playing video games somehow causes you to be a better surgeon--or even that kids that play video games could be somehow preparing themselves for a high paying career.

But here's the thing: this study did not have unexperienced video gamers play a game then perform surgery. It's purely point-in-time. All this study showed is that folks that are good at video games are good at the surgical task. Makes sense--they involve some of the same skills. But that's all we can really say. We can't suggest that people play more video games if they want to be better surgeons (and to their credit the news articles generally point this out).

Biodata pops up again

On the other hand, this type of information could be useful for another purpose--biodata. Google has been in the news recently (discussed here and elsewhere) for re-vamping its hiring practices to include biodata. So as part of a hiring practice, questions could be asked about hobbies, one of which could be video games. But you certainly wouldn't want this to be your entire selection process.

On the other hand, with so many kids playing video games these days, is there really going to be a lot of variance in experience 10 years from now for us to use in separating out applicants? Maybe by then we can ask applicants if they've completed Expert level on Surgeon Wars.

Good to Great: Business v. Social Sector

No doubt many of you have read Jim Collins' 2001 best seller, Good to Great: Why Some Companies Make the Leap...and Others Don't.

What you may not have read is a small monograph that was published four years later, titled Good to Great and the Social Sectors.

The line on the front, "Why business thinking is not the answer" tells part of the story. Collins' basic point is that while some lessons of great businesses can be fruitfully translated from private to social sector organizations (e.g., discipline), others cannot (e.g., financial measures of output).

This point has been made before, but the desire to mold social sector organizations into businesses keeps cropping up, and the HR field seems to be fertile ground for this tempting idea.

Tempting because many people, particularly those in the social sector, often feel (a) frustrated with inefficiencies, and (b) hopeful that lessons from the private sector, if adopted, would transform these organizations into lean, productive stars, garnering praise and respect from constituents.

Measuring greatness in the social sector

Collins gives several good examples of how social sector organizations have dealt with one of the most vexing challenges--how to measure performance:

- The New York Police Department went from focusing on "input variables" such as arrests and cases closed to outputs--namely, crime rate.

- The Cleveland Orchestra measures outputs such as number of standing ovations, number of invitations to prestigious festivals, and influence on other orchestras.

How does this relate to recruitment & selection?

One of Collins' big points from the 2001 book is that hiring the right people comes first--or as he states, "First who--getting the right people on the bus." While I would argue that you need to know some of the "what" before you can get the "who" (e.g., what are you looking for?), he makes some sound arguments in the monograph:

- Once the right people start joining the group/organization in large enough numbers, low performers/poor fits often self-select out, like "viruses surrounded by antibodies." (We'll generously ignore the fact that he's comparing people to viruses)

- "There is no perfect interviewing technique, no ideal hiring method." Can't agree with this one enough, although his solution (focus on a probationary period) is easier said than done. I would argue a reasonable amount of time spent upfront on defining the job and being smart about the hiring methods used saves time and energy down the road on poor fits. You wouldn't buy a house based just on a 30 minute tour--why would you hire someone this way?

- "The more selective the process, the more attractive a position becomes." Again, agree with him here. Organizations, whether private or social sector, need to make positions desirable to the highly qualified. Demanding selection procedures haven't kept Microsoft, Toyota, or Google from attracting top talent.

- Last but not least, "the social sectors have one compelling advantage: desparate craving for meaning in our lives. Purity of mission...has the power to ignite passion and commitment." I believe this is what attracts and retains most of the top talent in the social sector, and it's the factor that organizations need to capitalize on.

Social sector organizations aren't making widgets, they're keeping people from having their homes robbed, keeping children from being molested, keeping drinking water safe, etc. Those are powerful outcomes. Use them to recruit and retain the most qualified.