Unproctored internet testing: Safe for some tests?

One of the biggest trends in personnel assessment is the movement toward on-line testing. Many organizations are experimenting with so called unproctored Internet testing (UIT), where candidates are allowed to take the exams whenever, and wherever, they want.

Benefits? Extremely convenient for the candidate. Less administrative resources needed by the employer.

Costs? Bye-bye exam security, hello cheating opportunities. Not only is your test out for everyone to see, but you have no real way of knowing (sans biometric verification) who is taking the test.

Some organizations have decided the benefits outweigh the risks, and a new study in the June 2008 issue of the International Journal of Selection and Assessment may provide support for their position.

In it, the authors looked at over 800 applicants from nine European countries that took a test of perceptual speed in an unproctored setting, then followed this up with a proctored parallel version. Results? Not only was there no evidence of cheating, they found the opposite effect--people did better in the proctored setting.

Now before everyone throws out their proctored exams, note that this is a type of test that might be hard to cheat on--at least in one way. Because this is a perceptual speed test, there are no "right" answers that can be looked up. It also required very quick responses. So the only way to cheat would be to have someone take the test for you. Implication: it may make more sense to use certain UITs than others.

This topic is a source of much debate in the assessment community, and there is by no means consensus on the right way to go. But studies like this help!

Take a deep breath, because there's a lot more in this issue:

- The preliminary employment interview as a predictor of assessment center outcomes (fascinating look at how the AC may only make sense for mid-range interview scorers)

- A comparison of the common-item and random-groups equating designs using empirical data (for you IRT fans out there)

- The influence of external recruitment practices on job search practices across domestic labor markets: A comparison of the United States and China

- Beneath the surface: Uncovering the relationship between extraversion and organizational citizenship behavior through a facet approach (a more nuanced look at the relationship shows extraversion can predict OCBs)

- Comparing personality test formats and warnings: Effects on criterion-related validity and test-taker reactions (another good one...personality test added predictive validity beyond ability test but no validity difference between forced-choice and Likert scales, nor between warning and no-warning conditions; forced-choice and warnings may produce negative candidate reactions)

- Applicant selection expectations: Validating a multidimensional measure in the military (describes development of a new measure of applicant perception of the selection process)

- Selecting for creativity and innovation: The relationship between the innovation potential indicator and the team selection inventory

New blog: Criteria

Let's welcome a new blog in town. The new entrant is a corporate blog from Criteria, an employment testing firm based in LA.

They start off with a bang, taking a clever look at the relationship between Wonderlic scores (a measure of cognitive ability) and performance in the NFL.

Good stuff, check it out.

Rocket-hire releases 5th annual survey results

I don't usually link to a single article, but this is an exception. Every year, Rocket-Hire and our friend Dr. Charles Handler release the results of an annual survey on the usage of online screening and assessment tools. This year marks the fifth survey and it's got some interesting results based on responses from 141 recruiting and hiring professionals:

1. Many, if not most, organizations are using an ATS. For large organizations (5,000+ employees), usage was 100%.

2. Surprisingly, many organizations are not using online qualification prescreening. Only 47% ask about basic qualifications while only 24% ask about technical skills or certifications. Why surprising? Because it's one thing ATSs tend to be decent at.

3. Use of personality measures as an assessment tool jumped in the last five years from 21% to 59% (which certainly matches the interest I'm seeing). In fact, they were the most popular assessment tool reported, followed closely by skills/knowledge tests. Least used? Simulations and online interviews. I see lots of potential in the former as our technology improves.

4. Assessment tools in general are more widely used than they were five years ago. "Fit" measures went from 29% to 40%, cognitive ability tests from 26% to 41%, and skills/knowledge tests leaped from 12% to 56%. This is good news indeed (assuming the tests are good!).

5. Unfortunately only 27% of users of prescreening tools and 36% of assessment users collect metrics to measure their success. Those that did were much more likely to find these measures effective. Hey, you can't know if you don't measure!

There's a lot more in the article (e.g., take a look at biodata usage). By the way, full results and analysis will be in the May issue of the Journal of Corporate Recruiting Leadership.

Too fat or too thin? You may not get hired.

Job candidates that are either too fat or too thin may have a more a difficult time getting hired than those in the middle weight ranges according to a study by Swami, et al. reported in the most recent issue of the Journal of Applied Social Psychology.

Weighting in line
The authors found that when men were asked to rate a variety of female pictures for either a management position or for providing help (N=30 and 28, respectively), they were less likely to hire or help women with body mass indices (BMI) over 30 or under 15. Those with a slender body (BMI = 19-20) were most likely to be hired or helped. This shouldn't be surprising, given that studies have consistently linked physical attributes, including weight, with employment decisions, but it's certainly a reminder to watch your biases when evaluating candidates!

Predict-ability
In another article, Truxillo et al. found a relationship between cognitive ability and the ability to accurately judge one's performance on an employment test. Using a video-based situational judgment test of customer service skills, the authors found that those with high cognitive ability were able to predict their performance while those with low cognitive ability were not. Practical implications? Providing thorough test feedback may be particularly important for candidates lower in cognitive ability as they may be more likely to be surprised (and dismayed) by the results. This means providing information prior to the test as well as afterward (e.g., how it was developed, how it is scored, how you can improve your performance).

Working IT
In a third study, Johnson, et al. found gender and ethnic group differences in how IT careers are perceived as well as in self-efficacy related to IT. Using data from 159 African- and 98 Anglo-Americans, the authors found that African American men reported higher levels of IT self-efficacy than all other groups, whereas Anglo women reported the lowest levels. In addition, Anglos had more negative stereotypes of IT professionals than did African Americans. This study had a small sample size, but the implication is that how people see their own ability related to an occupation, as well as how they perceive those in it, influences their career choices. This will in turn impact your applicant demographics as well as your recruiting success.

The rest
There are some other interesting reads in here, including:

When emotional displays of leaders may increase follower performance

How to give performance feedback

Self-perceptions of ethical behavior

Content of the year

At the end of this, the first full year of this blog's existence, I decided to take a look back at 2007 and give you my Top 5 most popular posts of the year:

1. Jobfox plays matchmaker (there continues to be significant interest in Jobfox and their non-traditional approach to matching applicants with employers)

2. Reliability and validity--it's okay to Despair(.com). Whether it's the statistics words or Despair, I'll never know. But people sure like those little posters (and remember, you can make your own).

3. Personality testing basics (Part 2). As you can see from the sidebar survey, folks continue to be very interested in personality testing.

4. Wonderlic revises their cognitive ability test. Wonderlic, one of the oldest and most famous testing companies, continues to generate interest.

5. Checkster and SkillSurvey automate reference checking. There's further development to be had, but I do believe these tools could be a boon to HR and supervisors alike.

Okay, so enough about me. What about what everyone else is writing about? Here are my nominations for content of the year:

1. Morgeson et al. fired a shot across the bow of personality testing with their piece in Personnel Psychology that resulted in multiple, shall we say, not so thrilled responses. I don't know where this debate is going (although I suspect alternate measurement methods will play a part) but it sure is fun to watch!

2. There were some great books I came across this year. Particular props for Understanding statistics, Evidence-based management, and Personality and the fate of organizations. Yes, they were all published in 2006...are you saying I'm behind?

3. Dineen et al.'s great little piece of research on P-O fit and website design in the March issue of J.A.P. that I wrote about here. Take a look at your career website with these results in mind.

4. The Talent Unconference was a big success, and I'm very thankful that many of the presentations were videotaped; I put up links to some of them here

5. McDaniel et al.'s meta-analysis of situational judgment test instructions. Not only is this a great piece of research, it's (still) free!

So what about my New Years wish from last year? I'm still waiting. Although if people search databases like Spock eventually get up enough steam...perhaps I'll get my wish?

Here's to hoping 2008 is filled with interesting and useful things!

Links a go-go

A smörgåsbord of good reading for this holiday week, 2007:

More evidence that IQ has a environmental component

Teens creating Internet content in greater numbers

New law increases mandatory retirement age for commercial pilots

Wednesday best day of week to send out recruitment e-mail (hat tip)

How to improve your corporate career website (Part I, Part II, Part III, Part IV)

Ford, et al. wrap up EEOC lawsuit over written exam

Free ATS! No, really! (Okay, there are several installation steps)

...and last but not least, for those of you doing major shopping this week:

Recruiting at the mall (I've wondered why more orgs don't do this!)

November '07 Issue of J.A.P.

The November 2007 issue of the Journal of Applied Psychology is full of interesting articles, including several relating to recruiting and assessment. Let's take a look:

First, a field study by Hebl et al. on pregnancy discrimination. Female confederates posed as job applicants or customers at retail stores, sometimes wearing a pregnancy prosthesis. As "pregnant" customers, they received more "benevolent" behavior (e.g., touching and over-friendliness), but as job applicants they received more hostile behavior (e.g., rudeness). The latter effect was particularly noticeable when confederates applied for stereotypically male jobs. This isn't a form of discrimination that gets as much play as others, but may be much more common than we think. My guess is a lot of people associate pregnancy with impending time off and don't focus as much on the competencies these women bring to the job.

Second, a study on faking. But wait, not faking on personality tests, faking during interviews. Levashina and Campion developed an interview faking behavior scale and then tested it with actual interviews. Guess what? Scores on the scale correlated with getting a second interview. (Looks like those classes you took on answering vaguely are going to pay off!) But wait, there's more. The authors also found that behavioral questions were more resistant to faking than situational questions (another reason to use 'em!), and follow-up questions INCREASED faking (another reason NOT to 'use em!). Other goodies in this article: over 90% of undergraduate job candidates fake during employment interviews (I assume that's just this sample), BUT, the percentage that were actually lying, or close to it, was less (28-75%).

Third, Brockner et al. provide research results that underline how important procedural fairness (justice) is. Three empirical studies demonstrated that employees judge organizations as being more responsible for negative outcomes when they experienced low procedural fairness. So when applicants or employees get bad news, they'll blame the organization even more if they feel the process used was unfair. Why do we care? Because perceptions of procedural fairness impact all kinds of things, including recruiting (e.g., how someone reacts to not getting a job) and the likelihood of filing a lawsuit (for, say, discrimination).

Fourth, Lievens, Reeve and Heggestad with a look at the impact of people re-taking cognitive ability tests. Using a sample of 941 candidates for medical school that took an admissions exam with a cognitive component, the authors found that retesting introduced both measurement and predictive bias: the retest scores appeared to be measuring memory rather than g, and predictive validity (of GPA) was eliminated. More evidence that re-testing effects are non-trivial. Pre-publication version here.

Last but definitely not least, one of my favorite topics--web-based recruitment. Allen, Mahto, & Otondo present results from 814 students searching real websites. When controlling for a student's image of the employer, job and organizational information correlated with their intention to pursue employment. When controlling for information search, a student's image of the employer was related to the intention to pursue employment, but familiarity with the employer was not. Finally, attitudes about recruitment source influenced attraction and partially mediated the effects of organizational information. What does all this mean? Don't throw your eggs into one basket--organizational image is important, but so is the specific information you have on your website about your organization and the specific job.

There's a lot of other good stuff in this volume, including articles on the financial impact of specific HRM practices, a meta-analysis of telecommuting impacts, engaging older workers, and daily mood.

Winter '07 Personnel Psychology

Things are starting to heat up in the journal Personnel Psychology. The shot across the bow of personality testing that happened in the last issue of Personnel Psychology turns into a full-blown brawl in this issue. But first, let's not forget another article worth out attention...

First up, Berry, Sackett, and Landers revisit the issue of the correlation between interview and cognitive ability scores. Previous meta-analyses have found this value to be somewhere between .30 and .40. Using an updated data set, excluding samples in which interviewers likely had access to ability scores, and more accurately calculating range restriction, the authors calculate a corrected r of .29 based on the entire applicant pool. This correlation is even smaller when interview structure is high, when the interview is behavioral description rather than situational or composite, and job complexity is high. Why is this important? Because it impacts what other tests you might want to use--the authors point out that using their updated numbers they obtained a multiple correlation of .66 for a high structure interview combined with a cognitive ability test (using Schmidt & Hunters' methods and numbers). Pretty darn impressive.


Now that we have that under our belt, ready for the main event? As I said, in last issue Morgeson et al. came out quite strongly against the use of self-report personality tests in selection contexts--primarily because they claim the uncorrected criterion-related validity coefficients are so small. So it's not surprising that this edition contains two articles by personality researcher heavyweights defending their turf...

First, Tett & Christiansen raise several points; more than I have space for here. Some points include: considering conditions under which personality tests are used and validity coefficients aggregated; that there are occupational differences to consider; that coefficients found so far aren't as high as they could be if we used more sophisticated approaches like personality-oriented job analysis; and that coefficients increase when multiple trait measures are used. This sums their points up nicely: "Overall mean validity ignores situational specificity and can seriously underestimate validity possible under theoretically and professional prescribed conditions."

Second, Ones, Dilchert, Viswesvaran, and Judge come out swinging and make several arguments, including: conclusions should be based on corrected coefficients; coefficients are on par with other frequently used predictors, some of which are much more costly to develop (e.g., assessment centers, biodata); different combinations of Big 5 factors are optimal depending upon the occupation; and compound personality variables should be considered (e.g., integrity). Suggestions include developing more other-ratings instruments and investigating non-linear effects (hallelujah), dark side traits, and interactions. They sum up: "Any selection decision that does not take the key personality characteristics of job applicants into account would be deficient."

Not to be out-pulpited (yes, you can use that phrase), Morgeson et al. come back with a response to the above two articles, reiterating how correct they were the first time around. They state that much of what the authors of the above articles wrote was "tangential, if not irrelevant", that with respect to the ideas for increasing coefficients, "the cumulative data on these 'improvements' is not great", and that corrected Rs presented by Ones et al. aren't impressive when compared to other predictors. They point out some flaws of personality tests (applicants can find them confusing and offensive) but fail to mention that ability tests aren't everyone's favorite test either. They claim that job performance is the primary criterion we should be interested in (which IMHO is a bit short-sighted), and that corrections of coefficients are controversial.

So where are we? Honestly I think these fine folks are talking past each other in some respects. Some issues (e.g., adverse impact) don't even come up, while other issues (e.g., faking) are given way too much attention. It's difficult to compare the arguments side by side because each article is organized differently. It doesn't help that the people on both sides are some of the researchers with the most invested (and most to lose) by arguing their particular side.

I'm thinking what's needed here is an outside perspective. Here's my two cents: this isn't an easy issue. Criteria are never "objective." Job performance is not a singular construct. Job complexity has a huge impact on the appropriate selection device(s). And organizations are, frankly, not using cognitive or ability tests nearly as much as they are conducting interviews. So let's stop focusing on which type of test is "better" than the others. Frankly, that's cognitive laziness.

So is this just sound and fury, signifying nothing? No, because people are interested in personality testing. Hiring supervisors are convinced that it takes more than raw ability to do a job. We shouldn't ignore the issue. Instead we should be focusing on providing sound advice for practitioners and treating other researchers with respect and attention.

Should you use personality tests? I'll answer that question with more questions: what does the job analysis say? What does your applicant pool look like? What are your resources like? It's not something you want to use cookie-cutter, but not something you should write off completely.

Okay, I'm off my soap box. Last but not least there are some good book reviews in this issue. One is Bob Hogan's book (which I enjoyed immensely and actually finished which is rare for me), Personality and the Fate of Organizations, which the reviewer recommends; another is Alternative Validation Strategies, which the reviewer highly recommends; and the third for us is Recruiting, Interviewing, Selecting, & Orienting New Employees by Diane Arthur, which the reviewer...well...sort of recommends--for broad HR practitioners.

That's all folks!

September 2007 issue of IJSA

The September, 2007 issue (vol. 15, #3) of the International Journal of Selection and Assessment is out, with the usual cornucopia of good reading for us, particularly if you're into rating formats and personality assessment. Let's skim the highlights...

First, Dave Bartram presents a study of forced choice v. rating scales in performance ratings. No, not as predictors--as the criterion of interest. Using a meta-analytic database he found that prediction of supervisor ratings of competencies increased 50% when using forced choice--from a correlation of .25 to .38. That's nothing to sneeze at. Round one for forced choice scales--but see Roch et al.'s study below...

Next up, Gamliel and Cahan take a look at group differences with cognitive ability measures v. performance measures (e.g., supervisory ratings). Using recent meta-analytic findings, the authors find group differences to be much higher on cognitive ability measures than on ratings of performance. The authors suggest this may be due to the test being more objective and standardized, which I'm not sure I buy (not that they asked me). Not super surprising findings here, but it does reinforce the idea that we need to pay attention to group differences for both the test we're using and how we're measuring job performance.

Third, Konig et al. set out to learn more about whether candidates can identify what they are being tested on. Using data from 95 participants who took both an assessment center and a structured interview, the authors found results consistent with previous research--namely, someone's ability to determine what they're being tested on contributes to their performance on the test. Moreover, it's not just someone's cognitive ability (which they controlled for). So what is going on? Perhaps it's job knowledge?

Roch et al. analyzed data from 601 participants and found that absolute performance rating scales were perceived as more fair than relative formats. Not only that, but fairness perceptions varied among each of the two types. In addition, rating format influenced ratings of procedural justice. The researchers focus on implications for performance appraisals, but we know how important procedural justice is for applicants too.

Okay, now on to the section on personality testing. First up, a study by Carless et al. of criterion-related validity of PDI's employment inventory (EI), a popular measure of reliability/conscientiousness. Participants included over 300 blue-collar workers in Australia. Results? A mixed bag. EI performance scores were "reasonable" predictors of some supervisory ratings but turnover scores were "weakly related" to turnover intentions and actual turnover. (Side note: I'm not sure, but I think the EI is now purchased through "getting bigger all the time" PreVisor. I'm a little fuzzy on that point. What I do know is you can get a great, if a few years old, review of it for $15 here).

Next, Byrne et al. present a study of the Emotional Competence Inventory (ECI), an instrument designed to measure emotional intelligence. Data from over 300 students from three universities showed no relationship between ECI scores and academic performance or general mental ability. ECI scores did have small but significant correlations (generally in the low .20s) with a variety of criteria. However, relationships with all but one of the criteria (coworkers' ratings of managerial skill) disappeared after controlling for age and personality (as measured by the NEO-FFI). On the plus side, the factor structure of the ECI appeared distinct from the personality measure. More details on the study here.

Last but not least, Viswesvaran, Deller, and Ones summarize some of the major issues presented in this special section on personality and offer some ideas for future research.

Whew!

IPMAAC Presentations + Cheaper Membership

I posted earlier about presentations from the 2007 IPMAAC conference going up online. Well now there's a whole gaggle of 'em and there's some really great content. Check out these sample titles (PDF):

Applicant Reactions to Online Assessments

Succession Planning and Talent Management

2007 Legal Update

Potholes on the Road to Internet Applicant Compliance

Measuring Complex Reasoning

Tips on Writing an Expert Witness Report

And that's just the beginning. For all the goodies, check out the full list.

But wait, there's more...

In addition, IPMAAC recently enacted a change to its membership categories & fees. You can now become an IPMAAC member for only $75! Talk about cheap. $75 pays for the difference in conference fees between a member and a non-member! And you get all this to boot. Plus, you're "affiliated" with IPMA-HR, which means you get the awesome weekly HR newsletter and discounts on all sorts of IPMA-HR stuff (that's a technical term). And you DON'T have to work in the public sector to join.

There really aren't that many professional organizations associated with assessment. There's SIOP, but they're about a lot more than just staffing. There are local groups. But when it comes to national or international groups, IPMAAC is it. Which is a good thing, because it's a great group of people (not that I'm biased or anything).

New issue of Journal of Applied Psychology (v.92, #4)

Guess how many articles are in the most recent Journal of Applied Psychology. Go ahead, take a gander.

10? 15?

Try 23. I mean....that's just showing off.

So what's in there about recruitment & assessment? Believe it or not, only two articles. Let's take a look at 'em.

First up, a study by Klehe and Anderson looked at typical versus maximum performance (the subject of the most recent issue of Human Performance) during an Internet search task. Data from 138 participants indicated that motivation to perform well (measured by direction, level, and persistence of effort) rose when people were trying to do their best (maximum performance). But the correlation between motivation and performance diminished under this condition, while the relationship between ability (measured by declarative knowledge and procedural skills) and performance increased.

What the heck does this mean? If you're trying to predict the MAXIMUM someone can do, you're better off using knowledge-based and procedure-based tests. If, on the other hand, you want to know how well they'll perform ON AVERAGE, check out tests that target things like personality, interests, etc.

Second, Lievens and Sackett investigated various aspects of situational judgment tests (SJTs). The authors were looking at factors that could increase reliability when you're creating alternate forms of the same SJT. Using a fairly large sample (3,361) in a "high-stakes context", they found that even small changes in the context of the question resulted in lower consistency between versions. On the other hand, being more stringent in developing alternate forms proved to be of value.

What the heck does this mean? If you're developing alternate forms of SJTs (say, because you give the test a lot and you don't want people seeing the same items over and over) this study suggests you don't get too creative in changing the situations you're asking about.

As usual, the very generous Dr. Lievens has made this article available here. Just make sure to follow fair use standards, folks.

New issue of Human Performance: Typical v. maximum performance

There's a new issue of Human Performance out (Volume 20, Issue 3) and it's devoted to a very worth topic--typical versus maximum performance.

What is the distinction, you say? Well it's pretty much what it sounds like. From Smith-Jentsch's article in this issue:

"Typical performance in its purest sense reflects what a person "will do" on the job over a sustained period of time, when they are unaware that their performance is being evaluated. By contrast, maximum performance reflects what a person "can do" when they are explicitly aware that they are being evaluated, accept instructions to maximize effort, and are required to perform for a short-enough period of time that their attention remains focused on the task."

The recent interest in this area stems largely from Sacket, Zedeck, & Fogli's 1988 article in Journal of Applied Psychology. Previous research suggested that measures of ability (e.g., cognitive ability tests) more accurately predict maximum performance whereas non-ability measures (e..g, personality tests) are correlated more with typical performance. This of course has implications for who we recruit and how we assess: Are we trying to predict what people can do or will do? The answer, I think, depends on the job--for aircraft pilot or police officer, you want to know what people can do when they're exerting maximum effort. For customer service representatives, you may be more interested in their day-to-day performance.

This topic is mentioned often in I/O textbooks but (as the authors point out) hasn't been researched nearly enough. The authors of this volume attempt to remedy that in some part. Let's look at the articles in the recruitment/assessment area:

First, Kimberly Smith-Jentsch opens with a study of transparency of instructions in a simulation. Analyzing data from two samples of undergraduates, the results validate previous findings: Making assessment dimensions transparent (i.e. telling candidates what they're being measured on) allows for better measurement of the abilities necessary for maximum performance, while not making this information transparent appears to result in better measurement of traits that motivate typical performance. So if the question is, "Do we tell people what we're measuring?" the answer is: It depends on what your goal is!

Next, Marcus, Goffin, Johnston, and Rothstein tackle the personality-cognitive ability test issue with a sample of candidates for managerial positions in a large Canadian forestry products organization. The results underline how important it is to recognize that "personality" (measured here by the 16PF and PRF) has several components. While cognitive ability scores (measured by the EAS) consistently outperformed personality scores in predicting maximum performance, measures of extraversion, conscientiousness, dominance, and rule-consciousness substantially outperformed cognitive ability when predicting typical performance.

Third, Ones and Viswevaran investigate whether integrity tests can predict maximal, in addition to typical, performance. The answer? Yes--at least with this sample of 110 applicants to skilled manufacturing jobs. Integrity test scores (measured using the Personnel Reaction Blank) correlated .27 with maximal performance (measured, as is typical, with a work sample test). The caveat here, IMHO, is that job knowledge scores correlated .36 with maximal performance. So yes, integrity test scores (a "non-ability" test) can predict maximal performance, but perhaps still not as well as cognitively-loaded tests.

Last but not least, Witt & Spitzmuller look at the relationship between cognitive ability and perceived organizational support (POS) on typical and maximum performance. Results from two samples (programmers and cash vault employees) reinforce the other results we've seen: Cognitive ability (measured by the Wonderlic Personnel Test), was correlated with maximum performance but not typical performance, while POS was related to two out of three measures of typical performance but not with maximum performance.

Overall, the results reported here support previous findings: maximum performance is predicted well by ability tests while typical performance has stronger correlations with non-ability tests. But Ones & Viswevaran are correct when they state (about their own study): "Future research is needed with larger samples, different jobs, and different organizations to test the generalizability of the findings." Let's hope these articles motivate others to follow their lead.

Police depts relax hiring standards

In response to serious recruiting challenges, many U.S. police departments are "lowering" their standards for hiring.

The reasons behind the shortage are many, including a strong job market, the Iraq war, and a high number of retirements.

Departments are using whatever means they have at their disposal, including upping their advertising. Case in point: while driving down 880 the other day in Oakland, CA, I noticed a sign promoting the $69,000 starting salary for Oakland Police Officers (and people wonder why it's hard to hire in the Bay Area).

The article cited above describes many steps departments are taking, some of which may initially seem like cause for concern. Let's take a look at them:

1. Forgiving minor criminal convictions, particularly old ones. If someone got busted 10 years ago for doing Ecstasy in college, and hasn't been in trouble since, is that still relevant?

2. Relaxing the 2-year college degree requirement, or allowing experience substitutions. I'm familiar with some research indicating a relationship between college education and officer performance, but if an officer has relevant experience (and performed well), this seems like a wash.

3. Raising the age limit. Age and job performance has been a hot topic in I/O psychology for a long time. While there are some declines over age (e.g., working memory), my reading is that they aren't practically significant in most situations. And we're talking about raising the limit to 40 or 44, not 85.

4. Relaxing fitness requirements. To me this comes back to plain 'ol validation. Granted, it's not always easy to determine where a pass point should be set (do they have to run 300 meters in 55 seconds or 56 seconds?), but do the study. Find out where a reasonable point would be. Run the numbers. See if it makes sense.

A lot of the concerns that go along with these changes--hiring people with low integrity, hiring people physically or mentally unable to perform the job--can be mitigated with good assessment, such as memory tests, physical ability testing, integrity testing, and reference and background checks.

Overall, I think this is a good thing--minimum qualifications (MQ) are often barriers to employment for certain ethnicities, women, and individuals with disabilities. And the situation is even worse when they aren't based on any rigorous study of the necessity for the MQs to being with.

On the other hand, I have heard anecdotally that similar changes in standards for U.S. Army recruits has resulted in more challenges for training.

What do you think--big deal or not?

Red--it's not just for bulls anymore

Don't like red pen? Turns out you're not the only one, according to a new study by Andrew Elliott and colleagues titled, "Color and Psychological Functioning: The Effect of Red on Performance Attainment" in the February edition of the Journal of Experimental Psychology: General.

Using several different experiments, the researchers found that even a brief glimpse of the color red can lower scores on achievement tasks. For example, one of the experiments involving nearly 300 U.S. and German high school and undergraduate students found that simply looking at a red participant number (versus black or green) prior to completing an IQ test resulted in a performance decrease.

The authors hypothesize that the color red evokes an anxiety response which in turn interferes with the ability to complete the task. Where does the anxiety come from? Some possibilities, according to the authors, include:

- Evolution: we may be hardwired to respond to red (think of the association between red and aggression in nature)

- Daily life: red is often associated with warnings or commands (e.g., stop lights, stop signs, dash lights)

- School: who didn't cringe a little when they saw red marks on their essays or tests in school? Maybe you even have a supervisor who does this?

Lesson: be careful with using red in testing material. There's enough error out there being introduced in testing situations without worrying about color.

So...did this article make you nervous?

Hat tip: SIOP.org

2007 SIOP Conference: Highlights, Part 2

This is the third in a series of posts about the 2007 SIOP Conference. In Part 1 I talked about some of the new products out there and in Part 2 I went over some of the research that was presented. In this post I'll point out some more research that you may find interesting...

Legal risks and defensibility factors for employee selection procedures

Posthuma, Roehling, and Campion analyzed nearly 600 federal district court cases and came up with some very interesting results:

- Employers are most likely to win (by far) when defending tests of math or mechanical ability. Employers also fare well when defending assessments of employment history and interviews.

- Employers did worst when defending physical ability tests and medical examinations. Tests of verbal ability and job knowledge were also more likely to result in a plaintiff win.

Predicting Internet job search behavior and turnover

Using a sample of 110 nurses in Texas, Posthuma et al. found using longitudinal survey data that (among other things) Internet job search behavior was related to turnover--folks weren't just surfing for fun. This suggests that organizations need to pay close attention to job searching behavior among employees; not necessarily to curtail it but instead to figure out why high performers want to leave.

Gender differences in career choice influences

After analyzing survey data from nearly 1,400 fourth-year medical students from two U.S. schools, Behrend et al. found a gender difference in preferred career: specifically, female medical students valued "opportunities to provide comprehensive care" when choosing a specialty much more than men. This is consistent with other work that has showed women to be more "relationship-oriented" than men when it comes to choosing a career.

Portraying an organization's culture through properties of a recruitment website

In this study of 278 undergraduate students, Kroustalis and Meade found that inclusion of pictures on a website that were intended to portray a certain organizational culture did so--but only for certain cultural characteristics. Specifically, pictures that implied a culture of either innovation or diversity had the intended effect--but pictures representing a team orientation did not. Interestingly, "employee testimonials" designed to emphasize these cultural aspects failed to do so for any of the three aspects studied. Finally, individuals who perceived a greater fit between themselves and the organization (in terms of the three cultural aspects) reported being more attracted to the organization.

Recruiting solutions for adverse impact: Race differences in organizational attraction

Last but definitely not least, Lyon and Newman gathered data from nearly 600 university students on their reactions to 40 hypothetical job postings...and came away with some very interesting results. For example:

- Conscientious individuals were more likely to apply to postings that explicitly stated a preference for conscientious applicants.

- Conscientious individuals were more likely to apply to postings that described the company as results-oriented.

- Black applicants with higher cognitive ability were more likely to respond to ads seeking conscientious individuals while White applicants with higher cognitive ability were less likely to do so.

- When a company was described as innovative, Black applicants high on conscientiousness were more likely to apply; this was not the case for White applicants.

What does your web page say about you?

Many people use MySpace, Facebook, and other social networking sites to create personalized web pages that often include information we might call part of a "digital first impression"--music they like, books they read, TV shows they watch, etc.--the types of things that come up in conversation that we use to judge someone's similarity to us and what type of person they are.

From a recruiting and assessment perspective, the most obvious use of this information is to identify potential candidates and gather additional information about applicants to judge their qualifications and fit. This is already happening and these "digital selves" may increase in importance with the introduction of new web services.

Aside from legal concerns, an important issue is the accuracy of these judgments. Does the fact that someone has a link to classical music on their page mean anything? How about whether or not they provide contact information?

Much of the work done on how physical and digital displays relate to personality has been done by Dr. Sam Gosling at the University of Texas and Dr. Peter Rentfrow at the University of Cambridge. In addition, several studies have been done in both applied and university settings. Most of these studies have used the Big 5 as a way to describe adult personality.

Overall, research suggests people are fairly accurate in making personality judgments about people based on their web pages--this is particularly true when judging openness to experience and extraversion. Specifically, here's what different aspects of a web page have been linked to:

- Including information about one's hometown/region: Higher on agreeableness and conscientiousness, lower on neuroticism

- Expressing a lot of personal beliefs/emotions: Higher on neuroticism and openness to experience, lower on agreeableness and conscientiousness

- Having a blog: Higher on extraversion and openness to experience

- Seeking explicit feedback (e.g., comments): Lower on neuroticism

- Having links to Internet/computer sites: Lower on extraversion

- Having links to visual arts, having a music lyrics category (strongest relationship): Higher on openness to experience

- Having a webcam: Lower on agreeableness

- Linking to one's resume/vitae, posting family pictures, or having a visitor counter: Higher on conscientiousness


What about specific aspects of a person, such as personal tastes? Here's what research has uncovered about specific preferences:

Music

Much of the work done on how personal tastes reflect personality has focused on musical preferences. Here's what various musical tastes tend to correlate with (you can take this assessment here):

- Reflective and complex (e.g., jazz, classical): Higher on openness to experience (strongest overall relationship), higher verbal ability

- Intense and rebellious (e.g., rock, alternative): Higher on openness to experience, higher verbal ability

- Upbeat and conventional (e.g., country, pop): Higher on extraversion, agreeableness, and conscientiousness; lower verbal ability

- Energetic and rhythmic (e.g., hip-hop, electronica): Higher on extraversion


Here's a sample of how people that liked certain books, movies, and social activities described themselves:

Books

- Fiction & Literature: Creative
- Business: Attractive, successful
- Science Fiction: Intelligent, weird
- Fantasy: Weird
- Cooking: Lovable

Movies

- Adventure: Fun, lovable
- Comedy: Funny
- Independent: Creative, unique
- Science Fiction: Kind, weird

Social activities

- Clubbing: Attractive, fun, socially adaptable
- Bowling: Funny
- Computer Gaming: Intelligent, weird

==

The findings are intriguing, but we desperately need more research in this area, particularly looking at aspects of the observer that influence accuracy and research into additional forms of information linkages, such as tags. Hopefully over time we'll uncover more about how well (or poorly) the online self matches up with who you meet in person.

Sources:
"e-Perceptions: Personality impressions based on personal websites"
"The do re mi's of everyday life: The structure and personality correlates of music preferences"
"Message in a ballad: The role of music preferences in interpersonal perception"
"Personality impressions based on Facebook profiles"
"Personality in cyberspace: Personal websites as media for personality expressions and impressions"
"A social network caught in the web"

2007 SIOP Conference Part 2: Highlights Vol. 1

In a previous post I talked about some of the products showcased in the program for the 2007 SIOP Conference.

In this post I'd like to highlight some of the more interesting (to me--and to you too, I hope!) presentations being made. Because there are so many interesting presentations, I'll use several posts to cover a number of them.

Gender and letters of recommendation: Agentic and communal differences (249-8)

Here's a study that should make you think twice about those letters of recommendation you review (if you don't already). After analyzing nearly 700 letters of recommendation for psychology faculty positions, the researcher found that women in these letters tended to be described as more affectionate, warm, and kind, while men were described as more ambitious, dominant, and self-confident. In addition, letters for women contained more references to their physical appearance (insert shudder here).

Data trends in open mode, online, unsupervised cognitive ability testing (61-28)
Personality testing online (unsupervised) and paper and pencil (supervised) (103-21)
Retest effects on an unproctored Internet-based GMA test (205-29)
DFIT analysis of web-based and paper-based versions of the WPT (261-21)

These four studies all looked at online testing in some way or another. The first three provide some support for online testing; they suggest that online general mental ability (GMA) test scores are relatively stable over time and the psychometric qualities of a personality test were consistent regardless of whether the test was taken online & unproctored or in person & proctored. Before we get too excited, however, the last study found that the paper-and-pencil and online versions of the WPT were not completely identical. It also found that WPT-Q scores differed between proctored and unproctored settings. So overall, mixed support for online testing. Chances are other factors (e.g., physical environment, Internet self-efficacy) play major roles.

Fancy job titles: Effects on recruitment success (261-25)

Chief Fun Officer. Brand Evangelist. Some organizations are coming up with creative job titles in an attempt to lure candidates who may find "Marketing Executive" a tad dull. But does it work? In this study, Dr. Klaus Templer presented nearly 400 marketing students with four hypothetical job ads using various titles, including fancy (e.g., Global Brand Insights Manager) as well as traditional (e.g., Marketing Officer). Results? Attitude toward the job was significantly higher with the fancier title, as was the extent to which the job was recommended to a friend. Why? Templer hypothesizes that fancier job titles lend the job more prestige, making it more attractive. Interesting follow-up question I have: Does the response vary between high-potentials and low-potentials? Also, we should keep in mind that surveys suggest job titles may have less of an impact on retention.

More conference goodness in upcoming posts!

Journal of Applied Psychology, v.92, #2

I wrote an earlier post about a particularly good article in the most recent issue of the Journal of Applied Psychology. But what about the other articles? Let's take a look because there is plenty here whether you're interested in cognitive ability tests, personality tests, or meta-analysis.

First, a study for all you meta-analysts by Schmidt and Raju about the best way to combine new research with existing meta-analysis results. Results indicate that the traditional "medical model" of adding new studies to the database and re-calculating worked well, as did an alternative Bayesian model the authors describe.

Next up, a look at what happens to people's scores on cognitive ability tests when they've taken the test before (known as a "practice effect"), by Hausknecht and colleagues. Meta-analyzing the results of 50 studies yielded an adjusted overall effect size of .26. The effects were larger when individuals received coaching and when they re-took the identical test.

Third, a study by Ellingson, Sackett, and Connelly of response distortion on personality tests. Specifically, the authors looked at data from 713 people who had taken the California Psychological Inventory (CPI) twice--once in a selection context (presumably high motivation to "cheat") and once in a development context (presumably low motivation to "cheat). Results? Limited amount of response distortion going on. Good news for personality tests, although certainly not the last word on this topic.

Fourth, a look at when birds of a feather flock together--and when they don't. Specifically, Umphress and colleagues looked at whether demographic similarity attracted prospective employees or not. What they found was that it depended on people's preference for "group based social hierarchies" (i.e., were high on social dominance orientation). If they were, those in "high status groups" were attracted to demographic similarity in an organization while those in "low status groups" were repelled by it. Bottom line? Trying to attract applicants by pointing out similarities with current incumbents may or may not be a good idea...

Next, for you stats folk, a look by Sackett, Lievens, Berry, and Landers at the effect of range restriction on correlations between predictors (e.g., between a personality test and a cognitive ability test). Conclusion? That these correlations can be quite distorted when the predictors are used as a composite in an actual selection setting. Why do we care? Because it may mess up our conclusions about things like incremental validity of one test over another. (A draft of the article goes into more detail)

Last but not least, Kuncel and Klieger look at the very important issue of how knowledge of test score results impacts behavior. Their research revealed a 23% score difference between all individuals who had taken the LSAT and those that took the LSAT and applied to law school. This has implications for range restriction corrections (there's that term again).

There you have it! Quite the smorgasborg this issue.

Human Performance, Vol. 20, #1: The EQ-i:S

March journal madness continues with the latest issue of the journal Human Performance.

There's really only one article in here related to selection, but it's an interesting one, so let's take a look.

In this study of 229 students from an southeastern U.S. university, Grubb and McDaniel looked at the constructs measured and fakability of the Emotional Quotient Inventory Short Form (EQ-i:S), a popular proposed measure of emotional intelligence.

The bottom line results:

- Participants were able to "fake" their scores, raising them substantially (by .83 standard deviations). Not particularly surprising as it's fairly well established that non-cognitive measures can be "faked." (What's not clear is whether it matters...)

- The two "screens" built into the EQ-i:S to try to identify fakers correctly identified only 31% of the fakers.

- EQ-i:S scores were predicted by the Big Five measure with a multiple correlation of .79. This result, say the authors, "casts doubt on the construct of emotional intelligence as operationalized in the EQ-i:S."

But there's some additional goodness in this article, largely because the authors also had the participants take the Wonderlic Personnel Test (WPT) and a measure of the Big Five personality factors (IPIP):

- WPT scores were correlated strongly with only one Big Five factor--Openness.

- The correlation between WPT scores and gender was small.

- Black-White score differences on the Big Five factors were small.

- Gender was correlated with Big Five scales, but the nature varied depending on the condition (honest or faking).

- The ability to fake on the EQ-i:S was a function of cognitive ability and personality (mostly agreeableness).

There were other articles, one of which I'll discuss over at HR Coal.

New issue of IJSA--vol. 15, #1