Unproctored internet testing: Safe for some tests?

One of the biggest trends in personnel assessment is the movement toward on-line testing. Many organizations are experimenting with so called unproctored Internet testing (UIT), where candidates are allowed to take the exams whenever, and wherever, they want.

Benefits? Extremely convenient for the candidate. Less administrative resources needed by the employer.

Costs? Bye-bye exam security, hello cheating opportunities. Not only is your test out for everyone to see, but you have no real way of knowing (sans biometric verification) who is taking the test.

Some organizations have decided the benefits outweigh the risks, and a new study in the June 2008 issue of the International Journal of Selection and Assessment may provide support for their position.

In it, the authors looked at over 800 applicants from nine European countries that took a test of perceptual speed in an unproctored setting, then followed this up with a proctored parallel version. Results? Not only was there no evidence of cheating, they found the opposite effect--people did better in the proctored setting.

Now before everyone throws out their proctored exams, note that this is a type of test that might be hard to cheat on--at least in one way. Because this is a perceptual speed test, there are no "right" answers that can be looked up. It also required very quick responses. So the only way to cheat would be to have someone take the test for you. Implication: it may make more sense to use certain UITs than others.

This topic is a source of much debate in the assessment community, and there is by no means consensus on the right way to go. But studies like this help!

Take a deep breath, because there's a lot more in this issue:

- The preliminary employment interview as a predictor of assessment center outcomes (fascinating look at how the AC may only make sense for mid-range interview scorers)

- A comparison of the common-item and random-groups equating designs using empirical data (for you IRT fans out there)

- The influence of external recruitment practices on job search practices across domestic labor markets: A comparison of the United States and China

- Beneath the surface: Uncovering the relationship between extraversion and organizational citizenship behavior through a facet approach (a more nuanced look at the relationship shows extraversion can predict OCBs)

- Comparing personality test formats and warnings: Effects on criterion-related validity and test-taker reactions (another good one...personality test added predictive validity beyond ability test but no validity difference between forced-choice and Likert scales, nor between warning and no-warning conditions; forced-choice and warnings may produce negative candidate reactions)

- Applicant selection expectations: Validating a multidimensional measure in the military (describes development of a new measure of applicant perception of the selection process)

- Selecting for creativity and innovation: The relationship between the innovation potential indicator and the team selection inventory

Adverse impact of assessment centers (May Applied Psych)

The May '08 issue of the Journal of Applied Psychology is out with lots of great content. Unfortunately only one is directly related to recruitment and assessment, so let's check that one out then I'll give you links to some others that look interesting.

The study is Ethnic and gender subgroup differences in assessment center ratings: A meta-analysis by Dean, Roth, and Bobko. The authors found overall d-values of .52 for Black-White differences, .28 for Hispanic-White differences, and -.19 for male-female differences. (the second group in these comparisons performs better)

The results suggest that the Black-White difference is larger than previously thought but may be a more "diversity friendly" option for Hispanics and females.

There are some other great articles in here for fans of organizational behavior, including:

Subjective cognitive effort: A model of states, traits, and time. (which, by the way, suggests another reason why conscientiousness may predict job performance)

Early predictors of job burnout and engagement.

Event justice perceptions and employees' reactions: Perceptions of social entity justice as a moderator.

Harmful help: The costs of backing-up behavior in teams.

Trust that binds: The impact of collective felt trust on organizational performance.

Stirring the hearts of followers: Charismatic leadership as the transferal of affect.

The influence of psychological flexibility on work redesign: Mediated moderation of a work reorganization intervention.

...and several more!

First issue of new SIOP journal

SIOP (the Society for Industrial and Organizational Psychology) has a new journal. It's called "Industrial and Organizational Psychology: Perspectives on Science and Practice."

The idea of the journal is to offer a couple of focal articles and then print several peer commentaries associated with each.

In the inaugural (March 2008) issue, two topics are covered.

In the first focal article, Macey and Schneider treat us to a historical and research-based view on a very hot topic, employee engagement. Here's a sample:

"The notion of employee engagement is a relatively new one, one that has been heavily marketed by human resource (HR) consulting firms that offer advice on how it can be created and leveraged. Academic researchers are now slowly joining the fray, and both parties are saddled with competing and inconsistent interpretations of the meaning of the construct."

The authors provide a great overview of the different ways of viewing engagement. The treatment is generally positive, and they sum up their view this way:

"Although engagement may best fit...as a profile model of a multidimensional construct, we see engagement as not only a set of constructs but also a tightly integrated set, interrelated in known ways, comprising clearly identifiable constructs with relationships to a common outcome."

The focal article is followed by no less than 13 commentaries from a variety of authors, both academics and consultants. The authors follow with a reply and point out that the debate over engagement is a great example of the "research-practice gap."

The second topic is assessment centers. In it, Charles Lance investigates why they don't work the way they're supposed to. Specifically, candidate ratings seem to reflect the particular exercise they're completing--not the dimensions they're supposed to be rated on. The conclusion sums up their view nicely:

"It is now time to acknowledge the last quarter century's worth of research findings and reorient assessment away from broad dimensions and toward exercise-based assessment."

The focal article is followed by ten peer commentaries by folks such as Ann Howard, Winfred Arthur, and Filip Lievens, and the authors' response.

Great, in-depth stuff for those of you out there interested in either topic.

2008 PTC-NC Conference, Day 2

In my previous post I talked about the first day of the 2008 PTC-NC Conference. Today I'll give you a rundown of the second day.

The day started off with a bang due to a keynote address by noted personality researcher Robert Hogan. Dr. Hogan never disappoints, and his presentation was a mini-history of personality testing combined with executive assessment and organizational performance--all in 75 minutes. Major themes included the importance of leadership personality, how reputation is much more important (from a measurement perspective) than identity, why leadership effectiveness should be defined by team performance, and the characteristics of great leaders (e.g., integrity, decisiveness, competence, vision). You can get some of the same flavor in a recent American Psychologist article he co-authored.

The first breakout featured Jim Higgins (Cal DOJ) discussing an internet-based applicant self-assessment career tool and Greg Beatty (IRS) describing how the IRS has modernized using a competency-based approach. I attended Greg's (since I work within throwing distance of Jim), and he provided a great overview of some of the innovative things the IRS is doing both for incumbents and applicants, including job simulations, online assessment tools, and a career management center. "IRS...really?" you say? Yup. Don't believe me? Check out the simplicity and ease of use of their career site.

The last sessions of the conference included one on how to use MS Excel to automate selection by Dan Kuang (Biddle) and another on the leadership developmental assessment center (LDAC) by Matt Gruver (CPS). I attended Matt's, where he provided a great overview of how to develop an LDAC, including the importance of competency definition, how top management involvement is key, and how participants often are very appreciative of the (unusually rare) honest feedback they receive. Can't wait to put one together!

Overall, a great program and kudos to the organizers! Looking forward to next year.

December '07 IJSA, Part 2

In a previous post I described one of the most interesting articles in the December '07 issue of IJSA (the International Journal of Selection and Assessment (IJSA)). In this post I'll go over the rest of the issue.

First up, Carless & Imber with a study of interviewer characteristics. Using a sample of 450 graduate students, they found that personal aspects of the interviewer, such as warmth, friendliness, and humor, as well as job-related aspects such as job knowledge and general competence, had effects on how attracted the person was to the job as well as their likely job choice intentions. Not only that, aspects of the interviewer impacted how anxious the interviewee was. Lesson: Much like recruiters, make sure people doing the interviewing are trained and the type of people you'd like representing your organization!

Next, LaHuis, MacLane, and Schlessman with a study of reapplication among 542 applicants to a U.S. government position. Focusing on the 9% that didn't get the job but reapplied the following year, the authors found that "opportunity to perform" played a significant role in reapplication behavior. Lesson: If you want people to reapply, give them the chance to show their strengths.

Third, Carless and Wintle with a study of what attracts applicants to particular job opportunities--specifically looking at flexible career paths and who's doing the recruiting. Participants were 201 "young, inexperienced job seekers" who completed a questionnaire. Results? Flexible career paths were a big attraction (compared to traditional career paths) but recruiter background (HR or external agency) made no difference, in line with previous research that's found that recruiter personality is the key rather than things like background or demographics.

Next, Hermelin, Lievens, and Robertson conducted a meta-analysis of assessment center scores. Based on 26 studies (N=5,850) the authors found a corrected correlation of .28 between AC scores and subsequent supervisory ratings, which they hypothesize is lower than the true value due to range restriction of assessment center scores (and lower than the corrected value of .37 that Schmidt & Hunter reported). Alternate version (if the link gets fixed) here.

Heinsman, et al. provide the next study, which looks at how psychologists view the makeup and measure of competencies, still a hot topic in HR circles. Using data from over 900 applicants who participated in a 1-day selection process, they examined the relationship between competencies including Thinking, Feeling, and Power with traditional measures of cognitive ability, personality, and assessment center performance. Results? To assess Thinking, psychologists in this study relied upon measures of cognitive ability (makes sense!). To assess Feeling, they used interview simulation scores as well as personality tests scores. Finally, when analyzing Power, they relied mostly on personality scores.

Last but not least, Lievens and Anseel with a fascinating study of creating alternate versions of computerized in-baskets. The alternate version used a very similar situation that people had to respond to, but altered the context for 10 of the 20 items. Results? No significant difference in overall in-basket score between the two forms. So if you're looking to duplicate your in-baskets, check this out! Oh, and there's an alternate version of the article here.

Okay, I might as well tell you about the other two articles, because you might be interested. One is on self and peer ratings and one is on survey non-respondents (hint: it's not the star performers that aren't responding). Both have interesting results, so check 'em out!

September 2007 issue of IJSA

The September, 2007 issue (vol. 15, #3) of the International Journal of Selection and Assessment is out, with the usual cornucopia of good reading for us, particularly if you're into rating formats and personality assessment. Let's skim the highlights...

First, Dave Bartram presents a study of forced choice v. rating scales in performance ratings. No, not as predictors--as the criterion of interest. Using a meta-analytic database he found that prediction of supervisor ratings of competencies increased 50% when using forced choice--from a correlation of .25 to .38. That's nothing to sneeze at. Round one for forced choice scales--but see Roch et al.'s study below...

Next up, Gamliel and Cahan take a look at group differences with cognitive ability measures v. performance measures (e.g., supervisory ratings). Using recent meta-analytic findings, the authors find group differences to be much higher on cognitive ability measures than on ratings of performance. The authors suggest this may be due to the test being more objective and standardized, which I'm not sure I buy (not that they asked me). Not super surprising findings here, but it does reinforce the idea that we need to pay attention to group differences for both the test we're using and how we're measuring job performance.

Third, Konig et al. set out to learn more about whether candidates can identify what they are being tested on. Using data from 95 participants who took both an assessment center and a structured interview, the authors found results consistent with previous research--namely, someone's ability to determine what they're being tested on contributes to their performance on the test. Moreover, it's not just someone's cognitive ability (which they controlled for). So what is going on? Perhaps it's job knowledge?

Roch et al. analyzed data from 601 participants and found that absolute performance rating scales were perceived as more fair than relative formats. Not only that, but fairness perceptions varied among each of the two types. In addition, rating format influenced ratings of procedural justice. The researchers focus on implications for performance appraisals, but we know how important procedural justice is for applicants too.

Okay, now on to the section on personality testing. First up, a study by Carless et al. of criterion-related validity of PDI's employment inventory (EI), a popular measure of reliability/conscientiousness. Participants included over 300 blue-collar workers in Australia. Results? A mixed bag. EI performance scores were "reasonable" predictors of some supervisory ratings but turnover scores were "weakly related" to turnover intentions and actual turnover. (Side note: I'm not sure, but I think the EI is now purchased through "getting bigger all the time" PreVisor. I'm a little fuzzy on that point. What I do know is you can get a great, if a few years old, review of it for $15 here).

Next, Byrne et al. present a study of the Emotional Competence Inventory (ECI), an instrument designed to measure emotional intelligence. Data from over 300 students from three universities showed no relationship between ECI scores and academic performance or general mental ability. ECI scores did have small but significant correlations (generally in the low .20s) with a variety of criteria. However, relationships with all but one of the criteria (coworkers' ratings of managerial skill) disappeared after controlling for age and personality (as measured by the NEO-FFI). On the plus side, the factor structure of the ECI appeared distinct from the personality measure. More details on the study here.

Last but not least, Viswesvaran, Deller, and Ones summarize some of the major issues presented in this special section on personality and offer some ideas for future research.

Whew!

Spring '07 issue of Public Personnel Management

The Spring 2007 issue of Public Personnel Management is here (IPMA-HR membership required for full access), and it's got some great stuff inside. Let's take a look at the two articles specifically focused on recruitment/assessment...

First, "The validity of assessment center ratings and 16PF personality trait scores in police Sergeant promotions: A case of incremental validity" by Love & DeArmond. As the title suggests, what the authors looked at here was the ability of job-related personality test scores (measured by the 16PF) to add incremental validity above and beyond assessment center (AC) scores in predicting performance as a police Sergeant. (Not familiar with ACs? Check out this great intro by Bill Waldron and Rich Joines). Anyhoo, here's the rundown:

Sample: 54 candidates, 48 male, all Caucasian, from small and medium-size agencies
AC: Five work sample exercises
Previous hurdle: All had passed a written job knowledge exam
Criterion: Supervisor ratings
Results: AC performance dimensions significantly predicted performance ratings (R-square of .16, p<.01) and 16PF scores accounted for additional unique variance (change in R-square of .08, p<.05). When entered first in the regression, however, personality did NOT significantly predict performance ratings. Hmmmm....
Concerns I have: The reliabilities for the five 16PF factors investigated were not good, even considering the small sample (alphas ranged from .04 to .55). The sample was small, and not particularly diverse (a fact the authors acknowledge as a limitation). And, um...I'm not particularly impressed with that incremental validity, although it's pretty par for the course.

Second, in "Legal issues for HR professionals: Reference checking/background investigations" William Woska provides a great overview of why conducting reference and background checks is so important, what an employer's obligations are, and the importance of focusing on job-related factors. Aside from simply being plain 'ol best practice, avoiding a negligent hiring (tort) lawsuit is a great reason to always do reference checks. Woska also covers avoiding violating an applicant's privacy and/or the Fair Credit Reporting Act, and the nature of waivers. This is one of those "save and put in your files" articles.

By the way, there's some other great stuff in here, including legal analyses of constructive discharge and affirmative action, a look at turnover in jails, and an essay on (among other things) Civil Service reform in Florida.

New Personnel Psychology--vol. 60, no. 1

March brings the Spring issue of Personnel Psychology and some great material to sink our teeth into.

Let's take a look at the articles relevant for recruitment and assessment (I'll summarize some of the other articles on my general HR blog).

The articles

First up, a study of --wait for it--personality tests. Well, integrity tests really. Using a sample of employees and students from Canada and Germany (thankfully moving beyond the U.S.), Marcus, Lee, and Ashton found that for the overt integrity tests investigated, honesty-humility (as described in the HEXACO model) accounts for more of the criterion-related validity than did the Big 5 dimensions of personality. For the personality-based integrity tests, the results were the opposite. The results were the same regardless of study group or instrument. Specific findings included correlations of -.51 and -.67 between an shortened version of the CPI and self-reported counterproductive work behavior and academic behavior, respectively.

Next, an excellent meta-analysis of situational judgment tests (SJTs) by McDaniel, Hartman, Whetzel, and Grubb. The authors continue the exploration of the difference between SJTs with knowledge instructions (e.g., "What would be the best...") and those with behavioral tendency instructions (e.g., "What would you do..."). The analysis found (as have previous studies) that SJTs with knowledge instructions had higher correlations with cognitive ability, while those with behavioral tendency instructions correlated more highly with personality constructs (Big 5). Criterion-related validity was unaffected by response instruction and was reported at .26 for both types--lower than had been previously reported due to analysis of new data. Results also showed SJTs have (modest) incremental validity over cognitive ability, Big 5, and a composite of the two. Given that previous studies have found larger racial differences with knowledge-based instructions I'm tempted to think this is an overall win for behavioral tendency instructions, but the situation is more complicated than that and most likely depends on the the job (e.g., complexity).

Last but not least, an article on test validity, bias, selection errors, and adverse impact by Aguinis and Smith (full text available here, many other of Aguinis' publications available here). The authors present a framework that integrates the four concepts and describe a computer program (available here) that allows individuals to input test and criterion distributions and analyze the impact on and relationship with selection ratios, adverse impact, and performance levels. Useful but definitely not a casual read.

The books

In addition to the studies listed above (and several others), this issue reviews several books that are worth looking into:

1) A critique of emotional intelligence: What are the problems and how can they be fixed? edited by Kevin Murphy. Intended to be a balanced analysis of this popular concept, the reviewer sees the book as more one-sided (providing fuel for critics). The book features chapters by prominent researchers and practitioners including Landy, Van Rooy, Hogan, and Spector.

2) Assessment centers in human resource management: Strategies for prediction, diagnosis, and development by George Thornton and Deborah Rupp. This one looked good enough that I ordered it--especially since I'm lacking a good book devoted to assessment centers. The reviewer states, "we have a new comprehensive description and expert review of the literature on the practice and evaluation of the assessment center method." I have Thornton's book on simulations which I find very useful.

3) Essentials of personnel assessment and selection by Robert Guion and Scott Highhouse. If I didn't already have Guion's larger tome, I'd probably get this one. The authors state the book is intended for undergraduate and master's level students, and the reviewer says "I believe that the book is just right for first or second year graduate students. It is a handy and compact compendium of fact and best practice."

That's it! As I said, there are other articles in here that I'll summarize in my other blog.