Where there's a will, there's a way: OPM shows how to do UIT the right way

On August 10 and 12, PTC-NC was privileged to have Dr. Patrick Sharpe from the U.S. Office of Personnel Management (OPM) deliver a presentation about USA Hire, part of a suite of online platforms that has allowed OPM to revolutionize the way they deliver assessment services to their diverse customers.

The idea to ask Dr. Sharpe to present occurred to me when I read an article in the Washington Post on April 2 about USA Hire titled, "For federal-worker hopefuls, the civil service exam is making a comeback".   It provides an overview of what OPM has managed to accomplish with its partner, PDRI, in the area of unproctored internet testing (UIT).  Start there if you want to learn more because it includes some examples of the items--although significantly absent is an example of the excellent video avatar-based assessment used for things like situational judgment.

Dr. Sharpe did an excellent job painting the picture for the audience of how much work was involved in the project, and how important things like stakeholder communication and contract management were to ensuring the project was a success.  He then showed us a demo version of USA Hire, where he lead us through what it looks like from the applicant's perspective as they proceed through a series of competency-based assessments.  The item formats range from the traditional (e.g., reading comprehension multiple-choice) to the modern (avatar-based SJT) to the groundbreaking--at least for the public sector (forced-choice non-cognitive assessment).

Here are some of the key points I took away:

- The technology is just a part of successfully putting an UIT program together, you have to step back and look first at what you're trying to accomplish.  For example, are you interested in whole person assessment (as OPM is) or simply focusing on certain KSAs?

- USA Hire is the culmination of years of research and analysis, and traces its history back 20-30 years within the federal government.   Translation : don't jump into UIT without careful planning.

- Start with the basics when delivering UIT: make sure the customer has a solid job analysis foundation before jumping to the assessment platform

- Getting a larger, more influential, customer successfully implemented can cause others to jump on board

- Realize that, particularly in a decentralized testing environment, you may still end up with a hybrid of different testing approaches following the roll-out of UIT, and this includes T&Es.  But the best way to move the practice is to show what success looks like.

- Consider carefully whether you want to build, buy, or lease the technology.  There are benefits and drawbacks to each.

- Starting with a pilot can be a great way to test the system (no pun intended), and also demonstrate the potential to stakeholders.

- Collaboration between assessment professionals, HR specialists, and vendors is critical.

- Don't underestimate the importance of change management.  Fears (e.g., about losing control) come easily and have to be addressed head-on.

- Organizational and system readiness is very important.  Part of the reason this effort was successful is because hiring organizations were fed up with the extremely low utility (and perception) of point-based T&Es.

For someone passionate about assessment and technology, the presentation was educational and motivational.  I walked away, as did others, with a new-found optimism for what sufficient will, resources, and tenacity can accomplish.  It's seductive to focus on what can't be done in the public sector, so to hear and see what can be done reemphasizes the importance of leadership--both in HR and at the top of the organization.

Unproctored internet testing has been talked about for so long, but to see it in action, in a research-based way in the public sector, is truly inspirational.  Truly where there is a will, there is a way.

Mobile assessment comes of age + research update

The idea of administering employment tests on mobile devices is not new.  But serious research into it is in its infancy.  This is to be expected for at least two reasons: (1) historically it has taken a while with new technologies to have enough data to analyze (although this is changing), and (2) it takes a while for researchers to get through the arcaneness of publishing (this, to my knowledge, isn't changing, but please prove me wrong).

Readers interested in the topic have benefited from articles elsewhere, but we're finally at a point where good research is being published on this topic.  Case in point: the June issue of the International Journal of Selection and Assessment.

The first article on this topic in this issue, by Arthur, Doverspike, Munoz, Taylor, & Carr, studied data from over 3.5 million applicants who completed unproctored internet-based tests (UIT) over a 14-month period.  And while the percentage that completed them on mobile devices was small (2%), it still yielded data on nearly 70,000 applicants.

Results?  Some in line with research you may have seen before, but some may surprise you:

- Mobile devices were (slightly) more likely to be used by women, African-Americans and Hispanics, and younger applicants.  (Think about that for a minute!)

- Scores on a personality inventory were similar across platforms.

- Scores on a cognitive ability test were lower for those using mobile devices.  Without access to the entire article, I can only speculate on proffered reasons, but it's interesting to think about whether this is a reflection of the applicants or the platform.

- Tests of measurement invariance found equivalence across platforms (which basically means the same thing(s) appeared to be measured).

So overall, in terms of using UITs, I think this is promising in terms of including a mobile component.


The next article, by Morelli, Mahan, and Illingworth, also looked at measurement variance of mobile versus non-mobile (i.e., PC-delivered) internet-based tests, with respect to four types of assessment: cognitive ability, biodata, a multimedia work simulation, and a text-based situational judgment test.  Data was gathered from nearly 600,000 test-takers in the hospitality industry who were applying for maintenance and customer-facing jobs in 2011 and 2012 (note the different job types).  Nearly 25,000 of these applicants took the assessment on mobile devices.

Results?  The two types of administrations appeared be equivalent in terms of what they were measuring.  However, interestingly, mobile test-takers did worse on the SJT portion.  The authors reasonably hypothesize this may be due to the nature of the SJT and the amount of attention it may have required compared to the other test types.  (btw this article appears to be based on Morelli's dissertation, which can be found here--it's a treasure trove of information on the topic)

Again, overall these are promising results for establishing the measurement equivalence of mobile assessments.  What does this all mean?  It suggests that unproctored tests delivered using mobile devices are measuring the same things as tests delivered using more traditional internet-based methods.  It also looks like fakability or inflation may be a non-issue (compared to traditional UIT).  This preliminary research means researchers and practitioners should be more confident that mobile assessments can be used meaningfully.

I agree with others that this is only the beginning.  In our mobile and app-reliant world, we're only scratching the surface not only in terms of research but in terms of what can be done to measure competencies in new--and frankly more interesting--ways.  Not to mention all the interesting (and important) associated research questions:

- Do natively developed apps differ in measurement properties--and potential--compared to more traditional assessments simply delivered over mobile?

- How does assessment delivery model interact with job type?  (e.g., may be more appropriate for some, may be better than traditional methods for others)

- What competencies should test developers be looking for when hiring?  (e.g., should they be hiring game developers?)

- What do popular apps, such as Facebook (usage) and Candy Crush (score), measure--if anything?

- Oh, and how about: does mobile assessment impact criterion-related validity?


Lest you think I've forgotten the rest of this excellent issue...

- Maclver, et al. introduce the concept of user validity, which uses test-taker perceptions to focus on ways we can improve assessments, score interpretation, and the provision of test feedback.

- Bing, et al. provide more evidence that contextualizing personality inventory items (i.e., wording the items so they more closely match the purpose/situation) improves the prediction of job performance--beyond noncontexual measures of the same traits.

- On the other hand, Holtrop, et al. take things a step further and look at different methods of contextualization.  Interestingly, this study of 139 pharmacy assistants found a decrease in validity compared to a "generic" personality inventory!

- This study by Ioannis Nikolaou in Greece of social networking websites (SNWs) that found job seekers still using job boards more than SNWs, that SNWs may be particularly effective for passive candidates (!), and that HR professionals found LinkedIn to be more effective than Facebook.

- An important study of applicant withdrawal behavior by Brock Baskin, et al., that found withdrawal tied primarily to obstructions (e.g., distance to test facility) rather than minority differences in perception.

- A study of Black-White differences on a measure of emotional intelligence by Whitman, et al., that found (N=334) Blacks had higher face validity perceptions of the measure, but Whites performed significantly better.

- Last, a study by Vecchione that compared the fakability of implicit personality measures to explicit personality measures.  Implicit measures are somewhat "hidden" in that they measure attitudes or characteristics using perceptual speed or other tools to discover your typical thought patterns; you may be familiar with project implicit, which has gotten some media coverage.  Explicit measures are, as the name implies, more obvious items--in this case, about personality aspects.  In this study of a relatively small number of security guards and semiskilled workers, the researchers found the implicit measure to be superior in terms of fakability resistance.  (I wonder how the test-takers felt?)


That's it for this excellent issue of IJSA, but in the last few months we also got some more great research care of the March issue of the Journal of Applied Psychology:

- An important (but small N) within-subjects study by Judge, et al. of the stability of personality at work.  They found that while traits exhibited stability across time, there were also deviations that were explained by work experiences such as interpersonal conflict, which has interesting implications for work behavior as well as measurement.  In addition, the authors found that individuals high in neuroticism exhibited more variation in traits over time compared to those who were more emotionally stable.  You can find an in press version here; it's worth a read, particularly the section beginning on page 47 on practical implications.

- Smith-Crowe, et al. present a set of guidelines for researchers and practitioners looking to draw conclusions from tests of interrater agreement that may assume conditions that are rarely true.

- Another interesting one: Wille & De Fruyt investigate the reciprocal relationship between personality and work.  The researchers found that while personality shapes occupational experiences, the relationship works in both directions and work can become an important source of identity.

- Here's one for you assessment center fans: this study by Speer, et al. adds to the picture through findings that ratings taken from exercises with dissimilar demands actually had higher criterion-related validity than ratings taken from similar exercises!

- Last but not least, presenting research findings in a way that is understandable to non-researchers poses an ongoing--and important--challenge.  Brooks et al. present results of their study that found non-traditional effect size indicators (e.g., a common language effect size indicator) were perceived as more understandable and useful when communicating results of an intervention.  Those of you that have trained or consulted for any length of time know how important it is to turn correlations into dollars or time (or both)!

That's it for now!

March '14 IJSA

In my last research update just a couple days ago, I mentioned that the new issue of IJSA should be coming out soon.

I think they heard me because it came out literally the next day.

So let's take a look:

- This study adds to our (relatively little) knowledge of sensitivity reviews of test items and finds much room for improvement

- More evidence that the utility of UIT isn't eliminated by cheating, this time with a speeded ability test

- Applicant motivation may be impacted by the intended scoring mechanism (e.g., objective vs. ratings).

- The validity of work experience in predicting performance is much debated*, but this study found support for it among salespersons, with personality also playing a moderating role.

- A study of the moderating effect of "good impression" responding on personality inventories

- This review provides a great addition to our knowledge of in-baskets (a related presentation can be found through IPAC)

- Another excellent addition, this time a study of faux pas on social networking websites in the context of employer assessment

- According to this study, assessors may adjust their decision strategy for immigrants (non-native language speakers)

- Letters of recommendation, in this study of nonmedical medical school graduate students, provided helpful information in predicting degree attainment

- Interactive multimedia simulations are here to stay, and this study adds to our confidence that these types of assessments can work well

Until next time!

* Don't forget to check out the U.S. MSPB's latest research study on T&Es!

Research update

Okay, I'm just gonna say it.  There is an insane amount of research out there directly or indirectly related to our field.  I'll be honest, it's a little daunting thinking about reading and processing all of it.

Luckily, you have me to overly simplify it for you so you can incorrectly describe it to others.  So let's jump right in!

First up, the June issue of IJSA:

-  Fairness perceptions matter in web-based selection too!

-  The predictive validity of conscientiousness is moderated by self-enhancement

-  Speaking of conscientiousness, it's a better predictor of performance in routinized jobs than those with complex cognitive requirements

-  Psychological hardiness predicts adaptability in military leaders.  This study is awesome also for showing a negative relationship between SAT scores and West Point performance.

-  Recruiters, listen up: this study found that contingent workers converted to full-time status performed as well as referral and online hires, but not as well as those sourced internally (bookmark this one, there's a dearth of recruitment research)

- Scoring biodata: empirical keying reigns supreme over rational and quasi-rational approaches.  Quasi-rational?  Sounds like me most of the time.

- Personality and job performance: interactions are important beyond main effects


Okay, next, the May issue of JAP:

- Adaptive decision making in military leaders: both the brain and the mind are important (okay you monists out there, chill out)

- Are you a MANOVA (wo)man?  Then read this.

- Proactive leaders set more challenging goals and have higher sales performance--assuming they have the trust of their subordinates.


Next, the April issue of JOB.  Just one study, albeit interesting, in which female evaluators were less likely to recommend hiring or promoting Asian (versus White) applicants into jobs requiring social skills


Speaking of JOB, how about the May issue?

- Have you been calling for more research on calling?  Your wish is granted.

- Curious about the concept of curiosity? (okay, I'll stop)  Looks like it can be predictive of job performance above and beyond traditional cognitive and non-cognitive predictors.  I'm gonna assume it varies with job, but I think the authors are probably right that it will increase in importance over time.

- When selecting for teams that may experience crisis situations (e.g., nuclear power plant crews), consider homogeneity--not mean levels--of positive affect.

- Moderate levels of supervisory structure combined with high levels of consideration lead to lowest levels of CWB's

- The dark side of OCBs

- The light side of OCBs

- Dark side and light side working together in a picture that has nothing to do with selection


Okay, onwards and upwards: one from the April issue of JASP, on generalized self-efficacy, work-related self-efficacy, and job-related outcomes


And one from the May issue:  Sensation seeking and need for structure predict military field exercise performance


How about a little EI research?  Haven't had a lot of that lately.  Here's a piece from the May issue of Journal of Management that found emotion management ability to be a valid predictor of job performance.


Let's look in the May Psychological Science:

- Where we find a fascinating study that demonstrates assessment of profound cognitive abilities at a young age predicts outstanding contributions in many adults

- Another interesting study of how an individual responds to daily stressors (which I would posit are differentially found in various jobs) has long-term consequences for their mental health

- High schoolers with high math and verbal skills are less likely to choose STEM careers than those with math skills but moderate verbal skills

- Okay, check this one out: mindfulness training improved GRE reading comp scores as well as working memory.  Implications for personnel assessment??


Shifting to the May Psychological Bulletin:

- Rorschach, anyone?

- Believing that individuals are malleable, rather than fixed ("implicit theory") predicts goal achievement.  Implications for leaders?


Last but not least, let's hook up with our friends at PARE and see what they're up to:

- Into Excel?  Check out this article on forest plots

- Or maybe factor analysis is your thing.


That's all folks!  Hope your brain is spinning like mine!

Research update

 
Cold/wet yet?  Well sit back, have some hot tea, and let's catch up on our research...

Let's start with the biggie: the December issue of IJSA.

-  Juggling selection quality and adverse impact continues to be tricky.  The authors in this article suggest an optimum combination.

- Reflecting results from the turnover literature, this study found a relationship between perceptions of promotion practices and organizational justice and job satisfaction.

- Speaking of justice, the authors of this study found that the relationship between perceptions of distributive justice and intentions to recommend an employer were moderated by applicant affect.

- Job seekers would to well to keep up their psychological well being and self-esteem (easier said than done, right?).

- Back to fairness.  It's reasonable to think that ethnic minority applicants may not perceive video resumes well (due to the increased salience of their minority status).  But at least in this study, that assumption was not supported--although it depended on ethnic identity and language proficiency.

-  Honestly, I'm not one for the pure concept of "multi-tasking": in my experience people perform in serial, not parallel.  But that doesn't stop people from researching the predictive validity of multi-tasking assessments.

- Faking of personality inventories has been one of the hottest topics in assessment for years, but is often framed as dichotomies.  This study looks to bring some needed complexity to the issue using qualitative data.

- We all know it can be challenging to get hiring manager to give up their unstructured interviews of questionable validity.  Interestingly, according to this study, the more a hiring manager has used unstructured interviews, the less open they are to change.  I don't know if this is a causality issue, a mediating variable issue, or more evidence of the inability to accurately judge one's abilities. 

- Need more evidence for discrimination that occurs during resume screening?  Here ya go.

- Understanding why certain individuals perform better during interviews is a needed area for study.  In this article, the authors demonstrate the importance of simply being ability to generate ideas, rather than analyzing the situational requirements.

- This study investigates another area needing more attention: the selection into medical training and education programs.  The authors found strong predictive support for SJTs but they came with lower face validity.

- Last in this issue is another interesting study, this time of 360-ratings of innovation.  Interestingly and unlike a lot of other research on self-perception, self-ratings were lower than overall observer ratings.  However, the situation got more complex when the authors separated and analyzed by level of self-rating.


Next the November issue of Journal of Applied Social Psychology:

- One of the most pervasive (yet bizarrely under-discussed) areas of discrimination in the workplace is age discrimination.  This study illustrates some of the stereotypes held of younger and older workers--by both groups.  (Spoiler alert: you'll find out how accurate some of these are in just a second)

- I've been waiting for this one, partly because I love hearing about how bizarre and non-face valid some puzzle-based interviews are.  This particular study was looking at perceptions of these interviews compared to a behavioral interview.  Results?  The puzzle-based interviews were consistently less popular.  Oh, did I mention that they didn't work as well?


How about the Winter issue of Personnel Psych?

-  Why do certain applicants withdraw from the recruitment process?  This study suggests a relationship with organizational identification.

- Okay, back to stereotypes about older workers.  Whereas the earlier study looked at what the stereotypes are, this one looks at whether they're true.  The answer: no, but for one: older workers are less willing to participate in training and career development.

- Turns out it's not just the unemployed that are frustrated by the job search process--currently employed individuals feel the same in many ways.  Boy it's too bad we don't have a giant shared database that is able to match job demands with worker abilities...wait...


The November issue of Journal of Applied Psychology has a couple gems:

- Evidence for the predictive validity of the external manifestations of personality as well as the associated implicit motives.

-  A reminder that what makes for effective leadership behavior depends on the culture.


The latest issue of Personnel Review has an interesting research article on utility analysis, where the authors reiterate how challenging it can be to communicate UA information (hint: carrot and stick approach may work best).


There are a couple good ones in the November issue of Psychological Science.

- Multiple-choice tests have been beat up in the past for being nothing more than tests of recognition (rather than productive retrieval).  This study presents evidence that refutes that assumption.  Go multiple-choice!

- Conspiracy theorists take note: governments may be less likely to use the assumed relationship between genetic testing and intelligence to pigeon-hole us into tracks.  Why?  Because, at least according to this study, there appears to be little evidence connecting the two.


Okay, this one is pretty cool--in a slightly scary way.  The authors were looking at the impact that virtual avatar attractiveness has on interview ratings.  Turns out our bias toward attractive people is so strong it extends to the virtual world!  Of course maybe I should have seen that coming...I mean, ever read a comic book?  (hat tip)


Still with me? Last but not least, some disturbing new evidence regarding significance testing and potential publication bias (hat tip).  I'm guessing most of you won't be surprised at the finding.


I don't know if I'll have another update before the end of the year, so if I don't, happy holidays to everyone!

May mega research update

It's time once again for the monthly research round-up.  So let's dive right in:

The June International Journal of Selection and Assessment doesn't disappoint; let's take a look:

- More evidence of the link between personality variables and CWBs; this time with concurrent data in China

- Dovetailing nicely with a post I've been working on regarding promotional testing, this research indicates some interesting characteristics of internal test takers

- Why are supervisors open to behavioral interviews but shun discussion of "structure"?  Looks like how we communicate about them plays a big role.

- More research on self-efficacy, this time teasing apart the concept a bit.

- Always a popular topic: applicant reactions to selection mechanisms.  This time with a sample from Saudi Arabia.

- Speaking of applicant reactions...how about another study?  This one comparing U.S. and Vietnamese college students.  By the way, not surprisingly work samples came out a winner in both of these studies.

- Next, a fascinating study of a hidden bonus to UIT: despite the cheating element, it likely increases your candidate pool and eventually performance outcomes

- Speaking of response distortion, here's another study, this time of military cadet selection using personality inventories

- Okay, one more on inflation.  This time a study of Chinese applicants--no difference compared to American samples.

- Back in March I wrote about a study Jeremy Bernerth published in J.A.P.  that got a lot of attention.  This time, Bernerth studied ethnic differences and found minority status was negatively related to credit scores.


Moving on to the summer issue of Personnel Psychology:

-  The "file drawer problem" is the theory that nonsignificant results are less likely to get published.  According to this study, that appears unlikely.  But IMHO looking at all correlations is different than looking at the correlations key to one's hypothesis(es)...

- Back to faking (that may be this post's theme!), can response elaboration reduce faking on biodata items?  This study suggests so.  Although I'm left wondering...what was the impact on validity?

- Speaking of biodata, there are various ways of keying these items.  This research suggests the best method depends on your sample size, although rational keying performed the worst.


How about the May issue of Journal of Applied Psychology?

-  Well this is interesting...Chad Van Iddekinge and his colleagues have provided an updated meta-analysis on the criterion-related validity of integrity tests.  What did they find? Well, the results appear to be less promising than those published previously (e.g., corrected r=.18 for job performance).  Much like SIOP's research journal, this time J.A.P. published several commentaries in response to the study that...well, let's just say a debate ensued about the analysis...

- The Dark Triad.  It sounds like something in a Dan Brown novel.  But in this meta-analysis the authors show that personality characteristics that make up this triad (Machiavellianism, narcissism, and psychopathy) explain some variance in CWBs.

- Why are some people more proactive in seeking career goals than others?  It's an important and under-researched question.  In this study the authors show that part of the explanation lies in "future work selves", or how people's hopes and aspirations as they relate to work.

- Think self-reports of CWBs are biased?  Perhaps not, according to this new study.

- Interested in what causes proactive customer service behavior?  According to this multi-national study, self-efficacy is a key (along with service climate).

- Why do some leaders engage in more self-interested behavior than others?  Perhaps not surprisingly, it appears due in part to the strength of their moral identity.


The May issue of the Journal of Applied Social Psychology has a couple gems...

- Hey, look, turns out being sensitive to your subordinates pays off.  Talk about a lesson that needs frequent repeating...

- And that's it.  Oh, wait, just this little study about using Facebook profiles to predict job performance...that I wrote about before....available in FULL right now...


Okay, getting to the end...The May/June issue of HRM:

- An interesting study of adverse impact in promotion decisions for managers in a Fortune 500 retailer.  The authors compared three methods (top-down assessment, assessment centers, and  multisource appraisal) and the results demonstrate how complex these analyses are!

- Speaking of complex.  Think that successful job postings on the web is just fancy graphics?  Think again--it still involves some classic factors like the labor market, firm reputation, and compensation incentives.  The more things change...

- Identifying future leaders.  There are few other issues that are as important for most organizations.  Yet how exactly to do it eludes many.  These authors propose a model that focuses on four main features: analytical ability, learning agility, drive, and emergent leadership.


Finally, a few from PARE:

- Does item order impact response anxiety?  Not according to this study.

- What's that?  How do we use a new jacknife procedure to eliminating items and improve structural equation modeling?  You're in luck.

- Looks like a lot of research rely on beta weights when interpreting and reporting multiple linear regression results.  But there's so much more...


Happy reading!

Advice to bballstud_23@mail.com: Apply using a different email address

It's not that I don't appreciate your desire to be different, but I worry about your chances of getting a job.

Why? Because if you apply for a job using that email address, you're going to look:

a) anti-social
b) unprofessional
c) careless
d) inexperienced

This may seem obvious, but according to a little gem tucked away in the latest TIP by Sachau and colleagues, not only are people with, shall we say, "casual" email addressed perceived negatively, they actually score lower on preemployment tests.

The authors used data from over 15,000 actual job applicants who had completed various tests from SHL. Email addresses were coded on a variety of scales such as craziness/insanity, drugs/alcohol, and sci-fi/geeky/nerdy. Then they analyzed the differences in test scores between people with more professional emails with the more casual ones.

Not only did raters agree on the lack of professionalism these types of addresses exhibited, individuals with these addresses actually performed lower on measures of conscientiousness, professionalism, work experience, and total score.

And while the effect sizes weren't huge (there was about a 10% difference in group means between high and low groups), it should give you pause that even if your address is questionable as opposed to outright silly, you will be perceived as unprofessional and may be assumed to have lower test scores.

...unless that's what you're going for.

March IJSA: So much good stuff

The March 2011 issue of the International Journal of Selection and Assessment is out, and it's a doozie. Check it out:

- Beaty, et al. present evidence that unproctored Internet tests (noncognitive: personality and biodata) generally had similar criterion-related validities across a spectrum of job performance criteria compared to the same administered in a proctored setting. Mmmm....UIT...

- Generally when people re-take a cognitive ability test, they do better. But do they do better on other tests of cognitive ability? Matton, et al. describe a study that looked at just that and found the answer to be "no."

- One advantage that is frequently claimed about personality inventories is that usage results in less adverse impact compared to, say, ability or knowledge tests. But might the AI depend on how hires are made (e.g., top-down, compensatory)? Turns out the answer is yes--at least according to some results by Risavy and Hausdorf.

- Hey, government agencies, still debating whether to put more resources into your career web portal? Maybe this will convince you. Selden and Orenstein show that governments with more usable portals as well as better available content not only attract more applicants per opening, but have less voluntary turnover of new hires.

- With advances in technology and changes to the work environment, clerical jobs have changed a lot over the last 30 years and the old ways of selecting for these jobs (namely g-loaded tests such as perceptual speed and verbal ability) likely need to be re-thought...right? Well, not so much, at least according to a meta-analysis by Whetzel and her colleagues. In fact, the criterion-related validity values met or exceeded those found 30+ years ago. The more things change...

- De Goede, et al. present the results of a study that explores the relationship between P-O fit and organizational websites but include the concept of person-industry fit. One implication: if you're trying to attract a more diverse group of candidates, work on making your portal more attractive.

- We spend a lot of time trying to make sure interviews are loaded with job-relevant content. But how much attention do we pay to the impact of impression management tactics on the part of applicants? Huffcutt's results make a compelling argument that we ignore the latter to our detriment as it may have more to do with interview ratings than the job-relevant content.

- How does one determine managerial potential? Well, it depends who you ask. Thomason, et al. present results that indicate when supervisors are asked, they focus on task-based personality traits (e.g., conscientiousness), whereas peers focus on contextual traits such as agreeableness. Given that leadership is ultimately about achieving things through subordinates, I wonder what we should be paying attention to....hmmm...how about both?

- Thinking about using self-ratings of political skill as part of the application process? I can certainly see situations where this skill may be helpful, but might this method be susceptible to inflation? Not so much, at least according to results from Blickle, et al.

- Last but definitely not least, Carless & Hetherington with some data on the impact of recruitment timeliness on applicant attraction. The longer we make applicants wait, the less attracted to the organization they will be, right? Not so fast. According to this research, it is perceived timeliness that matters, not actual timeliness (hence the importance of communication). In addition, this relationship is partially mediated by job and organizational characteristics.

Jan TIP gems: HRO, UIT, SIOP, and VII

Yes, my goal was to create a blog post title using words no longer than four letters.

Anyway, for those non-SIOP'ers out there, or SIOP'ers that may have missed 'em, there were some gems in the latest issue of TIP:

How I/O can shape the practice of strategic human resources outsourcing (HRO)

A great little study on perceptions of various ways of mitigating cheating on unproctored Internet testing (UIT)

The difference between academics and practitioners in terms of what topics are valued at the SIOP conference (e.g., the latter were more interested in job analysis, staffing, and strategic HR)

Last but not least, a point, counter-point on whether the addition of sex as a protected category under Title VII was a joke

New issues of Personnel Psych and IJSA: Muslim applicants, selection perceptions, and more

Two of the big journals have come out with new issues, so let's take a look at the highlights:

First, in the Winter 2010 issue of Personnel Psychology:

King and Ahmad describe the results of several experiments that found interviewers and raters altered their behavior for confederates exhibiting obvious Muslim-identified behavior (e.g., clothing) depending on whether applicants exhibited stereotype-inconsistent behavior. For those that didn't, reactions were shorter and more negative. On the other hand, no difference was found in offers between those dressed in Muslim-identified clothing and those that weren't. So behavior--specifically its stereotypicality--and not simply something obvious like dress, may be key in predicting/preventing discriminatory behavior.

Do you consistently read the "Limitations" section of journal articles? Brutus et al. did, for three major I/O journals from 1995 to 2008 and found that threats to internal validity were the most commonly reported limitation. Interestingly, they also found that the nature of limitations reported changed over time (e.g., more sampling issues due to volunteers, variance issues). You can see an in press version here.

Next up, Henderson reports impressive criterion-related validity (combined operational = .86) for a test battery consisting of a g-saturated exam and a strength/endurance exam after following a firefighter academy class for 23 years. He suggests that employers have considerable latitude in choosing exams as long as they are highly loaded on these two factors, and also suggests approximately equal weighting.

Struggling to communicate the utility of sound assessment? Last but not least, Winkler et al. describe the results of sharing utility information with a sample of managers and found that using a casual chain analysis--rather than simply a single attribute--increased understanding, perceived usefulness, and intent to use.


Let's switch now to the December 2010 issue of the International Journal of Selection and Assessment (IJSA). There's a lot of great content in this issue, so take a deep breath:

Interested in unproctored internet testing? You'll want to check out Guo and Drasgow's piece on verification testing. The authors recommend using a Z-test over the likelihood ratio test for detecting cheating, although both did very well.

Walsh et al. discuss the moderating effect that cultural practices (performance orientation, uncertainty avoidance) have on selection fairness perceptions.

Speaking of selection perceptions, Oostrom et al. found that individual differences play a role in determining how applicants respond--particularly openness to experience. The authors recommend considering the nature of your applicant pool before implementing programs to improve perceptions of the assessment process.

Those of you interested in cut scores (and hey, who isn't) should check out Hoffman et al.'s piece on using a difficulty-anchored rating scale and the impact it has on SME judgments.

Back to perceptions for a second, Furnham and Chamorro-Premuzic asked a sample of students to rate seventeen different assessment methods for their accuracy and fairness. Not surprisingly, panel interviews and references came out on top in terms of fairness, while those that looked the most like a traditional test (e.g., drug, job knowledge, intelligence) were judged least accurate and fair. Interestingly, self-assessed intelligence moderated the perceptions (hey, if I think I'm smart I might not mind intelligence tests!).

And now for something completely different (those of you that get that reference click here for a trip down memory lane). A study by Garcia-Izquierdo et al. of information contained in online job application forms from a sample of companies found on the Spanish Stock Exchange. A surprisingly high percentage of firms asked for information on their applications that at best would be off-putting, at worst could lead to lawsuits, such as age/DOB, nationality, and marital status. The authors suggest this area of e-recruitment is ripe for scientist-practitioner collaboration.

Last but not least, a piece that ties the major topics of this post together: selection perceptions and recruitment. Schreurs et al. gathered data from 340 entry-level applicants to a large financial services firm and found that applicant perceptions, particularly of warmth/respect, mediated the relationship between expectations and attraction/pursuit intentions. This reinforces other research that has underlined the importance of making sure organizational representatives put your best foot/face forward.

Q&A with Piers Steel: Part 1

A few weeks ago I wrote about a research article that I think proposes a revolutionary idea: The creation of synthetic validity database that would generate ready-made selection systems that would rival or exceed the results generated through a traditional criterion validation study.

I had the opportunity to connect with one of the articles authors, Piers Steel, Associate Professor of Human Resources and Organizational Dynamics at the University of Calgary. Piers is passionate about the proposal and believes strongly that the science of selection has reached a ceiling. I wanted to dig deeper and get some details, so I posed some follow-up questions to him. Read on for the first set of questions, and I'll post the rest next time:

Q1) What is the typical state of today's selection system--what do we do well, and what don't we?

A1) Here is quote from well a respected selection journal, Personnel Psychology: “Psychological services are being offered for sale in all parts of the United States. Some of these are bona fide services by competent, well-trained people. Others are marketing nothing but glittering generalities having no practical value.... The old Roman saying runs, Caveat emptor--let the buyer beware. This holds for personnel testing devices, especially as regard to personality tests.”

Care to try and date it? It is from the article “The Gullibility of Personnel Managers,” published in 1958. Did you guess a different date? You might of, as the observation is as relevant today as yesterday -- nothing fundamental has changed. Just compare that with this more recent 2005 excerpt from HR Magazine, Personality Counts: “Personality has a long, rich tradition in business assessment,” says David Pfenninger, CEO of the Performance Assessment Network Inc. “It’s safe, logical and time-honored. But there has been a proliferation of pseudo tests on the market: Caveat emptor.”

Selection is typically terrible with good being the exception. The biggest reason is that top-notch selection systems are financially viable only for large companies with a high-volume position. Large companies can justify the $75,000 cost and months to develop and validate and perhaps, if they are lucky, have the in-house expertise to identify a good product. Most other employers don’t the skill to differentiate the good from the bad as both look the same when confronted with nearly identical glossy brochures and slick websites. And then the majority of hires are done with a regular unstructured job interview – it is the only thing employers have the time and resources to implement. Interviews alone are better than nothing but not much better – candidates are typically better at deceiving the interviewer than the interviewer is at revealing the candidate.

The system we have right now can’t even be described as being broken. That implies it once worked or could be fixed. Though ideally we could do good selection, typically, it is next to useless, right up there with graphology, which about a fifth of professional recruiters still use during their selection process. For example, Nick Corcodilos reviews how effective internet job sites are getting people a position. He asks us to consider “is it a fraud?”

Q2) What's keeping us from getting better?

A2) Well, there are a lot of things. First, sales and marketing works, even if the product doesn’t. When you have a technical product and untechnical employer or HR office, you have a lot of room for abuse. I keep hearing calls for more education and that management should care more. You are right they should care more and know more. People should also care and know more about their retirement funds as well. Neither is going to change much.

Second, the unstructured job interview has a lot of “truthiness” to it. Every professional selection expert I know includes a job interview component to the process even when it doesn’t do much, as the employer simply won’t accept the results of the selection system without it. There are some cases where people “have the touch” and are value added but this is the exception. Still, everyone thinks they are gifted, discerning, and thorough. This is the classic competition between clinical and statistical prediction, with evidence massively favoring the superiority of the latter over the former but people still preferring the former over the latter (here are few cites to show I’m not lying, as if you are like everyone else, you won’t believe me: Grove, 2005; Kuncel, Klieger, Connelly, & Ones, 2008).

Third, it just costs too much and takes too much time to do it right. Also, most jobs aren’t really large enough to do any criterion validation.

Q3) What might the future look like if we used the promise of synthetic validity?

A3) Well, to quote an article John Kammeyer-Mueller and I wrote, our selection systems would be "inexpensive, fast, high-quality, legally defensible, and easily administered.” Furthermore, every year they would noticeable improve, just like computers and cars. A person would have their profile taken and updated whenever they want, with initial assessments done online and more involved ones conducted in assessment centers. Once they have the profile, they would get a list of jobs they would likely be good at, ones that they would be likely good at and enjoy, and ones they would be likely good at, enjoy and that are in demand.

Furthermore, using the magic of person-organization fit, you inform them what type of organization they would like to work for. If someone submitted their profile to a job database, every day job positions would come to them automatically, with the likelihood of them succeeding at it. These jobs would come in their morning email if they wanted it. Organizations would also automatically receive appropriate job applicants and a ready built selection system to confirm that the profile submitted by the applicant was accurate.

Essentially, we would efficiently match people to jobs and jobs to people. I would recommend people update their profile as they get older or go through a major life change to improve the accuracy of the system, but even initially it would be far more accurate than anything available today -- a true game changer.

Follow-up: Some might see a contradiction here. You cite an article that bashes internet-based job matching, yet this is what you're suggesting. Would your system be more effective or simply supplement traditional recruiting methods (e.g., referrals)?

A: Yup, we can do better. The internet is just a delivery mechanism and no matter how high-speed and video enabled, it is just delivering the same crap. This would provide any attempt to match people to jobs or jobs to people with the highest possible predictiveness.


Next time: Q&A Part 2

References:
Grove, W. M. (2005). Clinical versus statistical prediction: The contribution of Paul E. Meehl. Journal of Clinical Psychology, 61(10), 1233-1243. doi: 10.1002/jclp.20179

Kuncel, N. R., Klieger, D., Connelly, B., & Ones, D. S. (2008, April). Mechanical versus clinical data combination in I/O psychology. In I. H. Kwaske (Chair), Individual Assessment: Does the research support the practice? Symposium conducted at the annual meeting of the Society for Industrial and Organizational Psychology, San Francisco, CA.

Stagner, R. (1958). The Gullibility of Personnel Managers. Personnel Psychology, 11(3), 347-352.

Latest IJSA: Emotional intelligence, multiple-choice formats, and lots more

The March 2010 issue of the International Journal of Selection and Assessment (IJSA) is out, and the research covers a wide variety of recruitment and assessment topics as well as being truly international:

Unproctored internet-based testing (UIT) response distortion may be less than we fear (sample included cognitive and personality measures)

What factors are most important to organizations when choosing a test? This study suggests applicant reaction, cost, and diffusion of the test type in the field.

Personality (esp. core self evaluation) is related to the type of work preferred, and hence P-O fit

Career site features may differentially attract men and women

Corporate images do matter when it comes to organizational attractiveness

Who uses job-search websites and how to improve them (the sites, not the people)

Support for performance-based (as opposed to self-report) measures of emotional intelligence

Work samples, interviews, and ability tests perceived best by employees (why? because they work, say the participants)

...and last but definitely not least:

A "2 of 5" multiple-choice format seems superior than traditional "1 of 6" (you just have to make sure you can score them that way!)

The state of UIT

Many recruiters and assessment professionals believe that the future of employment testing lies with screening applicants over the Internet (rather than in person). And while there are substantial benefits associated with this method (e.g., convenience, speed), there are plenty of concerns as well (e.g., cheating, validity). Research in this area is in its infancy, which is why it is refreshing to see a full half of the March 2009 issue of Industrial and Organizational Psychology is devoted to describing the state of unproctored internet testing (UIT).

What struck me most about when reading these articles is the variety and excitement surrounding this field but even more, the tremendous lack of consensus in the professional community has about important issues related to UIT.

The articles start off with an updated summary by Nancy Tippins, who with her colleagues provided a heavily cited summary in 2006. This focal article is followed by twelve response articles and a final summary by Tippins.

Here's a (very) brief summary of some of the important points raised by the authors:

1. Cheating happens. But let's not forget that proctored tests have always been susceptible to some degree of cheating (e.g., via test question memorization).

2. There are many ways to mitigate the risks associated with cheating. This includes retesting, identity verification, and response pattern analysis. But it's not clear how successful these measures are, or even how needed they are.

3. Although there are potential legal risks (e.g., lack of standardized administration), UITs have not been directly evaluated in court.

4. The choice of whether or not to use UIT is influenced by many factors, not the least of which is the organizational reality communicated by upper management.

5. Some applicants may be turned off by an organization that uses assessments so obviously prone to cheating. But this may be balanced by increased convenience, speed, and immediate feedback.

6. Although cheating and response distortion occurs, it's unclear to what extent it impacts validity.

As an interesting note, the most common types of tests delivered via UIT seem to be biodata, personality, situational judgment, and T&E and preference questionnaires. There are also those who are administering cognitive ability tests in this way, sometimes adaptive.

UIT is in many ways the Wild West of employment testing. It's exciting and innovative, but comes with risk and lots of unanswered questions.

Here's hoping our field quickly speeds up the research side because this ship has clearly sailed.

Living with T&Es

There are a number of ways to use the Internet to perform personnel assessment. Examples include timed cognitive and job knowledge tests, biodata instruments, and personality inventories. But one of the easiest--and thus most tempting to use--types of tests is known as training and experience questionnaires, or T&Es (also sometimes called E&E for education and experience).

A typical (poor) example of a T&E item might be something like this:

How much experience do you have conducing job analysis?

a) None
b) Less than 2 years
c) 2-4 years
d) 4 or more years

I could go on at length about the challenges inherent with using this type of assessment, but I'll spare you. Instead I'll point you to Jim Higgins' December 2008 newsletter, HR Rampage, in which he addresses this topic, among others (see page 2).

Jim points out several problems with this type of assessment, including the overwhelming urge to self-inflate in high-stakes scenarios (we already have problems with outright cheating), the inability of highly qualified individuals to give themselves sufficient credit, and the work required to validate responses.

So given all these challenges, what can we do to mitigate them? The solutions investigated so far (e.g., elaboration, warnings) have met with very limited success. But there are a number of tactics we can take in this situation. Here are some other methods to consider as we wait for more research in this area:

1) Accurately describe the job and requirements to prevent an unqualified individual from applying in the first place.

2) Clearly word stems and responses to avoid legitimate mis-reads.

3) Include lie items (e.g., "I have experience using the HR Tests Job Analysis Technique") and deduct points when candidates endorse them.

4) Use false bottoms (e.g., both (a) and (b) are worth zero points) and false tops (e.g., both (c) and (d) are worth the same).

5) Use scales appropriate to the item. For example, amount of experience is often the incorrect scale; type of experience is better.

6) Encourage hiring supervisors to follow up on specific items in their interview.

7) Use a friendly zero point, such as "I do not have any experience but I would be willing to learn" instead of "none." Remember there are egos involved here.

8) Ask questions that are appropriate for a T&E. Don't ask candidates to rate their oral communication skills.

9) Don't ask people to compare themselves to others (e.g., Average, Above Average). Instead use objective measures such as number of times.

10) Seriously consider weighting the items. This is of course dependent upon subject matter expert input, but it's highly likely that your SMEs consider certain training or experience areas more important than others.

11) Before they even get to the T&E, use willingness/pre-screening questionnaires that ask candidates to acknowledge they understand the less-than-perfect conditions related to the job (e.g., mandatory overtime, working outside in the heat).

12) Consider using them as a feedback tool for candidates rather than a scored instrument (e.g., "Your responses indicate you have very little experience and education related to this job. Would you like to continue to apply?").

13) Base everything on SME input. Yes, I realize this probably doesn't need to be stated, but one of the worst temptations is for HR folk to draft T&Es themselves thinking they're easy to write. This is a myth, and helps contribute to poor quality eligible lists.

This is by no means an exhaustive list, and I'm sure there are other methods out there for helping us live with T&Es. Feel free to add your suggestions!

Giving thanks for research

It's almost Thanksgiving here in the U.S., a time to give thanks, and I'd like to thank a largely unsung group of people. Thank you to all the researchers out there who try to help us put some science around the art we call personnel recruitment and selection. Thank you for all your work and insights.

What better way to celebrate this wish of thanks than by talking about a new issue of the International Journal of Selection and Assessment (v16, #4)! As usual it's chalk full of good articles, so let's take a look at some of them.

First, a study of applicant perceptions of credit checks, something many of us do for sensitive positions. Using samples of undergraduates, Kuhn and Nielsen found mostly negative reactions, especially for older participants, but they varied with the explanation given as well as privacy expectations. Worth a look for any of you that conduct large numbers of background checks (and if you do, don't miss the Oppler et al. study below).

Next up, a fascinating study of police officer selection in the Netherlands. Using data from over 3,000 applicants, De Meijer et al. found evidence for differential validity between ethnic majority and minority participants. Specifically, cognitive ability tests predicted training performance for minorities but not for those in the majority. Performance prediction for the latter group was low for cognitive ability tests and somewhat better using non-cognitive ability variables. By the way, the dissertation of the primary author, a fascinating look at similar issues, can be found here.

The third article is one of those articles that almost (...almost) makes me want to pay for it, and anybody interested in electronic applicant issues take note. In this study, Dunleavy et al. used simulations to show the tremendous impact that small numbers of applicants can have on adverse impact (AI) analysis. In fact, the authors reveal situations where AI can be caused or masked by a single applicant applying multiple times! The authors present ways of identifying and handling these cases. Scary stuff. Hope the OFCCP is reading.

Fourth, Lievens and Peeters present results of a study of elaboration and its impact on faking situational judgment tests. Using master students, the researchers found that requiring elaboration on items (i.e., the reason they chose the response) had several positive results. It reduced faking on items with high familiarity. It also reduced the percentage of "fakers" in the top of the distribution. Lastly, candidates reported that the elaboration allowed them to better demonstrate their KSAs. This could be a great strategy for those of you worried about the inflation effects of administering SJTs online.

Next, Furnham et al. with a study of assessment center ratings. The authors found that expert ratings of "personal assertiveness", "toughness and determination", and "curiosity" were significantly correlated with participant personality scores, particularly Extraversion. Correlations with intelligence test scores were low.

Last but definitely not least, Oppler et al. discuss results of a rare empirical study of financial history and its relationship to counterproductive work behaviors (CWBs). Using a "random sample of 2519 employees" the authors found that those with financial history "concerns" were significantly more likely to demonstrate CWBs after hire. Great support for conducting these types of checks.

There are other articles in here, so I encourage you to check them all out. Thank goodness for research!

Using video games to recruit and select candidates

A new study by the Pew Internet and American Life Project found that:

"virtually all American teens play computer, console, or cell phone games and...the gaming experience is rich and varied, with a significant amount of social interaction and potential for civic engagement."

and

"Game playing is universal, with almost all teens playing games and at least half playing games on a given day"

This raises a question:

Is there a benefit, or even a mandate, to make recruitment and assessment more like a video game?

We've already seen a massive amount of interest in using virtual worlds like Second Life for recruiting (which has met with mixed success). And the U.S. Army is always on the cutting edge with things like America's Army (which has enjoyed quite a bit of success).

When it comes to assessment, we've seen some valiant efforts, such as the virtual job tryout. And video-based testing has been around for a long time.

But with everything that's out there, would you describe your candidate experience as "rich and varied" with a "significant amount of social interaction"?

Laying aside for the moment the fact that many organizations lack even realistic job preview videos, what competitive advantage is to be gained by the employer that figures out how to make its recruitment and selection process interactive? What if instead of the process being a one-way street (candidates search for information about employers, employers try to figure candidates out), it was a two-way simultaneous sharing of information?

Doom came out 15 years ago. The Sims, 8 years ago. Isn't it time we developed realistic 3-dimensional worlds that allow candidates to make real-time branching decisions and learn about a potential employer, while we measure things like attention to detail and judgment?

Is it just me or are we missing an enormous opportunity to attract a new generation of workers and gather valuable competency information at the same time?

Unproctored internet testing: Safe for some tests?

One of the biggest trends in personnel assessment is the movement toward on-line testing. Many organizations are experimenting with so called unproctored Internet testing (UIT), where candidates are allowed to take the exams whenever, and wherever, they want.

Benefits? Extremely convenient for the candidate. Less administrative resources needed by the employer.

Costs? Bye-bye exam security, hello cheating opportunities. Not only is your test out for everyone to see, but you have no real way of knowing (sans biometric verification) who is taking the test.

Some organizations have decided the benefits outweigh the risks, and a new study in the June 2008 issue of the International Journal of Selection and Assessment may provide support for their position.

In it, the authors looked at over 800 applicants from nine European countries that took a test of perceptual speed in an unproctored setting, then followed this up with a proctored parallel version. Results? Not only was there no evidence of cheating, they found the opposite effect--people did better in the proctored setting.

Now before everyone throws out their proctored exams, note that this is a type of test that might be hard to cheat on--at least in one way. Because this is a perceptual speed test, there are no "right" answers that can be looked up. It also required very quick responses. So the only way to cheat would be to have someone take the test for you. Implication: it may make more sense to use certain UITs than others.

This topic is a source of much debate in the assessment community, and there is by no means consensus on the right way to go. But studies like this help!

Take a deep breath, because there's a lot more in this issue:

- The preliminary employment interview as a predictor of assessment center outcomes (fascinating look at how the AC may only make sense for mid-range interview scorers)

- A comparison of the common-item and random-groups equating designs using empirical data (for you IRT fans out there)

- The influence of external recruitment practices on job search practices across domestic labor markets: A comparison of the United States and China

- Beneath the surface: Uncovering the relationship between extraversion and organizational citizenship behavior through a facet approach (a more nuanced look at the relationship shows extraversion can predict OCBs)

- Comparing personality test formats and warnings: Effects on criterion-related validity and test-taker reactions (another good one...personality test added predictive validity beyond ability test but no validity difference between forced-choice and Likert scales, nor between warning and no-warning conditions; forced-choice and warnings may produce negative candidate reactions)

- Applicant selection expectations: Validating a multidimensional measure in the military (describes development of a new measure of applicant perception of the selection process)

- Selecting for creativity and innovation: The relationship between the innovation potential indicator and the team selection inventory

October '07 ACN

The October 2007 issue of the Assessment Council News is out with two great articles:

First, Dr. Mike Aamodt tackles the issue of validity co-efficients with Beauty May Be in the Eye of the Beholder, but is the Same True of a Validity Coefficient? (begins page 2)

In the article, Dr. Aamodt gathers data from experts in the assessment community on questions such as:

- Is there a minimum value for a validity coefficient that would generally be accepted by testing experts? If so, what is it?

(includes a great table summarizing where certain validity coefficient values have been referenced)

- What is the lowest uncorrected validity coefficient that you believe would indicate that an inference from a test has acceptable criterion validity?

- If a validity coefficient is statistically significant, is that enough to imply job relatedness?

The second article is by Natasha Riley and covers a topic near and dear to our hearts--Unproctored Internet Testing--The Technological Edge: Panacea or Pandora's Box? (page 11)

If the title seems a little odd and/or familiar, it's because it's a combination of various presentation titles from the 2007 IPMAAC conference where unproctored Internet testing was a hot topic. In the article, Natasha covers some pros and cons of this type of testing and describes how Riverside County in California is having some success with it.

Remember, internet-based testing does not have to be about selecting out. It can be about giving candidates tools they can use to determine whether they would be a good fit. Cheating is removed as an obstacle when we eliminate the motivation. Consider giving the applicant the "exam" and have them determine whether they want to move forward given their results and how they compare to successful job incumbents.

Anyway, kudos to IPMAAC for another illuminating issue of ACN. Keep 'em comin', and folks watch out for the call for proposals for the 2008 IPMAAC conference! (to be announced shortly)