Research update

Several recent journals to pore through:

July 2016 issue of Journal of Organizational Behavior

July 2016 issue of Journal Applied Psychology, including:

- Computer scoring of candidate essays

- Everything you've been told about assessment center ratings is confounded

and the biggie...

September 2016 issue of International Journal of Selection and Assessment, including:

- Faking on assessment measures

- Emotional intelligence and service jobs

- A self-report measure of psychopathy in the workplace

Just kidding...more research update!

Seriously?  Just yesterday I did my research update, ending with a note that the December 2014 issue of the International Journal of Selection and Assessment should be out soon.

Guess what?  It came out today.

So that means--you guessed it--another research update!  :)

- First, a test of Spearman's hypothesis, which states that the magnitude of White-Black mean differences on tests of cognitive ability vary with the test's g loading.  Using a large sample of GATB test-takers, these authors found support for Spearman's hypothesis, and that reducing g saturation lowered validity and increased prediction errors.

So does that mean practitioners have to choose between high-validity tests of ability or increasing the diversity of their candidate pool?  Not so fast.  Remember...there are other options.

- Next, international (Croatian) support for the Conditional Reasoning Test of Aggression, which can be used to predict counterproductive work behaviors.  I can see this increasingly being something employers are interested in.

- Applicants that do well on tests have favorable impressions of them, while those that do poorly don't like them.  Right?  Not necessarily.  These researchers found that above and beyond how people actually did on a test, certain individual differences predict applicant reactions, and suggest these be taken into account when designing assessments.

- Although personality testing continues to be one of the most popular topics, concerns remain about applicants "faking" their responses (i.e., trying to game the test by responding inaccurately but hopefully increase the chances of obtaining the job).  This study investigates the use of blatant extreme responding, consistently selecting the highest or lowest response option, to detect faking, and looked at how this behavior correlated with cognitive ability, other measures of faking, and demographic factors (level of job, race, and gender).

- Next, a study of assessment center practices in Indonesia.

- Do individuals high in neuroticism have higher or lower job performance?  Many would guess lower performance, but according to this research, the impact of neuroticism on job performance is moderated by job characteristics.  This supports the more nuanced view that the relationship between personality traits and performance is in many cases non-linear and depends on how performance is conceptualized.

- ...which leads oh so nicely into the next article!  In it, the authors studied air traffic controllers and found results consistent with previous studies--ability primarily predicted task performance while personality better predicted citizenship behavior.  Which raises an interesting question: which version of "performance" are you interested in?  My guess is for many employers the answer is both--which suggests of course using multiple methods when assessing candidates.

- Last but not least, an important study of using cognitive ability and personality to predict job performance in a three studies of Chilean organizations.  Results were consistent with studies conducted elsewhere, namely ability and personality significantly predicted performance.

Okay, I think that's it for now!

May mega research update

It's time once again for the monthly research round-up.  So let's dive right in:

The June International Journal of Selection and Assessment doesn't disappoint; let's take a look:

- More evidence of the link between personality variables and CWBs; this time with concurrent data in China

- Dovetailing nicely with a post I've been working on regarding promotional testing, this research indicates some interesting characteristics of internal test takers

- Why are supervisors open to behavioral interviews but shun discussion of "structure"?  Looks like how we communicate about them plays a big role.

- More research on self-efficacy, this time teasing apart the concept a bit.

- Always a popular topic: applicant reactions to selection mechanisms.  This time with a sample from Saudi Arabia.

- Speaking of applicant reactions...how about another study?  This one comparing U.S. and Vietnamese college students.  By the way, not surprisingly work samples came out a winner in both of these studies.

- Next, a fascinating study of a hidden bonus to UIT: despite the cheating element, it likely increases your candidate pool and eventually performance outcomes

- Speaking of response distortion, here's another study, this time of military cadet selection using personality inventories

- Okay, one more on inflation.  This time a study of Chinese applicants--no difference compared to American samples.

- Back in March I wrote about a study Jeremy Bernerth published in J.A.P.  that got a lot of attention.  This time, Bernerth studied ethnic differences and found minority status was negatively related to credit scores.


Moving on to the summer issue of Personnel Psychology:

-  The "file drawer problem" is the theory that nonsignificant results are less likely to get published.  According to this study, that appears unlikely.  But IMHO looking at all correlations is different than looking at the correlations key to one's hypothesis(es)...

- Back to faking (that may be this post's theme!), can response elaboration reduce faking on biodata items?  This study suggests so.  Although I'm left wondering...what was the impact on validity?

- Speaking of biodata, there are various ways of keying these items.  This research suggests the best method depends on your sample size, although rational keying performed the worst.


How about the May issue of Journal of Applied Psychology?

-  Well this is interesting...Chad Van Iddekinge and his colleagues have provided an updated meta-analysis on the criterion-related validity of integrity tests.  What did they find? Well, the results appear to be less promising than those published previously (e.g., corrected r=.18 for job performance).  Much like SIOP's research journal, this time J.A.P. published several commentaries in response to the study that...well, let's just say a debate ensued about the analysis...

- The Dark Triad.  It sounds like something in a Dan Brown novel.  But in this meta-analysis the authors show that personality characteristics that make up this triad (Machiavellianism, narcissism, and psychopathy) explain some variance in CWBs.

- Why are some people more proactive in seeking career goals than others?  It's an important and under-researched question.  In this study the authors show that part of the explanation lies in "future work selves", or how people's hopes and aspirations as they relate to work.

- Think self-reports of CWBs are biased?  Perhaps not, according to this new study.

- Interested in what causes proactive customer service behavior?  According to this multi-national study, self-efficacy is a key (along with service climate).

- Why do some leaders engage in more self-interested behavior than others?  Perhaps not surprisingly, it appears due in part to the strength of their moral identity.


The May issue of the Journal of Applied Social Psychology has a couple gems...

- Hey, look, turns out being sensitive to your subordinates pays off.  Talk about a lesson that needs frequent repeating...

- And that's it.  Oh, wait, just this little study about using Facebook profiles to predict job performance...that I wrote about before....available in FULL right now...


Okay, getting to the end...The May/June issue of HRM:

- An interesting study of adverse impact in promotion decisions for managers in a Fortune 500 retailer.  The authors compared three methods (top-down assessment, assessment centers, and  multisource appraisal) and the results demonstrate how complex these analyses are!

- Speaking of complex.  Think that successful job postings on the web is just fancy graphics?  Think again--it still involves some classic factors like the labor market, firm reputation, and compensation incentives.  The more things change...

- Identifying future leaders.  There are few other issues that are as important for most organizations.  Yet how exactly to do it eludes many.  These authors propose a model that focuses on four main features: analytical ability, learning agility, drive, and emergent leadership.


Finally, a few from PARE:

- Does item order impact response anxiety?  Not according to this study.

- What's that?  How do we use a new jacknife procedure to eliminating items and improve structural equation modeling?  You're in luck.

- Looks like a lot of research rely on beta weights when interpreting and reporting multiple linear regression results.  But there's so much more...


Happy reading!

Research update #583: Impression management and a lot more

Okay, I've got a lot of ground to cover this time, so buckle up...

Let's start with the December issue of IJSA:

- Looks like how much applicants try to make themselves look good varies by country

- Is applicant faking behavior related to job performance? Kinda depends on your definitions.

- Research has found that emotional intelligence can be related to work attitudes. This appears to be in part because of an increased situational judgment effectiveness.

- Speaking of situational judgment...in terms of job knowledge, knowing what to do is different than knowing what not to do

- What impact does a resume have on a recruiter? Depends on what assumptions they make about you after reading it.

- How to people select--and continue with--an executive coach? By looking at things like their ability to forge a partnership.

- How do Canadian firms do in terms of using tests other than interviews? Not so well, it turns out.


Let's move to the October issue of JASP, where there's just one article but it's a good one. Researchers continued the (depressing) finding that applicant names impact pre-interview impressions. Specifically, the more a name was Anglicized, the more favorable the impression was when hiring for an outside sales job.


Next comes the November issue of JAP:

- A new meta-analysis of the FFM of personality and its relationship to OCBs and task performance.

- Measures of interest haven't gotten a lot of love as selection devices. Looks like we need to tease out the constructs a little because they could be more helpful than we thought.

- Applicants trying to create a certain image during an interview are better off doing this after an initial flub or relying solely on self-promotion rather than making up an image.


A few from the November issue of JPSP:

- Another on impression management (not selection-specific) that goes into more detail about the topic (e.g., how many tactics people use, their accuracy)

- A caution about using the Revised NEO-PI in different cultures due to DIF.


Next, a call for more transparency in false-positive findings.


Last but not least, those of you interested in the potential of social ratings of performance being used for selection might be interested in this study of RateMyProfessors.com, which found student ratings are likely to be useful measures of teacher quality.

Research potpourri

Here's a very quick round-up of several recent pieces of research:

Bott et al. on how individual differences and motivation impact score elevation on non-cognitive measures (e.g., personality tests)

Ackerman, et al. on cognitive fatigue during testing

Robie, et al. on the impact of coaching and speeding on Big Five and IM scores

Evers, et al. on the Dutch review process for evaluating test quality (pretty cool; in press version here)

And now, back to figuring out why I eat too much on Thanksgiving. Every year.

September mega research update

I'm behind in posting updates of the September journals. Really behind. So instead of posting a series describing the detailed contents of each issue, I'm going to give you links to what's come out in the last month or so and let you explore. But I'll try to hit some high points:

Journal of Applied Psychology, v95(5)

Highlights include this piece from Ramesh and Gelfand who studied call center employees in the U.S. and India and found that while person-job fit predicted turnover in the U.S., person-organization fit predicted turnover in India.


Human Performance, v23(4)

This issue has some good stuff, including Converse et al.'s piece on different forced-choice formats for reducing faking on personality measures, Perry et al.'s piece on better predicting task performance using the achievement facet of conscientiousness, and Zimmerman et al.'s article on observer ratings of performance.


Journal of Business and Psychology, v25(3)

Another issue filled with lots of good stuff, but I'm almost 100% positive abstract views are session based, so use the title link for article on response rates in organizational research (full text here), the importance of using multiple recruiting activities, and the importance of communicating benefit information in job ads.


Journal of Applied Social Psychology, v40(9)

The article to check out here is by Proost et al. and deals with different self-promotion techniques during an interview and their effect on interviewer judgments.


Journal of Occupational and Organizational Psychology, v83(3)

Check this issue out for articles on organizational attraction, communication apprehension in assessment centers, and the impact of interviewer affectivity on ratings.

Personnel Psychology, Summer 2010

The latest issue of Personnel Psychology (v63, #2) marks the beginning of summer journal season. Let's take a peek at some of what's inside:

Practice makes...better. John Hausknecht studied over 15,000 candidates who applied for supervisory positions (and 357 who repeated the process) over a 4-year period with a large organization in the service industry. The selection process included a personality test. He found that candidates that failed the first time around showed practice effects on dimension-level scores of .40 to .60. Candidates that passed the first time, but were taking the test again for other reasons, generally showed no difference in scores. More interestingly, on several subscales low scores the first time around were associated with practice effects that exceeded one standard deviation. A good reminder that personality inventories are susceptible to "faking", but certainly not a nail in their coffin as they still work quite well in many situations.

Another reason to structure your interviews. As if you needed more convincing, McCarthy et al.'s study of nearly 20,000 applicants for a managerial-level position in a large organization found that the use of a structured interview resulted in zero main effects for applicant gender and race on interview performance. Similarly, there were no effects of applicant-interviewer similarity with respect to gender and race.

Users of the CRT-A take note. The conditional reasoning test of aggression (CRT-A) is used to detect individuals with a propensity for aggression. Previous studies have suggested the criterion-related validity of this test is around r=.44. In this study, by Berry et al., the authors meta-analyzed a large data set and found much lower values, in the .10-.16 range, that rose to .24-.26 when certain studies were excluded.

Assess your way into a job. Last but not least, Wanberg et al. describe the development of an inventory for job seekers called Getting Ready for your Next Job (YNJ, available here). The authors present results tying inventory components (e.g., job search intensity, Internet use) to subsequent employment outcomes.

Stay tuned, new issues of JAP, IJSA, and others should be out soon!

Setting cutoff scores on personality tests

What's the best way to set a cutoff score for a personality test, knowing that some candidates inflate their score? It all depends on your goal. Are you trying to maximize validity or minimize the impact of inflation?

According to a research study by Berry & Sackett published in the Winter '09 issue of Personnel Psychology, if your goal is to maximize validity, your best bet is to wait until applicants have taken the exam, then set your cut-score (e.g., the top two-thirds); this was particularly true when selection ratios are small (i.e., organization is very selective).

If your goal is to minimize the number of deserving applicants who are displaced by "fakers", you're better off establishing the cut point ahead of time, by using a non-applicant derived sample (e.g., job incumbents, research group). The results were generated using a Monte Carlo simulation.

Interestingly, the authors also replicated the work of other researchers who have shown that the impact of faking on the criterion-related validity of personality measures is relatively low. There are a few other very good points made in this article:

- Expert judgment methods of establishing pass points (e.g., Angoff method) may be difficult to use for personality tests since experts may find it difficult to judge individual items. Methods used to select a certain number of applicants or methods based on a criterion-related validity study (both used as variables in this study) are more appropriate for personality tests.

- There is no consensus of how prevalent faking on personality exams is; estimates range from 5-71%. It likely depends on the situation and how motivated test takers are to engage in impression management.

- Some recommend setting a very low cutoff score for personality tests, which would exclude only those likely not suitable for the position (and not faking), while others prefer a more stringent cutoff to maximize utility.

- A reasonable range of d-values for score inflation on personality inventories is .5-1.0 (used in this study).

- There exists very little research on the skewness of faking score increases. A positively-skewed distribution (meaning most people faked a small amount) was used in this study. (I would think this would also vary on the situation)

So bottom line: where--and how--you set your cutoff score on personality inventories depends on whether you want to maximize the predictive validity or minimize the number of deserving applicants that get left out of the process.

Other good reads in this issue:

- Police officer applicants reactions to promotional assessment methods

- The impact of diversity climate on retail store sales

- The construct validity of multisource performance ratings

- Labor market influences on CEO compensation

Living with T&Es

There are a number of ways to use the Internet to perform personnel assessment. Examples include timed cognitive and job knowledge tests, biodata instruments, and personality inventories. But one of the easiest--and thus most tempting to use--types of tests is known as training and experience questionnaires, or T&Es (also sometimes called E&E for education and experience).

A typical (poor) example of a T&E item might be something like this:

How much experience do you have conducing job analysis?

a) None
b) Less than 2 years
c) 2-4 years
d) 4 or more years

I could go on at length about the challenges inherent with using this type of assessment, but I'll spare you. Instead I'll point you to Jim Higgins' December 2008 newsletter, HR Rampage, in which he addresses this topic, among others (see page 2).

Jim points out several problems with this type of assessment, including the overwhelming urge to self-inflate in high-stakes scenarios (we already have problems with outright cheating), the inability of highly qualified individuals to give themselves sufficient credit, and the work required to validate responses.

So given all these challenges, what can we do to mitigate them? The solutions investigated so far (e.g., elaboration, warnings) have met with very limited success. But there are a number of tactics we can take in this situation. Here are some other methods to consider as we wait for more research in this area:

1) Accurately describe the job and requirements to prevent an unqualified individual from applying in the first place.

2) Clearly word stems and responses to avoid legitimate mis-reads.

3) Include lie items (e.g., "I have experience using the HR Tests Job Analysis Technique") and deduct points when candidates endorse them.

4) Use false bottoms (e.g., both (a) and (b) are worth zero points) and false tops (e.g., both (c) and (d) are worth the same).

5) Use scales appropriate to the item. For example, amount of experience is often the incorrect scale; type of experience is better.

6) Encourage hiring supervisors to follow up on specific items in their interview.

7) Use a friendly zero point, such as "I do not have any experience but I would be willing to learn" instead of "none." Remember there are egos involved here.

8) Ask questions that are appropriate for a T&E. Don't ask candidates to rate their oral communication skills.

9) Don't ask people to compare themselves to others (e.g., Average, Above Average). Instead use objective measures such as number of times.

10) Seriously consider weighting the items. This is of course dependent upon subject matter expert input, but it's highly likely that your SMEs consider certain training or experience areas more important than others.

11) Before they even get to the T&E, use willingness/pre-screening questionnaires that ask candidates to acknowledge they understand the less-than-perfect conditions related to the job (e.g., mandatory overtime, working outside in the heat).

12) Consider using them as a feedback tool for candidates rather than a scored instrument (e.g., "Your responses indicate you have very little experience and education related to this job. Would you like to continue to apply?").

13) Base everything on SME input. Yes, I realize this probably doesn't need to be stated, but one of the worst temptations is for HR folk to draft T&Es themselves thinking they're easy to write. This is a myth, and helps contribute to poor quality eligible lists.

This is by no means an exhaustive list, and I'm sure there are other methods out there for helping us live with T&Es. Feel free to add your suggestions!

Giving thanks for research

It's almost Thanksgiving here in the U.S., a time to give thanks, and I'd like to thank a largely unsung group of people. Thank you to all the researchers out there who try to help us put some science around the art we call personnel recruitment and selection. Thank you for all your work and insights.

What better way to celebrate this wish of thanks than by talking about a new issue of the International Journal of Selection and Assessment (v16, #4)! As usual it's chalk full of good articles, so let's take a look at some of them.

First, a study of applicant perceptions of credit checks, something many of us do for sensitive positions. Using samples of undergraduates, Kuhn and Nielsen found mostly negative reactions, especially for older participants, but they varied with the explanation given as well as privacy expectations. Worth a look for any of you that conduct large numbers of background checks (and if you do, don't miss the Oppler et al. study below).

Next up, a fascinating study of police officer selection in the Netherlands. Using data from over 3,000 applicants, De Meijer et al. found evidence for differential validity between ethnic majority and minority participants. Specifically, cognitive ability tests predicted training performance for minorities but not for those in the majority. Performance prediction for the latter group was low for cognitive ability tests and somewhat better using non-cognitive ability variables. By the way, the dissertation of the primary author, a fascinating look at similar issues, can be found here.

The third article is one of those articles that almost (...almost) makes me want to pay for it, and anybody interested in electronic applicant issues take note. In this study, Dunleavy et al. used simulations to show the tremendous impact that small numbers of applicants can have on adverse impact (AI) analysis. In fact, the authors reveal situations where AI can be caused or masked by a single applicant applying multiple times! The authors present ways of identifying and handling these cases. Scary stuff. Hope the OFCCP is reading.

Fourth, Lievens and Peeters present results of a study of elaboration and its impact on faking situational judgment tests. Using master students, the researchers found that requiring elaboration on items (i.e., the reason they chose the response) had several positive results. It reduced faking on items with high familiarity. It also reduced the percentage of "fakers" in the top of the distribution. Lastly, candidates reported that the elaboration allowed them to better demonstrate their KSAs. This could be a great strategy for those of you worried about the inflation effects of administering SJTs online.

Next, Furnham et al. with a study of assessment center ratings. The authors found that expert ratings of "personal assertiveness", "toughness and determination", and "curiosity" were significantly correlated with participant personality scores, particularly Extraversion. Correlations with intelligence test scores were low.

Last but definitely not least, Oppler et al. discuss results of a rare empirical study of financial history and its relationship to counterproductive work behaviors (CWBs). Using a "random sample of 2519 employees" the authors found that those with financial history "concerns" were significantly more likely to demonstrate CWBs after hire. Great support for conducting these types of checks.

There are other articles in here, so I encourage you to check them all out. Thank goodness for research!