Big research update

It's been a while since I provided a research update, so let's take a look at some recent highlights:

The March 2017 issue of the International Journal of Selection and Assessment (IJSA) (free right now!):

•  Some support for individual psychological assessment in a sample of middle- and senior-level civil service managers in the UK
• Concerns about personality measures that are non-contextualized (i.e., not tailored to the job)
• A study of the personality traits of high potentials (conscientiousness came out #1)
• Support for the equivalence of GMA tests on multiple devices, including mobile

The June IJSA:

• A fascinating twin study that found support for a strong relationship between counterproductive work behaviors in personal life and work.
• Applicants higher in conscientiousness and emotional stability may be more likely to re-test and improve their GMA scores
• A study of undergrads suggesting that Big 5 score results from internet-based tests and in-person tests are equivalent
• You may be able to decrease the number of candidates that withdraw from selection by engaging them more fully and ensuring your processes are fair
• Other-reports of personality appear to be fakable, but less so than self-reports

Vol 2(1) of Personnel Assessment and Decisions:

• Trusting, intelligent raters may be more able to pick up on impression management during interviews
• A study using the limited data available suggests applicants prefer in-person over technology-mediated interviews; interview ratings were also lower. Beware varying interview mode across applicants!
• A study of unproctored internet cognitive ability tests suggests cheating is common, and the researchers strongly suggest implementing technological precautions
• Hiring managers may resist standardized testing in part because they feel it reduces their usefulness/impact
• Free personality test items that can be used and mapped to the PPRF, a personality-based job analysis tool

April Psychological Bulletin:

• A meta-analysis showing that reaction time measures can be useful in detecting deception, but are themselves susceptible to counter-measures

March Journal of Applied Psychology:

•  A review of 100 years of selection and recruitment research in this journal

May Journal of Applied Psychology:

• A study of a large group of applicants suggests that recruiting candidates earlier (e.g., high school), and with more diagnostic methods, can influence their investment in their career and make them better candidates
• Using several computer simulations, researchers found that incumbent-based estimates of range restriction may be downwardly biasing criterion-related validity values, and can produce evidence of differential validity were none exists

March Journal of Organizational Behavior:

• The Big 5 factors of neuroticism, conscientiousness, and agreeableness may prevent supervisors from turning frustration into abusive supervision

May Journal of Organizational Behavior:

• Big age differences between subordinates and supervisors can negatively impact firm performance—if they're discussed

June Journal of Business and Psychology:

• Is the KSAO of communicating via computer just another version of face-to-face communication? According to this study, they are distinct.
• Do millennials have a worse work ethic than older employees? It appears not based on this meta-analysis

That's it for now!

Mini research update

Without further ado, a few research updates:

January 2017 Psychological Bulletin:

- Why is there better gender balance in some STEM fields than others?  Possible reasons these authors cite include masculine cultures, lack of early experience, and lower self-efficacy.

- There appears to be a strong relationship between emotional intelligence and other personality factors, particularly trait EI.


January 2017 Journal of Applied Psychology:

- Sackett & Lievens argue that a modular perspective on selection procedures --in other words, breaking selection procedures down into their design components such as response format -- allows for insights beyond a holistic view.  Read this quite fascinating article here.


Reminder: the Journal of Personnel Assessment and Decisions is free, available here.

Research update

Several research updates this time, including fascinating studies of how computers can assist with interview training, and how a brief writing exercise can lower stereotype threat for women:


International Journal of Selection and Assessment, December, 2016:

- International support for the cultural intelligence scale

- Looking to improve applicant interview performance? Maybe a computer can help.

- This study found that time lag and g-loading are important factors impacting re-testing results

- Status-seeking seems to be an important individual difference when looking at self-presentation behaviors, including exaggeration and faking in job search

- Development and validation of a 360-degree measure of leadership personality


Personnel Psychology, Winter 2016:

- Do CEOs significantly impact firm performance?  This study found evidence that they do.

- More evidence that the assumption that performance is normally distributed is questionable

- A more accurate correction for range restriction is presented, and an example analysis indicates the relationship between the Big 5 and job satisfaction may be greater than previously believed



Journal of Organizational Behavior, November, 2016 (which includes several articles devoted to the importance of theory in organizational sciences):

- Are self-focused or other-focused recruiting advertisements more effective?  This study suggests it depends not only on cultural differences but individual regulatory focus



Journal of Applied Psychology, October, 2016:

- A suggestion for improving meta-analytic structural equation modeling



Journal of Applied Psychology, November, 2016:

- A fascinating study of how having women compose a brief written description of their personal values can help ameliorate stereotype threat in competitive environments

Research update

Several recent journals to pore through:

July 2016 issue of Journal of Organizational Behavior

July 2016 issue of Journal Applied Psychology, including:

- Computer scoring of candidate essays

- Everything you've been told about assessment center ratings is confounded

and the biggie...

September 2016 issue of International Journal of Selection and Assessment, including:

- Faking on assessment measures

- Emotional intelligence and service jobs

- A self-report measure of psychopathy in the workplace

Research update

A few new journal issues have come out lately:

Summer 2016 Personnel Psychology, including:

Transparency of Assessment Centers: Low Criterion-related Validity but Greater Opportunity to Perform?

May 2016 Journal of Applied Psychology, including:

Initial impressions: What they are, what they are not, and how they influence structured interview outcomes.

Racioethnicity, community makeup, and potential employees’ reactions to organizational diversity management approaches.

June 2016 International Journal of Selection and Assessment, including:

Applicant Reactions to Selection Events: Four studies into the role of attributional style and fairness perceptions

Behavioral Cues as Indicators of Deception in Structured Employment Interviews

The Role of Self-focused Attention and Negative Self-thought in Interview Anxiety: A test of two interventions

The Influence of Candidate Social Effectiveness on Assessment Center Performance Ratings: A field study

Discrimination due to Ethnicity and Gender: How susceptible are video-based job interviews?

A Comparison of General and Work-specific Personality Measures as Predictors of Organizational Citizenship Behavior

The Perceived Nature and Incidence of Dysfunctional Assessment Center Features and Processes

Who is Being Judged Promotable: Good actors, high performers, highly committed or birds of a feather?

New journal issues

Two new journal issues to make you aware of:

International Journal of Selection and Assessment - March 2016

Unintended Consequences of Transparency During Personnel Selection: Benefitting some  candidates, but harming others?

Ethnic Differences in Perceptions of Cognitive Ability Tests: The explanatory role of self-serving attributions

Conditional Reasoning Test for Aggression: Further evidence about incremental validity

For Love or for Money: Intrinsic and extrinsic value congruence in recruitment

Social Influences in Recruitment: When is word-of-mouth most effective?

Highlighting Tensions in Recruitment and Selection Research and Practice

Tests of Integrity, HEXACO Personality, and General Mental Ability, as Predictors of Integrity Ratings in the Royal Dutch Military Police

Training Affects Variability in Training Performance Both Within and Across Jobs

Examining Applicant Reactions to Different Media Types in Character-based Simulations for Employee Selection

When Will Interviewers Be Willing to Use High-structured Job Interviews? The role of personality


Journal of Applied Psychology - March 2016


How and why do interviewers try to make impressions on applicants? A qualitative study.

The long road to employment: Incivility experienced by job seekers.

The role of self-determined motivation in job search: A dynamic approach.

2014 Research of the Year (+ research update)

Happy New Year!  As I've done in previous years, I present below the research articles I ran across in this area that I think were the most impactful and/or important of 2014.  But first, let's catch up on two issues:

First, the Winter issue of Personnel Psychology:

- Situational judgment tests have been shown to be useful for measuring interpersonal skills, but beware: levels of "angry hostility" moderate that relationship.  (Is there a happy hostility?)

- When hiring leaders, should you look for those that have a busy home life, or be wary of them?  In this fascinating study, the authors found that leader family-to-work conflict negatively impacts followers in that it can increase their burnout. However, family-to-work enrichment increased follower engagement through leader engagement.  So the answer is, as usual, not simple: home/family life can be a good thing for followers if it makes the leader more engaged; but if the home/family life is increasing burnout, the leader may pass that along to others.  So it would seem it all depends on how the individual is handling their life outside of work!

Let's look at the November issue of the Journal of Applied Psychology:

- Are men or women perceived as better leaders?  According to this meta-analysis, it depends on how you ask the question.  If you limit the question to other-ratings, women are rated significantly higher.  But if you look at self-ratings, men rate themselves significantly higher.  Which leads to the next question: is it a biological perception or a gender perception, and if the latter, what traits are the most important?

- An intriguing study of how applicant confidence interacts with and can be altered by the recruitment experience, in this case among recruits for the U.S. military.

- Next, a study of employment and job search efficacy.  Not surprisingly, within-person frequency of job search behavior correlated with job offers; interestingly, the relationship between perceived job search progress and efficacy beliefs were moderated by beliefs of internal attribution.

- Last but not least, more evidence of the importance of defining the criteria when predicting job performance.  In this meta-analysis, the researchers found more support for personality traits out-predicting cognitive ability in predicting counterproductive work behavior, that the two predictors are approximately equal in predicting organizational citizenship behaviors, and that cognitive ability outperforms personality when predicting task and overall performance.  So do you want high task performance, OCBs, or do you want to avoid CWB?  :)  (of course the situation is even more complicated depending on whether you're looking at individual, team, leader performance, over what period of time, etc.)


Okay, on to the awards!  Without further ado, here are my nominations for Research of the Year for 2014:

1) Important advancements in our understanding of weight-based discrimination at work: Vanhove & Gordon.

2) A study of applicants posting faux pas on their social networking sites: Roulin.

3) Two important looks at assessments delivered remotely via mobile devices: Arthur, Doverspike, Munoz, Taylor, & Carr, and Morelli, Mahan, & Illingworth.

4) Two fascinating looks at personality at work: Judge, Simon, Hurst, & Kelley; and Wille & De Fruyt

5) An excellent study of how effective staffing and training practices impact firm-level flexibility and adaptability: Kim & Ployhart.

6) An important study of the movement of impactful I/O researchers to business schools: Aguinis, Bradley, & Brodersen.

7) The relationship between conscientiousness and job performance is more accurately described as curvilinear: Carter, Dalal, Boyce, O'Connell, Kung & Delgado

Finally, honorable mention to two great developments in 2014:  the change of some publishers to making access to articles more affordable, and the announcement of an additional journal, the Journal of Personnel Assessment and Decisions.


I'm continually amazed at the quality of thought and research in our area and the passion and practicality you exhibit.  Here's to an amazing 2014 and more in 2015!

Just kidding...more research update!

Seriously?  Just yesterday I did my research update, ending with a note that the December 2014 issue of the International Journal of Selection and Assessment should be out soon.

Guess what?  It came out today.

So that means--you guessed it--another research update!  :)

- First, a test of Spearman's hypothesis, which states that the magnitude of White-Black mean differences on tests of cognitive ability vary with the test's g loading.  Using a large sample of GATB test-takers, these authors found support for Spearman's hypothesis, and that reducing g saturation lowered validity and increased prediction errors.

So does that mean practitioners have to choose between high-validity tests of ability or increasing the diversity of their candidate pool?  Not so fast.  Remember...there are other options.

- Next, international (Croatian) support for the Conditional Reasoning Test of Aggression, which can be used to predict counterproductive work behaviors.  I can see this increasingly being something employers are interested in.

- Applicants that do well on tests have favorable impressions of them, while those that do poorly don't like them.  Right?  Not necessarily.  These researchers found that above and beyond how people actually did on a test, certain individual differences predict applicant reactions, and suggest these be taken into account when designing assessments.

- Although personality testing continues to be one of the most popular topics, concerns remain about applicants "faking" their responses (i.e., trying to game the test by responding inaccurately but hopefully increase the chances of obtaining the job).  This study investigates the use of blatant extreme responding, consistently selecting the highest or lowest response option, to detect faking, and looked at how this behavior correlated with cognitive ability, other measures of faking, and demographic factors (level of job, race, and gender).

- Next, a study of assessment center practices in Indonesia.

- Do individuals high in neuroticism have higher or lower job performance?  Many would guess lower performance, but according to this research, the impact of neuroticism on job performance is moderated by job characteristics.  This supports the more nuanced view that the relationship between personality traits and performance is in many cases non-linear and depends on how performance is conceptualized.

- ...which leads oh so nicely into the next article!  In it, the authors studied air traffic controllers and found results consistent with previous studies--ability primarily predicted task performance while personality better predicted citizenship behavior.  Which raises an interesting question: which version of "performance" are you interested in?  My guess is for many employers the answer is both--which suggests of course using multiple methods when assessing candidates.

- Last but not least, an important study of using cognitive ability and personality to predict job performance in a three studies of Chilean organizations.  Results were consistent with studies conducted elsewhere, namely ability and personality significantly predicted performance.

Okay, I think that's it for now!

Research update

Okay, so it been a couple months, huh?  Well, what say we do a research update then.

But before I dive in, I discovered something interesting and important.  Longtime readers know that one of my biggest pet peeves is how difficult research articles are to get a hold of.  And by difficult I mean expensive.  Historically, unless you were affiliated with a research institution or were a subscriber, you had to pay exorbitant (IMHO) fees to see research articles.  So imagine my pleasure when I discovered that at least one publisher--Wiley, who publishes several of the research journals in this area--now allows you to read-access for an article for as low as $6.  Now that's only for 48 hours and you can't print it, but hey--that's a heck of a lot better than something like $30-40, which historically has been the case!  So kudos.

Moving on.

Let's start with a bang with an article from the Autumn 2014 issue of Personnel Psych.  A few years back several researchers argued that the assumption that performance is distributed normally was incorrect; and it got a bit of press.  Not so fast, say new researchers, who show that when defined properly, performance is in fact more normally distributed.

For those of you wondering, "why do I care?"  Whether we believe performance is normally distributed or not significantly impacts not only the statistical analyses performed on selection mechanisms but theories and practices surrounding HRM.


Moving to the July issue of the Journal of Applied Psychology:

- If you're going to use a cognitively-loaded selection mechanism (which in many cases has some of the highest predictive validity), be prepared to accept high levels of adverse impact.  Right?  Not to fast, say these researchers, who show that by weighting the subtests, you can increase diversity decisions without sacrifice validity.

- Here's another good one.  As you probably know, the personality trait of conscientiousness has shown value in predicting performance in certain occupations.  Many believe that conscientiousness may in fact have a curvilinear relationship with performance (meaning after a certain point, more conscientiousness may not help)--but this theory has not been consistently supported.  According to these researchers, this may have to do with the assumption that higher scores equal more conscientiousness.  In fact, when using an "ideal point" model, results were incredibly consistent in terms of supporting the curvilinear relationship between conscientiousness and performance.

- Range restriction is a common problem in applied selection research, since you only have performance data on a subset of the test-takers, requiring us to draw inferences.   A few years back, Hunter, Schmidt, and Le proposed a new correction for range restriction that requires less information.  But is it in fact superior?  According to this research, the general answer appears to be: yes.


Let's move to the September issue of JAP:

- Within-person variance of performance is an important concept, both conceptually and practically.  Historically short-term and long-term performance variance have been treated separately, but these researchers show the advantage of integrating the two together.

- Next, a fascinating study of the choice of (and persistence in) STEM fields as a career, the importance of both interest and ability, and how gender plays an important role.  In a nutshell, as I understand it, interest and ability seem to play a more important role in predicting STEM career choices for men than for women, whereas ability is more important in the persistence in STEM careers for women.


Let's take a look at a couple from recent issue of Personnel Review:

- From volume 43(5), these researchers found support for ethics-based hiring decisions resulting in improved work attitudes, include organizational commitment.

- From 43(6), an expanded conceptual model of how hiring supervisors perceive "overqualification", which I would love to see more research on.


Last but not least, for you stats folks, what's new from PARE?

- What happens when you have missing data on multiple variables?

- Equivalence testing: samples matter!

- What sample size is needed when using regression models?  Here's one suggestion on how to figure it out.


The December 2014 issue of IJSA should be out relatively soon, so look for a post on that soon!

Research update

I can't believe it's been three months since a research update.  I was waiting until I got critical mass, and with the release of the September issues of IJSA, I think I've hit it.

So let's start there:

- Experimenting with using different rating scales on SJTs (with "best and worst" response format doing the best of the traditional scales)

- Aspects of a semi-structured interview added incremental validity over cognitive ability in predicting training performance

- Studying the use of preselection methods (e.g., work experience) prior to assessment centers in German companies

- The proposed general factor of personality may be useful in selection contexts (this one was a military setting)

- Evidence that effective leaders show creativity and political skill

- Investigating the relationship (using survey data) between personality facets and CWBs (with emotional stability playing a key role)

- Corrections for indirect range restriction boosted the upper end of structured interview validity substantially

- A method of increasing the precision of simulations that analyze group mean differences and adverse impact

- A very useful study that looked at the prediction of voluntary turnover as well as performance using biodata and other applicant information, including recruitment source, among a sample of call center applicants.  Reuslts?  Individuals who had previously applied, chose to submit additional information, were employed, or were referrals had significantly less voluntary turnover.



Moving on...let's check out the May issue of JAP; there are only two articles but both worth looking at:

- First, a fascinating study of the firm-level impact of effective staffing and training, suggesting that the former allow organizations greater flexibility and adaptability (e.g., to changing financial conditions).

- Second, another study of SJT response formats.  The researchers found, using a very large sample, the "rate" format (e.g., "rate each of the following options in terms of effectiveness") to be superior in terms of validity, reliability, and group differences.


Next, the July issue of JOB, which is devoted to leadership:

- You might want to check out this overview/critique of the various leadership theories.

- This study suggests that newer models proposing morality as an important component of leadership success have methodological flaws.

- Last, a study of why Whites oppose affirmative action programs

Let's move to the September issue of Industrial and Organizational Psychology:

- The first focal article discusses the increasing movement of I/O psychology to business schools.  The authors found evidence that this is due in large part to some of the most active and influential I/O researchers moving to business schools.

- The second is about stereotype threat--specifically its importance as a psychological construct and the paucity of applied research about it.


Coming into the home stretch, the Summer issue of Personnel Psych:

- The distribution of individual performance may not be normal if, as these researchers suggest, "star performers" have emerged

- Executives with high levels of conscientiousness and who display transformational leadership behavior may directly contribute to organizational performance

Rounding out my review, check out a few recent articles from PARE:

- I'm not even gonna attempt to summarize this, so here's the title: Multiple-Group confirmatory factor analysis in R – A tutorial in measurement invariance with continuous and ordinal indicators

- Improving exploratory factor analysis for ordinal data

- Improving multidimensional adaptive testing


Last but not least, it's not related to recruitment or assessment, but check out this study that found productivity increases during bad weather :)

That's all folks!

Mobile assessment comes of age + research update

The idea of administering employment tests on mobile devices is not new.  But serious research into it is in its infancy.  This is to be expected for at least two reasons: (1) historically it has taken a while with new technologies to have enough data to analyze (although this is changing), and (2) it takes a while for researchers to get through the arcaneness of publishing (this, to my knowledge, isn't changing, but please prove me wrong).

Readers interested in the topic have benefited from articles elsewhere, but we're finally at a point where good research is being published on this topic.  Case in point: the June issue of the International Journal of Selection and Assessment.

The first article on this topic in this issue, by Arthur, Doverspike, Munoz, Taylor, & Carr, studied data from over 3.5 million applicants who completed unproctored internet-based tests (UIT) over a 14-month period.  And while the percentage that completed them on mobile devices was small (2%), it still yielded data on nearly 70,000 applicants.

Results?  Some in line with research you may have seen before, but some may surprise you:

- Mobile devices were (slightly) more likely to be used by women, African-Americans and Hispanics, and younger applicants.  (Think about that for a minute!)

- Scores on a personality inventory were similar across platforms.

- Scores on a cognitive ability test were lower for those using mobile devices.  Without access to the entire article, I can only speculate on proffered reasons, but it's interesting to think about whether this is a reflection of the applicants or the platform.

- Tests of measurement invariance found equivalence across platforms (which basically means the same thing(s) appeared to be measured).

So overall, in terms of using UITs, I think this is promising in terms of including a mobile component.


The next article, by Morelli, Mahan, and Illingworth, also looked at measurement variance of mobile versus non-mobile (i.e., PC-delivered) internet-based tests, with respect to four types of assessment: cognitive ability, biodata, a multimedia work simulation, and a text-based situational judgment test.  Data was gathered from nearly 600,000 test-takers in the hospitality industry who were applying for maintenance and customer-facing jobs in 2011 and 2012 (note the different job types).  Nearly 25,000 of these applicants took the assessment on mobile devices.

Results?  The two types of administrations appeared be equivalent in terms of what they were measuring.  However, interestingly, mobile test-takers did worse on the SJT portion.  The authors reasonably hypothesize this may be due to the nature of the SJT and the amount of attention it may have required compared to the other test types.  (btw this article appears to be based on Morelli's dissertation, which can be found here--it's a treasure trove of information on the topic)

Again, overall these are promising results for establishing the measurement equivalence of mobile assessments.  What does this all mean?  It suggests that unproctored tests delivered using mobile devices are measuring the same things as tests delivered using more traditional internet-based methods.  It also looks like fakability or inflation may be a non-issue (compared to traditional UIT).  This preliminary research means researchers and practitioners should be more confident that mobile assessments can be used meaningfully.

I agree with others that this is only the beginning.  In our mobile and app-reliant world, we're only scratching the surface not only in terms of research but in terms of what can be done to measure competencies in new--and frankly more interesting--ways.  Not to mention all the interesting (and important) associated research questions:

- Do natively developed apps differ in measurement properties--and potential--compared to more traditional assessments simply delivered over mobile?

- How does assessment delivery model interact with job type?  (e.g., may be more appropriate for some, may be better than traditional methods for others)

- What competencies should test developers be looking for when hiring?  (e.g., should they be hiring game developers?)

- What do popular apps, such as Facebook (usage) and Candy Crush (score), measure--if anything?

- Oh, and how about: does mobile assessment impact criterion-related validity?


Lest you think I've forgotten the rest of this excellent issue...

- Maclver, et al. introduce the concept of user validity, which uses test-taker perceptions to focus on ways we can improve assessments, score interpretation, and the provision of test feedback.

- Bing, et al. provide more evidence that contextualizing personality inventory items (i.e., wording the items so they more closely match the purpose/situation) improves the prediction of job performance--beyond noncontexual measures of the same traits.

- On the other hand, Holtrop, et al. take things a step further and look at different methods of contextualization.  Interestingly, this study of 139 pharmacy assistants found a decrease in validity compared to a "generic" personality inventory!

- This study by Ioannis Nikolaou in Greece of social networking websites (SNWs) that found job seekers still using job boards more than SNWs, that SNWs may be particularly effective for passive candidates (!), and that HR professionals found LinkedIn to be more effective than Facebook.

- An important study of applicant withdrawal behavior by Brock Baskin, et al., that found withdrawal tied primarily to obstructions (e.g., distance to test facility) rather than minority differences in perception.

- A study of Black-White differences on a measure of emotional intelligence by Whitman, et al., that found (N=334) Blacks had higher face validity perceptions of the measure, but Whites performed significantly better.

- Last, a study by Vecchione that compared the fakability of implicit personality measures to explicit personality measures.  Implicit measures are somewhat "hidden" in that they measure attitudes or characteristics using perceptual speed or other tools to discover your typical thought patterns; you may be familiar with project implicit, which has gotten some media coverage.  Explicit measures are, as the name implies, more obvious items--in this case, about personality aspects.  In this study of a relatively small number of security guards and semiskilled workers, the researchers found the implicit measure to be superior in terms of fakability resistance.  (I wonder how the test-takers felt?)


That's it for this excellent issue of IJSA, but in the last few months we also got some more great research care of the March issue of the Journal of Applied Psychology:

- An important (but small N) within-subjects study by Judge, et al. of the stability of personality at work.  They found that while traits exhibited stability across time, there were also deviations that were explained by work experiences such as interpersonal conflict, which has interesting implications for work behavior as well as measurement.  In addition, the authors found that individuals high in neuroticism exhibited more variation in traits over time compared to those who were more emotionally stable.  You can find an in press version here; it's worth a read, particularly the section beginning on page 47 on practical implications.

- Smith-Crowe, et al. present a set of guidelines for researchers and practitioners looking to draw conclusions from tests of interrater agreement that may assume conditions that are rarely true.

- Another interesting one: Wille & De Fruyt investigate the reciprocal relationship between personality and work.  The researchers found that while personality shapes occupational experiences, the relationship works in both directions and work can become an important source of identity.

- Here's one for you assessment center fans: this study by Speer, et al. adds to the picture through findings that ratings taken from exercises with dissimilar demands actually had higher criterion-related validity than ratings taken from similar exercises!

- Last but not least, presenting research findings in a way that is understandable to non-researchers poses an ongoing--and important--challenge.  Brooks et al. present results of their study that found non-traditional effect size indicators (e.g., a common language effect size indicator) were perceived as more understandable and useful when communicating results of an intervention.  Those of you that have trained or consulted for any length of time know how important it is to turn correlations into dollars or time (or both)!

That's it for now!

March '14 IJSA

In my last research update just a couple days ago, I mentioned that the new issue of IJSA should be coming out soon.

I think they heard me because it came out literally the next day.

So let's take a look:

- This study adds to our (relatively little) knowledge of sensitivity reviews of test items and finds much room for improvement

- More evidence that the utility of UIT isn't eliminated by cheating, this time with a speeded ability test

- Applicant motivation may be impacted by the intended scoring mechanism (e.g., objective vs. ratings).

- The validity of work experience in predicting performance is much debated*, but this study found support for it among salespersons, with personality also playing a moderating role.

- A study of the moderating effect of "good impression" responding on personality inventories

- This review provides a great addition to our knowledge of in-baskets (a related presentation can be found through IPAC)

- Another excellent addition, this time a study of faux pas on social networking websites in the context of employer assessment

- According to this study, assessors may adjust their decision strategy for immigrants (non-native language speakers)

- Letters of recommendation, in this study of nonmedical medical school graduate students, provided helpful information in predicting degree attainment

- Interactive multimedia simulations are here to stay, and this study adds to our confidence that these types of assessments can work well

Until next time!

* Don't forget to check out the U.S. MSPB's latest research study on T&Es!

Research update

Okay, past time for another research update, so let's catch up!

Let's start with the Journal of Applied Social Psychology (Dec-Feb):

- Cultural intelligence plays a key role in multicultural teams

- Theory of Planned Behavior can be used to explain intent to submit video resumes

- More on weight-based discrimination, including additional evidence that this occurs more among women (free right now!)

- Does the physical attractiveness bias hold in same-sex evaluative situations?  Not so much, although it may depend on someone's social comparison orientation

- "Dark side" traits play a role in predicting career preference

- Evidence that efficacy beliefs play a significant role not only in individual performance, but in team performance


Next up, the January issue of JAP:

- The concept of differential validity among ethnic groups in cognitive ability testing has been much debated, and this study adds to the discussion by suggesting that the effects are largely artifactual due to range restriction.

- Or are they?  This study on the same topic found that range restriction could not account for observed differential validity findings.  So the debate continues...

- A suggestion for how to increase the salience of dimension ratings in assessment centers

- Ambition and emotional stability appear related to adaptive performance, particularly for managers


Spring Personnel Psych (free right now!)

- First, a fascinating study of P-E fit across various cultures.  Turns out relational fit may be more important in collectivistic and high power distance cultures (e.g., East Asia), whereas rational fit may be more important in individualistic and lower power distance cultures (e.g., the U.S.).

- Next, a study of recruitment messaging for job involving international travel.

- Last but definitely not least, a narrative and quantitative extensive review of the structured interview


Not quite done: One from Psychological Science on statistical power in testing mediation, and just in case you needed more evidence, the Nov/Dec issue of HRM has several research articles supporting the importance of line manager behavior and HRM practices on things like employee engagement.

The Spring issue of IJSA should be out soon, so see ya soon!

Top 10 Assessment and Recruitment Research of 2013

So I'm gonna try something a little different this year.  I'm going to present the research from 2013 that I think has the best chance of fundamentally changing research directions, has the biggest implication for practice, or is just plain interesting.

So without further ado, and in no particular order, here are my choices for the Best Assessment and Recruitment Research of 2013:

1) Murphy, et al. call into question one of the fundamental assumptions of test development: that judgments of subject matter experts have a direct relationship to test utility.

2) Kim, et al. demonstrate a real value of age diversity in work groups: better emotional regulation.

3) Ghumman and Barnes with a simple but elegant study that demonstrated how important sleep is in preventing a persistent thorn: prejudicial assessments by raters.

4) Konradt, et al. showed that perceptions of fairness matter in web-based assessments too.

5) Personality research continued to dominate in 2013, and one of the best studies was by Shaffer and Postlewaite.  In it, they demonstrate that conscientiousness is best used as a predictor of performance in highly routinized jobs.

6) Mrazek et al. focused on a topic near and dear to my heart: mindfulness.  They showed that training in mindfulness increased GRE scores.  The implication for employment testing is clear, we just need more research in that direction.

7) Early in the year, Bobko and Roth gave us one of those "I better print this out" articles, showing that assessment methods historically assumed to result in lower levels of adverse impact, like biodata and work samples, may be more prone to d than we thought.  Side note: this article is still free in its entirety!

8) Kuhn, et al. presented the results of an elegantly simple experiment illustrating that the impact of a minor resume embellishment depended on the pre-existing perception of the applicant.

9) Discrimination, sadly, knows no demographic boundaries.  In this study, Conley found that non-Whites described White women as attractive, blonde, ditsy, shallow, privileged, sexually available, and appearance focused.

10) In a study of M.B.A. program admission judgments that has implications for employment selection, Simonsohn and Gino found that as the day progressed, fewer applicants were rated as highly recommended if several were recommended earlier in the day.

There were, of course, many other well-done and interesting studies in 2013, but these were some of my favorites.  Here's to a productive, stimulating, and successful 2014!

Research update

Well, it's that time of year again.  No, not the holidays.  No, not winter (or summer, depending on where you are!).  Research update time!  And I think you will agree with me that there is a lot of interesting research being reported, on traditional topics as well as emerging ones.

First, the November issue of JOB:

- Do transformational leaders increase creative performance and the display of OCBs?  Well, that may depend on how much trait affectivity they had to begin with. A reminder to not make blanket statements like "X type of leadership causes Y type of behavior."

- There is seemingly endless debate about the utility of personality inventories.  This study reminds us--again--that in assessment research there are few simple answers.  The authors describe how a particular combination of personality measures correlated with task performance among professional employees, but not non-professionals.  (yes, I said task performance)


Next, the Winter issue of Personnel Psychology (free right now!), much of which is devoted to corporate social responsibility (CSR):

- Do perceptions of CSR drive job pursuit intentions?  It may depend on the applicant's previous justice experiences and their moral identity.

- Oh, and it may also depend on the extent to which applicants desire to have an impact through their work.

- There is a debate in the assessment center literature about whether competency dimensions are being measured or if it's purely a function of the assessment type.  This study suggests that previous research has been hamstrung by a methodological artifact and that measured properly, assessment centers do in fact assess dimensions.


Let's switch to the November issue of the Journal of Applied Social Psychology:

- Engagement is all the rage, having seemingly displaced the age-old concept of job satisfaction (we'll see).  This study reminds us that personality plays an important role in predicting engagement (so by extension our ability to increase engagement may be bounded).

- Here's another good one and it's related to internal motivations.  The authors developed an instrument that helps organizations measure the "perception of the extant motivational climate."  What does that mean?  As I understand it, it's essentially whether most people are judging their performance against their peers or their own internal standards.  It seems the latter may result in better results, such as less burnout.

- On to something more closely tied to assessment: letters of recommendation (LORs).  There's surprisingly little research on these, but this study adds to our knowledge by suggesting that gender and racial bias can occur in their review, but requiring a more thorough review of them may reduce this (I don't know how likely this is for the average supervisor).

- Finally, a study looking at the evaluation of job applicants who voluntarily interrupted their college attendance.  Unfortunately this does not appear to have been perceived as a good thing, and the researchers found a gender bias such that women with interrupted attendance had the lowest evaluations.


Next, the November issue of Industrial and Organizational Psychology, where the second focal article focus on eradicating employment discrimination.  This article looks pretty juicy.  I haven't received this one yet in the mail, so I may have more to say after digesting it.  There are, as always, several commentaries following the focal article, on topics including background checks, childhood differences, and social networks.


Okay, let's tackle the 800-pound gorilla: the December issue of IJSA:

- Are true scores and construct scores the same?  According to this Monte Carlo study, it seems how the scales were constructed makes a difference.

- Can non-native accents impact the evaluation of job applicants?  Sure seems that way according to this study.  But the effect was mediated by similarity, interpersonal attraction, and understandability.

- Here's a fascinating one.  A study of applicants for border rangers in the Norwegian Armed Forces showed that psychological hardiness--particularly commitment--predicted completion of a rigorous physical activity above and beyond physical fitness, nutrition, and sensation seeking.

- Psst....recruiters...make sure when you're selling your organization you stay positive.

- Spatial ability.  It's a classic KSA that's been studied for a long time, for various reasons including its tie to military assessments and the finding that measures can result in sex differences.  But not so fast, spatial ability is not a unitary concept.

- Another study of assessment centers, this time in Russia and using a consensus scoring model.

- And let's round it out with one that should rock some worlds: the authors presents results that suggest that subject matter expert judgment of ability/competency importance bore little relation to test validity!  Okay, I'm really curious about what the authors say about the implications, so if anyone reads this one, let us know!


Last but not least, the November issue of the Journal of Applied Psychology:

- Another on personality testing, this one underlining the important distinction between broad and narrow traits.  This is another article I'm very curious about.

- Here's on one leadership: specifically, on the impact of different power distance values between leader and subordinates on team effectiveness

- And another on nonnative speakers!  This one found discriminatory judgments made against nonnative speakers applying for middle management positions as well as venture funding.  Interestingly, it appears to be fully mediated by perceptions of political skill--a topic that is hot right now.

- Okay, let's leave on a big note.  This meta-analysis found an improvement in performance prediction of 50% when a mechanical combination of assessment data was used rather than a holistic (judgment-based) method.  BOOM!  Think about that the next time a hiring supervisor derides your spreadsheet.

Until next time!

Research update: September, 2013

Okay, it's mega research update time!

First off, the September IJSA; lots of good stuff, including:

- a constructed response multimedia test for entry-level police resulted in minor ethnic group differences

- panel interviews once again prove their superiority (also: more on interview reliability)

- further analysis of the Hogan Personality Inventory with a Spanish sample

- how to applicants form impressions of person-organization fit?  This study suggests contextual factors may be more important than interview content

- circumplex traits (combinations of personality factors) may predict counterproductive work behaviors better than simple FFM scores

- speaking of CWBs, conditional reasoning tests may not be the best predictor of them

- last but not least, what looks to be a good overview of competency modeling


Next up, the September JAP:

- an interesting, large study of the impact of candidate reactions on test scores, organizational perception, and criterion-related validity

- a study of the dynamics of the job search process and the impact of efficacy and focus

- highlighting certain factors during an interview may reduce discrimination toward pregnant applicants


Next, the Autumn 2013 Personnel Psychology:

- first, an important study of self-efficacy that suggests it is a product of past performance and not necessarily a predictor of future performance (free right now!)

- second, a study indirectly on selection that suggests that age diversity in work groups leads to more emotion regulation


Let's move on to the September JASP:

- okay, this may be a bit of a stretch, but if you're considering interviewing for a position as a dentist or a lawyer, make sure you suit up

- knowledge of service encounters predicts service effectiveness (and is related to conscientiousness)

- can use of biodata instruments result in adverse impact?  This study suggests so, but also suggests that removal of problematic items has no impact on validity


Starting to wrap up, let's move to the October JOB:

- perceptions of the fairness of promotion practices is one of those "bubbling beneath the surface" issues in most organizations.  This study found that perceptions are impacted by having been promoted in the past, organizational commitment, and ego defensiveness.  Good stuff.

- do more creative sales agents produce higher sales?  Perhaps only when there is a high quality of leader-member exchange.

- is validity generalization overgeneralized? (say that five times fast)  These folks seem to think so.


In the home stretch, from the September Psychological Science:

- older employees may have lower average cognitive performance, but it's more consistent

- spatial ability has a valuable role to play in the development of creativity, and can predict things like patents and publications


Second to last, for you stats geeks out there, a study that suggests that t-tests can be used reliably with small samples, thank you very much


Finally, something that has nothing to do with selection but is a nominee for the 2013 HR Tests Coolest Study Award, and something we all are very familiar with: time bandits (no, not the movie).

Another (smaller) research update

Seems like this always happens, I post a research update and then Personnel Psychology comes out with their latest issue.

Oh well!  At least you won't have to read as much this time  And while we're at it, let's take a look at the latest Journal of Applied Social Psychology.  But first, Personnel Psych:

- The relationship between task performance and citizenship behavior: more complicated than you might think

- Propensity scoring: a statistical technique that may improve our ability to make causal inferences from quasi-experimental designs

- Can creative employees drive higher organizational performance?  Well that seems to depend on how open to risk your organization is...

- It's easier said than done, but those looking for a job should try to stay positive: it influences how quickly you're likely to land your next position

- Last but not least, a small correction to a study posted earlier on the advantage of contextualizing personality inventory items


Okay, on to June 2013 JASP, which is a special issue on prejudice:

-Trying to reduce bias in your selection process? Make sure your raters get enough sleep.

- Discrimination against people based on their weight: it starts early!  (there's another study on weight discrimination that looked at its impact on perceived social status)

- Discrimination against men perceived to be Muslim increased among Western observers when subjects were presented in traditional dress.

- Those of Black/White biracial descent were less likely to be perceived as minorities and thus less appropriate receipts of affirmative action

- Okay, I like this one in particular, and it happens to be the last one I'll describe.  Researchers showed that competency perceptions of an African-American man hired varied depending on whether it was under "affirmative action" or a "diversity initiative"--the latter resulting in higher ratings.  This was particularly true for conservative, White raters.  Just goes to show how important words are, as any political consultant will tell you.

Fin.

Research update

Okay, I'm just gonna say it.  There is an insane amount of research out there directly or indirectly related to our field.  I'll be honest, it's a little daunting thinking about reading and processing all of it.

Luckily, you have me to overly simplify it for you so you can incorrectly describe it to others.  So let's jump right in!

First up, the June issue of IJSA:

-  Fairness perceptions matter in web-based selection too!

-  The predictive validity of conscientiousness is moderated by self-enhancement

-  Speaking of conscientiousness, it's a better predictor of performance in routinized jobs than those with complex cognitive requirements

-  Psychological hardiness predicts adaptability in military leaders.  This study is awesome also for showing a negative relationship between SAT scores and West Point performance.

-  Recruiters, listen up: this study found that contingent workers converted to full-time status performed as well as referral and online hires, but not as well as those sourced internally (bookmark this one, there's a dearth of recruitment research)

- Scoring biodata: empirical keying reigns supreme over rational and quasi-rational approaches.  Quasi-rational?  Sounds like me most of the time.

- Personality and job performance: interactions are important beyond main effects


Okay, next, the May issue of JAP:

- Adaptive decision making in military leaders: both the brain and the mind are important (okay you monists out there, chill out)

- Are you a MANOVA (wo)man?  Then read this.

- Proactive leaders set more challenging goals and have higher sales performance--assuming they have the trust of their subordinates.


Next, the April issue of JOB.  Just one study, albeit interesting, in which female evaluators were less likely to recommend hiring or promoting Asian (versus White) applicants into jobs requiring social skills


Speaking of JOB, how about the May issue?

- Have you been calling for more research on calling?  Your wish is granted.

- Curious about the concept of curiosity? (okay, I'll stop)  Looks like it can be predictive of job performance above and beyond traditional cognitive and non-cognitive predictors.  I'm gonna assume it varies with job, but I think the authors are probably right that it will increase in importance over time.

- When selecting for teams that may experience crisis situations (e.g., nuclear power plant crews), consider homogeneity--not mean levels--of positive affect.

- Moderate levels of supervisory structure combined with high levels of consideration lead to lowest levels of CWB's

- The dark side of OCBs

- The light side of OCBs

- Dark side and light side working together in a picture that has nothing to do with selection


Okay, onwards and upwards: one from the April issue of JASP, on generalized self-efficacy, work-related self-efficacy, and job-related outcomes


And one from the May issue:  Sensation seeking and need for structure predict military field exercise performance


How about a little EI research?  Haven't had a lot of that lately.  Here's a piece from the May issue of Journal of Management that found emotion management ability to be a valid predictor of job performance.


Let's look in the May Psychological Science:

- Where we find a fascinating study that demonstrates assessment of profound cognitive abilities at a young age predicts outstanding contributions in many adults

- Another interesting study of how an individual responds to daily stressors (which I would posit are differentially found in various jobs) has long-term consequences for their mental health

- High schoolers with high math and verbal skills are less likely to choose STEM careers than those with math skills but moderate verbal skills

- Okay, check this one out: mindfulness training improved GRE reading comp scores as well as working memory.  Implications for personnel assessment??


Shifting to the May Psychological Bulletin:

- Rorschach, anyone?

- Believing that individuals are malleable, rather than fixed ("implicit theory") predicts goal achievement.  Implications for leaders?


Last but not least, let's hook up with our friends at PARE and see what they're up to:

- Into Excel?  Check out this article on forest plots

- Or maybe factor analysis is your thing.


That's all folks!  Hope your brain is spinning like mine!

Research update

 
Cold/wet yet?  Well sit back, have some hot tea, and let's catch up on our research...

Let's start with the biggie: the December issue of IJSA.

-  Juggling selection quality and adverse impact continues to be tricky.  The authors in this article suggest an optimum combination.

- Reflecting results from the turnover literature, this study found a relationship between perceptions of promotion practices and organizational justice and job satisfaction.

- Speaking of justice, the authors of this study found that the relationship between perceptions of distributive justice and intentions to recommend an employer were moderated by applicant affect.

- Job seekers would to well to keep up their psychological well being and self-esteem (easier said than done, right?).

- Back to fairness.  It's reasonable to think that ethnic minority applicants may not perceive video resumes well (due to the increased salience of their minority status).  But at least in this study, that assumption was not supported--although it depended on ethnic identity and language proficiency.

-  Honestly, I'm not one for the pure concept of "multi-tasking": in my experience people perform in serial, not parallel.  But that doesn't stop people from researching the predictive validity of multi-tasking assessments.

- Faking of personality inventories has been one of the hottest topics in assessment for years, but is often framed as dichotomies.  This study looks to bring some needed complexity to the issue using qualitative data.

- We all know it can be challenging to get hiring manager to give up their unstructured interviews of questionable validity.  Interestingly, according to this study, the more a hiring manager has used unstructured interviews, the less open they are to change.  I don't know if this is a causality issue, a mediating variable issue, or more evidence of the inability to accurately judge one's abilities. 

- Need more evidence for discrimination that occurs during resume screening?  Here ya go.

- Understanding why certain individuals perform better during interviews is a needed area for study.  In this article, the authors demonstrate the importance of simply being ability to generate ideas, rather than analyzing the situational requirements.

- This study investigates another area needing more attention: the selection into medical training and education programs.  The authors found strong predictive support for SJTs but they came with lower face validity.

- Last in this issue is another interesting study, this time of 360-ratings of innovation.  Interestingly and unlike a lot of other research on self-perception, self-ratings were lower than overall observer ratings.  However, the situation got more complex when the authors separated and analyzed by level of self-rating.


Next the November issue of Journal of Applied Social Psychology:

- One of the most pervasive (yet bizarrely under-discussed) areas of discrimination in the workplace is age discrimination.  This study illustrates some of the stereotypes held of younger and older workers--by both groups.  (Spoiler alert: you'll find out how accurate some of these are in just a second)

- I've been waiting for this one, partly because I love hearing about how bizarre and non-face valid some puzzle-based interviews are.  This particular study was looking at perceptions of these interviews compared to a behavioral interview.  Results?  The puzzle-based interviews were consistently less popular.  Oh, did I mention that they didn't work as well?


How about the Winter issue of Personnel Psych?

-  Why do certain applicants withdraw from the recruitment process?  This study suggests a relationship with organizational identification.

- Okay, back to stereotypes about older workers.  Whereas the earlier study looked at what the stereotypes are, this one looks at whether they're true.  The answer: no, but for one: older workers are less willing to participate in training and career development.

- Turns out it's not just the unemployed that are frustrated by the job search process--currently employed individuals feel the same in many ways.  Boy it's too bad we don't have a giant shared database that is able to match job demands with worker abilities...wait...


The November issue of Journal of Applied Psychology has a couple gems:

- Evidence for the predictive validity of the external manifestations of personality as well as the associated implicit motives.

-  A reminder that what makes for effective leadership behavior depends on the culture.


The latest issue of Personnel Review has an interesting research article on utility analysis, where the authors reiterate how challenging it can be to communicate UA information (hint: carrot and stick approach may work best).


There are a couple good ones in the November issue of Psychological Science.

- Multiple-choice tests have been beat up in the past for being nothing more than tests of recognition (rather than productive retrieval).  This study presents evidence that refutes that assumption.  Go multiple-choice!

- Conspiracy theorists take note: governments may be less likely to use the assumed relationship between genetic testing and intelligence to pigeon-hole us into tracks.  Why?  Because, at least according to this study, there appears to be little evidence connecting the two.


Okay, this one is pretty cool--in a slightly scary way.  The authors were looking at the impact that virtual avatar attractiveness has on interview ratings.  Turns out our bias toward attractive people is so strong it extends to the virtual world!  Of course maybe I should have seen that coming...I mean, ever read a comic book?  (hat tip)


Still with me? Last but not least, some disturbing new evidence regarding significance testing and potential publication bias (hat tip).  I'm guessing most of you won't be surprised at the finding.


I don't know if I'll have another update before the end of the year, so if I don't, happy holidays to everyone!