2014 Research of the Year (+ research update)

Happy New Year!  As I've done in previous years, I present below the research articles I ran across in this area that I think were the most impactful and/or important of 2014.  But first, let's catch up on two issues:

First, the Winter issue of Personnel Psychology:

- Situational judgment tests have been shown to be useful for measuring interpersonal skills, but beware: levels of "angry hostility" moderate that relationship.  (Is there a happy hostility?)

- When hiring leaders, should you look for those that have a busy home life, or be wary of them?  In this fascinating study, the authors found that leader family-to-work conflict negatively impacts followers in that it can increase their burnout. However, family-to-work enrichment increased follower engagement through leader engagement.  So the answer is, as usual, not simple: home/family life can be a good thing for followers if it makes the leader more engaged; but if the home/family life is increasing burnout, the leader may pass that along to others.  So it would seem it all depends on how the individual is handling their life outside of work!

Let's look at the November issue of the Journal of Applied Psychology:

- Are men or women perceived as better leaders?  According to this meta-analysis, it depends on how you ask the question.  If you limit the question to other-ratings, women are rated significantly higher.  But if you look at self-ratings, men rate themselves significantly higher.  Which leads to the next question: is it a biological perception or a gender perception, and if the latter, what traits are the most important?

- An intriguing study of how applicant confidence interacts with and can be altered by the recruitment experience, in this case among recruits for the U.S. military.

- Next, a study of employment and job search efficacy.  Not surprisingly, within-person frequency of job search behavior correlated with job offers; interestingly, the relationship between perceived job search progress and efficacy beliefs were moderated by beliefs of internal attribution.

- Last but not least, more evidence of the importance of defining the criteria when predicting job performance.  In this meta-analysis, the researchers found more support for personality traits out-predicting cognitive ability in predicting counterproductive work behavior, that the two predictors are approximately equal in predicting organizational citizenship behaviors, and that cognitive ability outperforms personality when predicting task and overall performance.  So do you want high task performance, OCBs, or do you want to avoid CWB?  :)  (of course the situation is even more complicated depending on whether you're looking at individual, team, leader performance, over what period of time, etc.)


Okay, on to the awards!  Without further ado, here are my nominations for Research of the Year for 2014:

1) Important advancements in our understanding of weight-based discrimination at work: Vanhove & Gordon.

2) A study of applicants posting faux pas on their social networking sites: Roulin.

3) Two important looks at assessments delivered remotely via mobile devices: Arthur, Doverspike, Munoz, Taylor, & Carr, and Morelli, Mahan, & Illingworth.

4) Two fascinating looks at personality at work: Judge, Simon, Hurst, & Kelley; and Wille & De Fruyt

5) An excellent study of how effective staffing and training practices impact firm-level flexibility and adaptability: Kim & Ployhart.

6) An important study of the movement of impactful I/O researchers to business schools: Aguinis, Bradley, & Brodersen.

7) The relationship between conscientiousness and job performance is more accurately described as curvilinear: Carter, Dalal, Boyce, O'Connell, Kung & Delgado

Finally, honorable mention to two great developments in 2014:  the change of some publishers to making access to articles more affordable, and the announcement of an additional journal, the Journal of Personnel Assessment and Decisions.


I'm continually amazed at the quality of thought and research in our area and the passion and practicality you exhibit.  Here's to an amazing 2014 and more in 2015!

Research update

It's that time, February research update--and there are some really fascinating studies out there:

First the March 2013 issue of IJSA:

- Procedural fairness is an important aspect of organizational attraction, especially for rejected applicants.

- Does personality type impact perceptions of the selection process?  According to this study and their measures (Bohemian is a personality type?), sometimes.

- Do you use SEM to generate a SED and ultimately score bands? Then you will definitely want to read this piece, which suggests the traditional method is flawed.

- Emotional intelligence continues to be a hot topic.  In this study the authors describe the development of a SJT-based measure.

- Staying on the topic of personality, this study adds to the debate over the utility of constructs vs. facets in predicting performance--this time with conscientiousness predicting police officer performance.

- P-O fit is often used to explain why people apply for certain jobs.  In this study researchers found that values that applicants find attractive are particularly important (as opposed to neutral or aversive aspects).

- I've written about automated reference check systems before.  In fact I've written about the company that the first author of this study is from (SkillSurvey).  What is partially a demonstration of the value of their product also adds to our knowledge of reference checks by finding no demographic differences and the ability to predict involuntary (but not voluntary) turnover.

- How big of a deal is it when it is discovered that an applicant "embellished" on their resume?  Depends on how much you already liked them.

- In this study of applicants in Iran, the authors found that the usefulness of a web-based application system was more important than how easy it was to use.  So basically looks are important but not sufficient.


Next, the January issue of the Journal of Applied Psychology: 

- Construct validity has always been a sticky wicket.  The authors of this study take an interesting approach, using video-based testing to illustrate a method for supporting construct validity.

- This study explores the variance in assessment center ratings stemming from various sources, and the authors argue that existing literature has masked this variance.

-The authors of this study show how the application of a particular bootstrap method applied to meta-analyses increases accuracy.

- Speaking of meta-analysis, the authors of this study demonstrate that the differences between coefficient alpha and composite reliability in actual data sets is quite small.


On to the January issue of the Journal of Applied Social Psychology, which as of the time of this post was available for free!

- Similar to findings in other areas, the impact of negative publicity is stronger than positive publicity on organizational attraction, according to the authors of this study.  The authors also found in these cases, organizations would do well to provide detailed recruiting ads to mitigate the impact.

- Age discrimination seems to be increasing in popularity as a research topic (which makes sense from a demographic perspective).  In this study, the authors found raters discriminated against older and younger applicants, with the former being the least likely to be hired.

- Here's an interesting one: in this study, people of color perceived White women to be, among other things, ditsy, shallow, privileged, and appearance focused.  Stereotypes know no color boundaries.


Let's take a look at the February 2013 issue of the Journal of Personality and Social Psychology:

- First, a fascinating look at how generalizable the five-factor model of personality is across cultures.  Specifically, the authors looked at a largely illiterate, indigenous culture in Bolivia.  They failed to find support for the widely used model, and suggest that the structure of personality may differ across societies.

- Second, a study of sex and gender differences that points out the importance of considering the construct of gender (as opposed to the binary concept of sex) for a variety of behavioral and psychological indicators, including big five personality.


Let's move now to the February 2013 issue of Psychological Science, where:

- In this study, the authors found that age differences in cognitive functioning within cohorts were often as large as those between cohorts, which calls into question the cohort explanation for age differences.

- In a study that I think has implications for personnel assessment, the authors found that raters of M.B.A. applications that had already given several high recommendations were less likely to do so later in the day.  Another bias to watch out for!


Last but not least, there was an article recently in the New York Times about how important referrals are for some organizations and the increased importance this places on social networking sites.  There was a reference to a study done in August of 2012 on source of hire that I hadn't seen before, which found that not only were referrals more likely to be hired, they had longer tenure--which is consistent with other research I've seen.  Unfortunately, no word about performance differences.

That's all for now!

Final research update of 2011

Welcome to the last HR Tests research update of 2011! This year has been filled with research developments, building on previous thinking as well as venturing out into new areas. Let's see what the end of the year brings us (hint: a lot)...

We'll start with the Winter 2011 issue of Personnel Psychology:

- If you're looking for executives who excel at strategic thinking, you'll want to pay attention to not only their cognitive ability but their accumulated work history, according to Dragoni, et al.

- I tend to think of realistic job previews (RJPs) as occurring pre-hire, but research by Earnest, et al. suggests that an effective technique is to conduct an oral and written RJP post-hire--and be as honest as possible.

- Meta-analyses are relied upon heavily as summaries of large bodies of research. Roth, et al. point out ways we can make them better, particularly with respect to estimates of validity and group differences.

- One of the downsides of cognitive ability tests is they're not always perceived very well by applicants. Sumanth & Cable show how this perception is influenced by the status of the organization as well as individual status.

- Next, a fascinating description of a scale to measure the idea of calling by Dobrow and Tosti-Kharas. What I found most interesting was how the measure was associated with different criteria for different domains (e.g., art, music, management). You can read a draft version here.


Let's turn now to the January issue of JOB:

- Political skill is a hot topic, and Gentry, et al. demonstrate how the perception of promotability related to this skill varies between bosses, peers, and direct reports. Long story short: it differs, and also depends on behavior (attitude will only take you so far, right?). You can read the full version (at least right now!) here.

- Anyone that's worked with (or has in their family) engineers or scientists knows that they often share some strong traits. This suggests leading these groups that engage in creative activity may require specific attributes, which is what Robledo, et al. set out to describe.


On to the January issue of JOM:

- Core self-evaluation (CSE) is another hot topic, and Chang, et al. provide an important review of 15 years of research, including the meta-analytic support for CSE predicting in-role and extra-role performance.

- Boswell, et al. provide an integrative review of the concept of job search across different situations (e.g., following job loss, while employed).

- One area that deserves more attention is selection into the highest positions within organizations. Withers, et al. provide a review of the process of selecting a director for a board.


Let's not forget v41(2) of Personnel Review:

- Hoglund delves into the topic of talent management as a strategic HRM practice. A fascinating topic that reinforces the importance of HRM as an influence over employee perceptions and behavior.

- The wording of job ads can have important impacts on applicant perceptions and behavior. De Cooman and Pepermans analyze the differences between for-profit and non-profit job ads, and show how only a fraction of the information potentially relevant for job-person match is published.

- Another topic that deserves additional attention is the motivation to expatriate. Altman and Baruch describe results of a qualitative study that may be useful to organizations thinking about attracting and selecting for positions that require this substantial move.


Don't forget this one:

- Johnson, et al. with an important reminder that when looking at the issue of discrimination, using single categories to define groups is probably not the best strategy.


Whew! And last but certainly not least, in the December issue of IOP, Michael McDaniel and colleagues present an argument (similar to ones made elsewhere) that the Uniform Guidelines are outdated and, worse than that, a detriment to the field of selection. The commentaries are many and range from support to passionate disagreement, with a healthy dose of caution (and dare I say...intransigence?) thrown in. Worth a read, particularly for those following the Guidelines literally and those engaged in related litigation. You can read a draft version here.


I hope everyone has a great New Years; here's to a wonderful 2012!

New emotional intelligence meta-analysis: Now with 65% more studies!

Emotional intelligence (EI) continues to be a hot topic in the I/O and HR communities. Some are big fans, some (including me) are more skeptical.

Last year's article in IOP by Cherniss, et al. and the accompanying commentaries provided a great overview of the current situation, with the bottom line that while many seem to agree on a definition of EI, measurement is all over the place, with different methods measuring different things and a distinct lack of convergent and discriminant validity.

EI has been shown to have important relationships with a variety of criteria, but many researchers and practitioners have been left wondering, what exactly is being measured?

In the latest Journal of Organizational Behavior, O'Boyle et al. attempt to shed light on the situation, primarily by looking at whether measures of EI add predictive validity above and beyond cognitive ability and five-factor personality measures.

You may remember a similar study published back in early 2010 by Joseph and Newman. So why the need for this one? The current authors point out that in addition to updating the data and using more current estimates of other relationships, "our data set includes 65 per cent more studies that examine the relationship between EI and job performance, with an N that is over twice as large." I don't know why I found this so funny--there's nothing wrong with trying to get a better handle on things--I guess it just reminded me of packaging on dishwasher detergent or something. In case you're curious, their final sample included 43 effect sizes relating EI to job performance.

ANYWAY, what did they find? In the words of the authors: "We found that all three streams of EI correlated with job performance. Streams 2 and 3 incrementally predicted job performance over and above cognitive intelligence and the FFM. In addition, dominance analyses showed that when predicting job performance, all three streams of EI exhibited substantial relative importance in the presence of the FFM and intelligence." [Stream 1 = ability measures, Stream 2 = self report, Stream 3 = mixed models]

Let's do the numbers: the correlation between EI and job performance varied depending on which "stream" of EI research was analyzed, from about .24 to about .30. So if you're considering using a measure of EI, any of the reputably-built measures should add validity to your process (and in fact these numbers may be low since the job performance measures were primarily task-based). Since the numbers were pretty similar, it seems they support a similar construct, right? Not so fast.

When they looked at how the streams related to personality measures and cognitive ability, they were all over the place. In their words, "For all six correlates (FFM and cognitive ability), we found significant Q-values indicating that the three streams relate to other personality and cognitive ability measures differently...These differences in how the EI streams related to other dispositional traits provide a contrasting perspective to the assertion that the various measures of EI assess the same construct."

What about the incremental validity? It ranged from pretty much nothing (stream 1) to a respectable .07 (stream 3) with stream 2 in the middle at .05. The stream 1 measures seem to have much more overlap with cognitive ability measures, and since ability measures dominate the prediction of performance, well...there you go.

So does that mean employers should avoid stream 1 measures (e.g., MSCEIT)? Again, not so fast. The authors point out that this type of EI measure may be more resistant to social desirability and faking effects--it primarily seems to lose its value when you're already using a cognitive ability measure. (Although again, we're talking task performance here)

All of which leaves me still wondering: what exactly is being measured? More to come I suppose; after all, EI is a newbie in the history of I/O constructs. The authors themselves point out that "much more work is still needed" on the construct validaty of EI.

On the other hand, the more practical among you may be thinking, "who cares? it works!" And to them, I would only say, remember what Kurt Lewin said: "There is nothing so practical as a good theory." Without the theory...

You can see an in-press version here. It's good stuff.

Oh, and on a totally unrelated point, I highly recommend upgrading your Adobe Reader to version X if you haven't already. Highlighting with bookmarking at the click of a button.

Spring '11 P-Psych: Leadership, interviews, competency modeling

It's journal season--this time let's take a look at the Spring 2011 issue of Personnel Psychology:

First up, Derue, et al. with an important meta-analysis on leadership effectiveness. After looking at 59 studies, they found that leader traits and behaviors explained a minimum of 31% of leadership effectiveness. Interestingly, group performance (which I would argue is the most important criterion) was the most difficult to predict. Behaviors tended to explain more than traits, but the authors suggest a model where behavior mediates the relationship between traits and effectiveness is warranted. Not surprisingly, the best trait predictor depended on the criterion: conscientiousness predicted leader effectiveness, group performance, and follower job satisfaction the best, while satisfaction with leader was best predicted by leader agreeableness (reminds me of a recent IJSA study). The same was true with leader behaviors, although consideration was a good predictor across the criteria. A must for anyone interested in leadership research, and you can read an in-press version here.

Next, a piece by Melchers, et al. on whether more interview structure really leads to better rating quality (I'll ruin it for you: yes). Specifically, using a sample of primarily undergraduates, the authors found that providing subjects with frame-of-reference (FOR) training and descriptively anchored rating scales led to substantial increases in rater accuracy and interrater reliability. You can read an in-press version here.

Those of you interested in the concept of core self-evaluations will want to read the study by Ferris, et al.

One area that we don't see enough research in is newcomer adaptation. The longitudinal study by Wang, et al. of a group of Chinese subjects helps fill that hole by exploring the relationship between adaptability, person-environment fit, and work-related outcomes.

Last but most certainly not least, Campion, et al. provide us with a review of best practices in competency modeling. Specifically, 20 of them. Of particular note to you may be that they distinguish competency modeling from job analysis. Want to read the whole thing? Me too! Wish I could have found an in-press version but no such luck.

Sidenote: while not specifically related to recruitment or selection, Christian, et al.'s piece on engagement may be of interest to several readers.

Relax, researchers

Researchers make a lot of choices: What to study. How to measure. What statistic(s) to use. Researchers conducting meta-analyses (essentially a study of studies) face similar choices. So it's nice to know that any agonizing meta-analysts go through may be largely unimportant in terms of the results.

In the January 2011 issue of the Journal of Management, Aguinis et al. report the results of their study of 196 meta-analyses published in places like Journal of Applied Psychology, Academy of Management Journal, and Personnel Psychology. Specifically, they looked at the impact that various methodological choices had on the overall effect sizes. Things like:

- Should we eliminate studies from the database?

- Should we include a list of the primary-level studies used in the meta-analysis?

- What meta-analytic procedure should we use?

Here, in their words, were the results:

"Our results, based on 5,581 effect sizes reported in 196 published meta-analyses, show that the 21 methodological choices and judgment calls we investigated do not explain a substantial amount of variance in the meta-analytically derived effect sizes. In fact, the mean effect size...for the impact of methodological choices and judgment calls on the magnitude of the meta-analytically derived effect sizes is only .007. The median [effect size] value is an even smaller value of .004."

So not only does this suggest researchers should spend less time worrying about methodological choices, it raises a question about the value of including too much of this history in published articles--if it doesn't make any difference, do I really need to read about why you chose to eliminate studies due to independence issues or missing information?

The article has another, perhaps even more interesting, finding. And it's something that us personnel research nerds like to argue over: corrections. Things like correcting for range restriction (e.g., because you only hire people that get the top scores) and criterion unreliability (e.g, because supervisor performance ratings are less than perfect). For every "corrections were necessary to understand the conceptual-level relationships" you'll get a "that's great, but in the real world the data's the data."

So that's why it's fairly amusing to note the differences these types of corrections tended to make. Again, in the authors' own words:

"across 138 meta-analyses reporting approximately 3,000 effect sizes, the use of corrections improves the resulting meta-analytically derived effect size by about .05 correlation coefficient units (i.e., from about .21 to about .25). Stated differently, implementing corrections for statistical and methodological artifacts improves our knowledge of variance in outcomes by about only 25%."

So are all going to stop arguing and hold hands? Not likely. For a variety of reasons, including the fact that opinions are hard to change, and social science researchers are not immune. You could also argue that in some cases an increase from .21 to .25 has significant implications--and to be sure that's true. But I agree with the authors that the number of cases where this greatly increases the practical usefulness of a theory is small.

Does this mean we should forget about refining our results to make them more accurate? Absolutely not. It just means that our current approaches may not warrant a lot of hand-wringing. It also means we should focus on high-quality primary studies to avoid the garbage-in, garbage-out problem.

So let's take a deep breath, congratulate ourselves for a productive year, and look forward to 2011, which I'm sure will bring with it all kinds of exciting and provoking developments. Thanks for reading.

By the way, you can access the article (at least right now) here.

Observer ratings of personality: An exciting possibility?

Using measures of personality to predict job performance continues to be one of the most active areas in I/O psychology. This is due to several things, including research showing that personality measures can be used to usefully predict job performance, and a persistent interest on the part of managers in tools that go beyond cognitive measures.

Historically the bulk of research on personality measures used for personnel assessment has been done using self-report measures—i.e., questionnaires that individuals fill out themselves. But there are other ways of measuring personality, and a prominent method involves having other people rate one’s personality. This is known as “other-rating” or “observer rating.”

Observer ratings are in some sense similar to self-report measures that ask the rater to describe their reputation (such as the Hogan Personality Inventory) in that the focus is on visible behavior rather than internal processes; in the case of observer ratings this takes on even greater fidelity since there is no need to “guess” at how behavior is perceived.

Research on observer ratings exists, but historically has not received nearly the attention given to self-ratings. Fortunately, in the November 2010 issue of Psychological Bulletin, Connelly and Ones present the results of several meta-analyses with the intent of clarifying some of the questions surrounding the utility of observer ratings and, with some caveats, largely succeed. The study was organized around the five-factor model of personality, namely Openness to Experience, Conscientiousness, Extraversion, Agreeableness, and Emotional Stability.

The results of three separate meta-analyses with a combined 263 independent samples of nearly 45,000 individuals yielded several interesting results, including:

- Self- and other-ratings correlate strongly, but not perfectly, and are stronger for the more visible traits, such as Extraversion and Conscientiousness, compared to those less visible (e.g., Emotional Stability).

- Each source of rating seems to contribute unique variance, which makes sense given that you’re in large part measuring two different things—how someone sees themselves, their thoughts and feelings, compared to how they behave. Which is more important for recruitment and assessment? Arguably other-ratings, since we are concerned primarily not with inner processes but with behavioral results. And it certainly is not uncommon to find someone who rates themselves low on a trait (e.g., Extraversion) but is able to “turn it on” when need be.

- Self/other agreement is impacted by the intimacy between rater and ratee, primarily the quality of observations one has had, not simply the quantity. This is especially true for traits low in visibility, such as Emotional Stability.

- Inter-rater reliability is not (even close to) perfect, so multiple observers is necessary to overcome individual idiosyncrasies. The authors suggest gathering data from five observers to achieve a reliability of .80.

- Observer ratings showed an increased ability to predict job performance compared to self-ratings. This was particularly true for Openness, but for all traits except Extraversion, observer ratings out-predicted self-ratings.

- Just as interesting, observer ratings added incremental validity to self-ratings, but the opposite did not hold true. This was especially true for Conscientiousness, Agreeableness, and Openness.

- Not only were observer ratings better at predicting job performance, the corrected validities were considerably higher than have been documented for self-report measures, and in one case—Conscientiousness—the validity value (.55) was higher than that reported previously for cognitive ability (.50), although it should be noted that the corrections were slightly different. Values before correcting for unreliability in the predictor were significantly lower.

Why do observer ratings out-perform self-ratings? Perhaps other-ratings are more related to observable job performance. Perhaps they do not contain as much error in the form of bias (such as self-management tendencies). Or perhaps because the measure is completely contextualized in terms of work behavior, whereas the questions in most self-report measures are not restricted to the workplace.

Despite these exciting results, some caution is in order before getting too carried away. First, the authors were unable to obtain a very large sample for observer ratings—sample sizes were around 1,000 for each trait. Second, there are currently a limited number of assessment tools that offer an observer rating option—the NEO-PI being a prominent example. Finally, there is an obvious logistical hurdle in obtaining observer ratings of personality. It is difficult to see how in most cases an employer would obtain this type of data to use in assisting with selection decisions.

However, even with that said the results are promising and we know of at least one way that technology may enable us to take advantage of this technique. What we need is a system where trait information can be gathered from a large number of raters relatively quickly and easily and the information is readily available to employers. One technology is an obvious solution, and it’s one of my favorite topics: social networking websites. Sites like Honestly (in particular), LinkedIn, and (to a lesser extent) Facebook have enormous potential to allow for trait ratings, although the assessment method would have to be carefully designed and some of the important moderators (such as degree of intimacy) would have to be built in.

This is an intriguing possibility, and raises hope that employers could soon have easy access to data that would allow them to add substantial validity to their selection decisions. But remember; let’s not put the cart before the horse. The question of what assessment method to use should always come after an analysis of what the job requires. Don’t jump headlong into other-ratings of personality when the job primarily requires analytical and writing skill.

This is just another tool to put in your belt, although it’s one that has a lot of exciting potential.

Personnel Psychology, August 2010

The August, 2010 issue of Personnel Psychology came out a while ago, so I'm overdue in taking a look at some of the content:

Greguras and Diefendorff write about their study of how "proactive personality" predicts work and life outcomes. Using data from 165 employees and their supervisors across three time periods, the authors found that proactive individuals were more likely to set and attain goals, which itself predicted psychological need satisfaction. It was the latter that then predicted job performance and OCBs as well as life satisfaction.

Speaking of personality, next is an interesting study by Ferris et al. that attempts to clarify the relationship between self-esteem and job performance. Using multisource ratings across two samples of working adults, the authors found that the importance participants placed on work performance to their self-esteem moderated this relationship. In other words, this suggests that whether self-esteem predicts job performance depends on the extent to which people's self-esteem exists outside of their performance. Interesting.

Lang et al. describe the results of a relative importance analysis of GMA compared to seven narrower cognitive abilities (using Thurstone's primary mental abilities). Using meta-analysis data, the authors found that while GMA accounted for between 10 and 28% of the variance in job performance, it was not consistently the strongest predictor. Add this study to a number of previous ones suggesting that one solution to the validity-adverse impact dilemma may be in part to use narrower cognitive abilities (e.g., verbal comprehension, reasoning).

Last but definitely not least, Johnson and Carter write about a large study of synthetic validity (a topic Johnson writes more about in the August issue of IOP). For those that need a reminder, synthetic validity is the process of inferring validity rather than directly analyzing predictor-criteria relationships. After analyzing a fairly large sample, the authors found that synthetic validity coefficients were very close to traditional validity coefficients--in fact within the bounds of sampling error for all eleven job families studied. Validity coefficients were highest when both predictors and criterion measures were weighted appropriately.

So what the heck does that mean? Essentially this provides support for employers (or researchers) who lack the resources to conduct a full-blown criterion validation study but are looking for either (a) a logical way to create selection processes that do a good job predicting performance, or (b) support for said tests. Good stuff.

Research update

Here are a few research articles you may have missed. Just links this time because...well, I'm really enjoying World Cup.

McGrath, et al.: Evidence for response bias as a source of error variance in applied assessment.
(May Psych. Bulletin)

Hulleman, et al.: A meta-analytic review of achievement goal measures: Different labels for the same constructs or different constructs with similar labels? (May Psych Bulletin)

Thornton & Potemra: Utility of assessment center for promotion of police sergeants. (March Public Personnel Management)

Bright: Why age matters in the work preferences of public employees: A comparison of three age-related explanations (March Public Personnel Management)

Thomas, et al.: Employee proactivity in organizations: A comparative meta-analysis of emergent proactive constructs (June JOOP)

Bjørkelo, et al.: Predicting proactive behaviour at work: Exploring the role of personality as an antecedent of whistleblowing behaviour (June JOOP)

Zibarras & Woods: A survey of UK selection practices across different organization sizes and industry sectors (June JOOP)

May '10 J.A.P.

Summer journal madness continues with the May issue of the Journal of Applied Psychology. It's a diverse issue, check it out:

- Taras et al. conducted a very large meta-analysis of the association between Hofstede's cultural value dimensions (e.g., power distance, masculinity, individualism) and a wide variety of individual outcomes. One interesting finding is the stronger relationship between these values and emotions (organizational commitment, OCBs, etc.) compared to job performance.

- Are high performers more likely to stay or leave? In a study of over 12,000 employees in the insurance industry over a 3-year period, Nyberg found the answer was: it depends. Specifically, it depends on the labor market and pay growth.

- Think g (cognitive ability) is just related to job performance? In a (albeit small) study by Judge, et al., it turns out it was also related to physical and economic well-being. Maybe their next study will address my personal hypothesis: g is related to choice of car.

- A study by Lievens, et al. (in press version here) found with a sample of 192 incumbents from 64 occupations that 25% of the variance in competency ratings (like you might find in a job analysis) was due to the nature of the rater's job, such as level of complexity. Not surprisingly, the greatest consensus was reached for jobs that involved a lot of equipment or contact with the public.

- Self-efficacy (i.e.., confidence) has been proposed as an important predictor of job performance. In a study by Schmidt & DeShon, the authors found that this relationship depends on the ambiguity present in the situation--in situations high in ambiguity, self-efficacy was negatively related to job performance; in situations low in ambiguity, the opposite was true.

- Finally, for anyone citing Ilies, et al.'s 2009 study of the relationship between personality and OCB, there have been a couple corrections.

Personnel Psychology, Spring 2010: SJTs, affect, and job offer timing

The Spring 2010 (v.63, #1) issue of Personnel Psychology is out. Let's look at the highlights:

First out of the gate, a great meta-analysis for anyone interested in situational judgment tests (SJTs; and who isn't?). Christian, et al. looked at 84 studies and found some pretty interesting things:

1) SJTs reported in the literature have been used to measure a variety of things, including leadership skills (37%), some type of composite (33%), interpersonal skills (12.5%), personality tendencies (9.6%), teamwork skills (4.4%) and job knowledge/skills (3%).

2) Criterion-related validity depends--as you might expect--on the match between predictor and performance measure. Conscientiousness measures, for example, predicted task performance much better than managerial performance (rho=.39 and .06 respectively). The highest correlations (albeit based on relatively small samples) were for teamwork skills and personality composites predicting task performance (.50 and .45 respectively).

3) Video-based SJTs tended to have stronger criterion-related validity values compared to paper-based measures. This was particularly true when measuring interpersonal skills (.47 compared to .27).

Second, a small but interesting study by Johnson, et al. on the relationship between trait affect (i.e., being generally disposed to feeling positive or negative emotions) and job performance. Results from 120 matched employee-supervisor pairs from a variety of jobs using both explicit (survey) and implicit (word fragment completion) measures of affect found substantial correlations, particularly between positive affect and performance (in the .50 range), and particularly when using implicit measures.

Something to add to a selection battery, perhaps? Could be perceived negatively by applicants, however, and I can see some questions being raised about the link to medical issues. But the same types of concerns were originally leveled at personality tests and were mitigated by creating measures specifically tied to work behavior. Definitely an area for more research.

Third, check out this study by Becker, et al. on the impact that job offer timing has on acceptance, performance and turnover. The authors found (using data from a Fortune 500 engineering technology company) that for both student and experienced samples, faster offers were associated with higher acceptance rates. Specifically, for experienced candidates, the difference between 2 weeks and 3 weeks taken to make the offer was substantial, whereas for the students 3 weeks versus 4 weeks was important. But, no differences were found in terms of either performance ratings or turnover among employees hired through different offer speeds.

Implication? The study suggests that offer time does impact the likelihood that the offer will be accepted, but viewed broadly this may not have long-term impacts in terms of how employees do on the job. Maybe in cases of good candidate-employer fit, candidates are willing to wait.

Last but not least are the book reviews. Two books are particularly relevant for us, The Structured Interview (Pettersen & Durivage) and Outliers (Gladwell). The first is received very positively and sounds like a great source for anyone wanting more details about the support for and use of structured interviews. The latter is "well worth [a] few evenings" but requires you to overlook the lack of evidence and convenient inferences.

Final notes: those of you interested in multisource performance ratings should check out Hoffman, et al.'s article, which reinforces the impact of having raters from different levels. Chuang and Liao's article also includes a useful measure of a high-performance work system.

Recruitment v. Assessment, Round 1: Fight!

I'm sure some of you are avid readers of ERE (Electronic Recruiters Exchange), but for those of you that aren't (and don't receive my shared items), there's a lively discussion going on over there regarding the practical value of recruitment versus assessment practices.

It started with Wendell Williams' first post on how to identify a bad test (the second part is also worth reading). The comments begin relatively benignly, debating the strength of various predictors of performance (e.g., P-O fit versus behavioral interviews), but turns into, let's say, a lively debate that includes a discussion of the Gallup 12, the limitation of assessments, the Uniform Guidelines, and a lot more. The most heated exchange occurs between Wendell and Lou Adler, where accusations and sarcasm fly.

Speaking of Lou, he continues to advocate his perspective with his next post on whether increasing interview accuracy increases quality of hire (yes, he's suggesting that's an open question). While the comments following are fewer in number, the debate continues regarding the value of assessment and the evidence used to support it (e.g., Schmidt & Hunter's 1998 piece).

Who said HR is boring?

Want better prediction? Gather more data.

That's the bottom line from a study in the November 2009 issue of the Journal of Applied Psychology.

Oh & Berry looked at how adding personality ratings from peers and supervisors added incremental validity to self-ratings using a five-factor model measure. What were the results? Increases of 50-74% in operational validity across personality facets. They also looked at differential prediction of task and contextual performance (unfortunately those results weren't reported in the abstract). Bottom line? If you're using a personality assessment for promotions, strongly consider gathering data from co-workers.

Speaking of self-presentation, in the same issue Barrick et al. report the results of a meta-analysis of how self-presentation tactics (e.g., appearance, non-verbal behavior) impact interview ratings and later job performance. Results? "What you see in the interview may not be what you get on the job and...the unstructured interview is particularly impacted by these self-presentation tactics." An important reminder of how who the candidate seems to be impacts your assessment, and another reason to collect multiple sources of data.

There are a number of other great articles in this issue, such as:

How Major League Baseball CEO personalities impact important outcomes (like, um, winning).

How SJT and biodata measures add to the prediction of college student performance.

How personality scale validities change over time among a group of medical students.

Differences among letters of recommendation in academia between genders.

Explain away

Of all the low-hanging fruit in recruitment and selection, perhaps none are easier to implement than explaining your process. It's shocking how few selection processes are fully (and coherently) explained, not just in terms of getting from point A to point B, but why the points exist in the first place.

Turns out explaining the process to candidates matters. A lot. And while we already knew that applicant perceptions were important, a recent meta-analysis by Truxillo, et al. published in the International Journal of Selection and Testing clarifies the impact that explanations have. Specifically, explanations were related to:

- Fairness perceptions (important in their own right)

- Perceptions of the hiring organization

- Test-taking motivation

- Performance on cognitive ability tests

Furthermore, fairness effects were greater when paired with personality tests rather than cognitive ability exams.

What does all this mean? It's pretty simple, really--communicate, communicate, communicate. Explain in clear terms to all applicants what the full selection process is, and why. Imagine something like this:

"Thank you for your interest in applying for the Blog Reader position. The selection process will consist of the following:

Step 1. You submit your application and work sample by December 5.

Step 2. Your application is reviewed to ensure you meet our minimum qualifications found on the job posting.

Step 3. If so, your work sample is scored by internal subject matter experts. It will be judged on relevance to the position, complexity, and contribution to the profession. The top scorers move on to Step 4.

Step 4. You are contacted by Human Resources to set up an interview. The interview will last approximately 2 hours and will take place at our San Jose campus. They will take place in mid-January.

Step 5. You interview with the hiring supervisor and 2-3 potential co-workers. You will be asked a series of questions designed to measure your knowledge of Blogs. You will also be asked to complete a writing sample which will be judged for style as well as content. Top performers in these steps will be asked for a final interview with the head of the Blogger Division.

Should you fail to proceed to any Step, you will be contacted and told why."

That wasn't so hard, was it?

For more about applicant perspectives in selection, check out the entire December '09 issue.

And while I'm on the subject of research, check out the latest issue of the International Journal of Testing for good stuff on test compromise, DIF, and P-O fit. Oh, and don't forget about the entire issue devoted to test adaptation.

Last but not least, Practical Assessment, Research, and Evaluation (PARE) has put out several things lately worth reading, check 'em out.

Personality tests: Situation matters

Is a personality test the right selection mechanism for your needs? In trying to answer that question, an important consideration is: Does the job allow for the expression of personality facets?

There's a concept in psychology called situation strength. It refers to how the "strength" of a situation impacts the display of personality via behavior, and it's something important to remember when using measures of personality to predict job performance.

A situation's "strength" refers to the environment under which personality aspects are displayed; think of it like a rulebook. If Job A is described perfectly in exacting detail with very little room for deviation ("Place container A over part B..."), does one's personality really matter when it comes to successfully performing the job? Or thought of another way, if the rulebook repeatedly emphasizes using an aspect of personality (e.g., extraversion), how will people without the ability to express that behavior consistently fare?

If you're a software programmer, it probably matters little in the grand scheme of things how extraverted you are; analytical ability is likely much more important. Similarly, if you work in a call center and follow a script, differences in openness to experience probably don't mean much; extraversion is likely more important.

But what about something like conscientiousness--could the strength of a situation impact the relationship between this aspect of personality and job behavior? That's the question that Meyer and colleagues set out to answer, and an answer is published in a recent issue of the Journal of Organizational Behavior.

The way they went about trying to solve this puzzle was to conduct a meta-analysis at the occupation (rather than job) level. What did they find? A few things:

1) Uncorrected correlations between conscientiousness and performance varied widely between .06 and .23.

2) Correlations appear slightly stronger when using overall performance as the criterion rather than task performance (not surprising given previous research).

3) Stronger correlations were found in "weak" occupations.

What does this mean? Well, for one it reinforces the fact that the answer to "Do personality tests work?" varies greatly depending on what the job is and how you measure performance. But perhaps more interestingly, it suggests that at an occupation level we can expect that for jobs that come with built in rules regarding behavior ("strong" occupations), measures of personality aspects such as conscientiousness may not predict performance as well as for jobs with more flexibility ("weak" occupations).

So the next time you're thinking about using a personality inventory for selection purposes, consider: To what extent will incumbents be allowed to express their personality?

Latest IJT and PP

The latest issue of the International Journal of Testing has some good stuff in it, particularly if you're in education. Here's a sample of what's available:

Correcting fallacies in (construct) validity, reliability and classification


Then head on over to Personnel Psychology to check out the Autumn issue. Here's some of what it covers:

Conscientiousness and KSAs predicts leadership in the military

Job component validation, meta-analysis, and DOT ratings

Personality variables and job search behavior and success

The importance of specificity and observability in job analysis ratings

September '09 IJSA

The September 2009 issue of IJSA (International Journal of Selection and Assessment) is chalk full of good stuff. Let's dive in.

1) An important update of the "guidelines and ethical considerations for assessment center operations"--a must for anyone interested in the appropriate use of assessment centers.

2) Speaking of assessment centers, here's a meta-analysis of how they correlate with cognitive ability and personality, as well as the proper way to weight the results.

3) Speaking of cognitive ability, curious about the correlation between ability and faking? Check out this large-sample study of faking on a biodata measure.

4) Worried about what your applicants think of your selection method? Frame it as select in (accept) rather than select out (reject).

5) Want to make sure your raters are rating accurately? You may want to re-think stocking your panel with agreeable people (sounds like a lot of fun for the exam analyst!).

6) Before you put the finishing touches on your new online job application system, make sure you pay attention to its features, user friendliness, and efficiency. I like to think of this as "Googley."

7) Looking for a measure of person-job fit that relates equity of contribution to reward? Check this out.

That's all for now!

Discrimination, assessment centers, and handshakes

The title sounds like a strange combination, no? That's because it refers to three separate pieces of research published in the September 2008 Journal of Applied Psychology.

First, Umphress et al. describe a study that demonstrates how important leaders can be in setting the tone for selection. Specifically, the authors found that when authority figures focused team selection decisions on job performance factors, individuals that had a tendency to discriminate based on social dominance orientation were less likely to do so. Implication? To help avoid discriminatory hiring and promotion decisions, focus decision makers on job-related performance factors. That way, they're less likely to rely on their own biases.

Next, Meriac et al. with a meta-analysis of the incremental validity of assessment center (AC) ratings over other assessment tools. Specifically, the authors found that AC ratings explained a "sizable proportion of variance in job performance" beyond cognitive ability and personality tests. Good news for fans of assessment centers out there.

Last but not least, Stewart, et al. describe the results of a study of 98 undergraduate students that participated in mock interviews. In the words of the authors, "quality of handshake was related to hiring recommendations." How exactly does that work? Apparently how you shake hands sends messages about your degree of extraversion (above and beyond your appearance). The authors also found that the effect seemed to be stronger for women than men. Implication? For those of you interviewing for sales jobs, pay attention to your handshake!

Honorable mentions:

- Judge & Livingston on how traditional gender role orientation impacts the wage gap

- Cascio & Aguinis on trends in I/O psychology from 1963 - 2007 (good stuff; read here)

- Chiaburu & Harrison's meta-analysis on how co-workers impact job performance

- Levi & Fried on differences between African Americans and Whites on attitudes toward affirmative action programs

Is age related to job performance?

Is there a relationship between age and job performance? It's an important question for many reasons, including the fact that claims of age discrimination appear to be on the rise. Ng and Feldman set out to better understand this issue and their meta-analysis is published in the March '08 issue of Journal of Applied Psychology.

Previous research has generally shown a weak relationship between age and job performance--at least when we look at objective measures. But the current authors set out to use a much broader array (10 to be exact) of criterion measures, including workplace aggression, safety performance, and OCBs.

So what did they find? Well, there's where things get a bit complex. Although there did not appear to be a relationship between age and several outcomes, including core task performance, creativity, and performance in training programs, it had stronger relationships with the other seven measures. In addition, age had a curvilinear relationship with core task performance and CWBs, and results varied depending on how the study was conducted.

So does age have a relationship with job performance? Like all important research questions, the answer is an emphatic it depends.


Other articles
There's quite a bit of good research in this volume, including:

- The development of a potentially useful way to predict team member performance

- A fascinating look at how frame-of-reference influences the validity of personality measures (pre-published version here)

- A discussion of the importance of the distinction between constructs (e.g., ability, personality) and methods (e.g., interviews) when comparing predictors in personnel selection

- How to test for adverse impact when your numbers are small (hint: significant testing bad, z-score good)

- Last but not least, a meta-analysis of the relationship direction between attitudes and job performance--what causes what? (hint: attitude matters...but not that much)

January '08 issue of J.A.P.

The January 2008 issue of the Journal of Applied Psychology is out. Unfortunately there are only three articles directly related to recruitment and assessment, but they're pretty good ones, so let's dive in.

First up, a Monte Carlo investigation of the impact of faking on personality tests by Komar, et al. "What is a Monte Carlo investigation?" you may ask. Essentially it's when researchers use computers to simulate data scenarios rather than collecting actual data from participants/subjects/victims. Anyway, the researchers looked at the impact on the criterion-related validity (as measured by supervisory ratings) of conscientiousness scores adjusting for various "faking" scenarios. They found that the validity is impacted by a variety of factors, most notably proportion of fakers, magnitude of faking, and the relationship between faking and performance. Another shot across the bow of self-report personality inventories, methinks, although the debate will no doubt continue!

Next a fascinating study of motherhood bias in both expectations and screening decisions by Heilman and Okimoto. The researchers found a bias against both male and female parents when it came to anticipated job commitment, achievement striving, and dependability, although anticipated competence was uniquely low for mothers and seemed to be the major contributing factor to lowered expectations and screening recommendations. An unfortunate reminder that these factors do matter and something to watch out for. The results are reminiscent of negative behavior toward "pregnant" women found in a previous study.

Finally, Zyphur, Chaturvedi, and Arvey present a discussion of job performance. They address two subjects: the impact of past performance on future performance and individual differences in performance trajectories. Analyzing past literature, the authors note that performance feedback influences future performance directly and different individuals do have different latent performance trajectories, which has big implications for selection. Why? Because many assessment techniques (e.g., T&Es, behavioral interviews) rely on an general assumption that more experience equals better performance. This study adds ammunition to those that argue that assumption has serious flaws (or at least is overly simplistic).

In addition to these three, you may find the following interesting as well:

Challenging conventional wisdom about who quits: Revelations from corporate America. (great stuff for those of you interested in retention)

Effectiveness of error management training: A meta-analysis. (for all you trainers out there)

Effects of task performance, helping, voice, and organizational loyalty on performance appraisal ratings. (for those interested in performance ratings)