Research update

Okay, I'm just gonna say it.  There is an insane amount of research out there directly or indirectly related to our field.  I'll be honest, it's a little daunting thinking about reading and processing all of it.

Luckily, you have me to overly simplify it for you so you can incorrectly describe it to others.  So let's jump right in!

First up, the June issue of IJSA:

-  Fairness perceptions matter in web-based selection too!

-  The predictive validity of conscientiousness is moderated by self-enhancement

-  Speaking of conscientiousness, it's a better predictor of performance in routinized jobs than those with complex cognitive requirements

-  Psychological hardiness predicts adaptability in military leaders.  This study is awesome also for showing a negative relationship between SAT scores and West Point performance.

-  Recruiters, listen up: this study found that contingent workers converted to full-time status performed as well as referral and online hires, but not as well as those sourced internally (bookmark this one, there's a dearth of recruitment research)

- Scoring biodata: empirical keying reigns supreme over rational and quasi-rational approaches.  Quasi-rational?  Sounds like me most of the time.

- Personality and job performance: interactions are important beyond main effects


Okay, next, the May issue of JAP:

- Adaptive decision making in military leaders: both the brain and the mind are important (okay you monists out there, chill out)

- Are you a MANOVA (wo)man?  Then read this.

- Proactive leaders set more challenging goals and have higher sales performance--assuming they have the trust of their subordinates.


Next, the April issue of JOB.  Just one study, albeit interesting, in which female evaluators were less likely to recommend hiring or promoting Asian (versus White) applicants into jobs requiring social skills


Speaking of JOB, how about the May issue?

- Have you been calling for more research on calling?  Your wish is granted.

- Curious about the concept of curiosity? (okay, I'll stop)  Looks like it can be predictive of job performance above and beyond traditional cognitive and non-cognitive predictors.  I'm gonna assume it varies with job, but I think the authors are probably right that it will increase in importance over time.

- When selecting for teams that may experience crisis situations (e.g., nuclear power plant crews), consider homogeneity--not mean levels--of positive affect.

- Moderate levels of supervisory structure combined with high levels of consideration lead to lowest levels of CWB's

- The dark side of OCBs

- The light side of OCBs

- Dark side and light side working together in a picture that has nothing to do with selection


Okay, onwards and upwards: one from the April issue of JASP, on generalized self-efficacy, work-related self-efficacy, and job-related outcomes


And one from the May issue:  Sensation seeking and need for structure predict military field exercise performance


How about a little EI research?  Haven't had a lot of that lately.  Here's a piece from the May issue of Journal of Management that found emotion management ability to be a valid predictor of job performance.


Let's look in the May Psychological Science:

- Where we find a fascinating study that demonstrates assessment of profound cognitive abilities at a young age predicts outstanding contributions in many adults

- Another interesting study of how an individual responds to daily stressors (which I would posit are differentially found in various jobs) has long-term consequences for their mental health

- High schoolers with high math and verbal skills are less likely to choose STEM careers than those with math skills but moderate verbal skills

- Okay, check this one out: mindfulness training improved GRE reading comp scores as well as working memory.  Implications for personnel assessment??


Shifting to the May Psychological Bulletin:

- Rorschach, anyone?

- Believing that individuals are malleable, rather than fixed ("implicit theory") predicts goal achievement.  Implications for leaders?


Last but not least, let's hook up with our friends at PARE and see what they're up to:

- Into Excel?  Check out this article on forest plots

- Or maybe factor analysis is your thing.


That's all folks!  Hope your brain is spinning like mine!

Test validity: A fairy tale

Once upon a time there was a field called Industrial and Organizational Psychology.  Its researchers and practitioners dealt with a myriad of magical issues ranging from individual differences and behavior to organizational structures.

Within this field, there was a specialty called Personnel Psychology.  It dealt with narrower--but no less mysterious--issues such as defining and designing jobs and, most relevant for us, finding and hiring the right people.  The I/O psychologists and HR professionals that quested for these answers often found themselves on dangerous missions like battling Monsters of Doubt (i.e., first-line supervisors).

These adventurers had two main weapons at their disposal when fighting these monsters: the Carrot of Truth and the Stick of Pain.

When invoked, the Carrot of Truth, fashioned deep in the Mines of Correlation, caused monsters to realize that hiring the right people was the right thing to do for their realm.  It increased productivity and morale, customer satisfaction, and organizational flexibility.  It also allowed supervisors to spend more time leading and less time dealing with gremlins (i.e., employees with performance problems).

The Stick of Pain used an opposing form of magic but was sometimes equally effective.  It attempted to slay the monsters using a peculiar power called The Law.  The Law frightened monsters because it meant they could experience emotional pain and suffering, and--more importantly--fewer bags of gold.

For a long time, Personnel adventurers used both of these weapons to slay all kinds of monsters, on high mountains and in dungeons.  But over time, the adventurers discovered something: the Stick of Pain was becoming less and less effective.

It wasn't that the Stick was powerless.  It's just that its magic didn't seem to frighten the monsters as much.  The monsters saw their gold piling up and didn't feel the sting of suffering as they once had.  And they started developing an addiction to carrots all on their own.

So there came a day where the adventurers and the monsters met on the battlefield and came to an agreement.  No longer would the adventurer wield the Stick of Pain.  And in return, the monsters pledged to respect the Carrot of Truth.  They forged an eternal partnership and lived happily ever after.

The End.


Okay, so I've taken a little artistic license with my blog post today.  But hopefully you see where I'm going.

Back in the old days (ya know, like the 80s), employers were faced with a foreboding world of testing, with the Civil Rights Act and cases like Griggs vs. Duke Power looming large over their assessment programs.  I/O psychologists were brought in to help organizations navigate the complicated world of employment testing, which required an appreciation of statistics and the law alike.  Large awards and settlements brought C-level attention, and regulatory agencies like the EEOC and DOJ served in ongoing oversight roles, requiring employers to clean up their act with procedural requirements that could be burdensome.

Nowadays, I/O psychologists are as likely to be valued for their ability to crunch "big data" to detect employee behavior trends as their ability to conduct thorough job analyses (not that the two are mutually exclusive).  Lawsuits regarding testing are infrequent compared to issues like wages and hours, harassment, and terminations.  The selection cases that do come up are as likely to involve disabilities as adverse impact due to cognitive loading.

Sure, we have the occasional big case that gets attention.  But the bottom line is over the years the "stick" has become much less effective as an argument for sound assessment than the "carrot."

Smart employers like Google have started crunching the numbers and realized the true business value of defining the right competencies for jobs.  They're doing so not because they're afraid of litigation, but because they see more clearly the direct line between best practices in selection that we've been preaching for years--i.e., focusing on valid assessment results--and the bottom line.


So where does that leave the stick (i.e., fear of lawsuits)?  Is it time to put it away along with phrenology and T&Es (woops, that slipped in)?

I don't think so.  Organizations will always be subject to legal scrutiny when their selection processes have adverse impact and the right person talks to the right attorney.  Personnel psychologists and HR professionals should always have a healthy respect for the legal climate we operate in, and not forget that "job related and consistent with business necessity" isn't fictional gibberish.

But what it does mean is that because organizations are paying attention to their assessments, they are more likely to yield valid results and be more free of illegal bias.  That means management's quest and the selection professional's quest are more likely to converge, with a lot more cooperation.

And hopefully a lot fewer monsters.

Presentations from 2013 ATP Conference

Many of the presentations from the 2013 Association of Test Publishers conference are now available and I strongly recommend checking them out.  Strong themes include gamification, global assessment, item development, and test security, just to name a few.

I found the one from HumRRO on creating a virtual role-play fascinating.  If you haven't already seen it, I strongly recommend you check out their demo at www.humrro.org/simdemo.html.  The username and password are both humrrodemo.  A small taste of what's possible in assessment!

Hat tip.

Research update addendum

Okay, I know I just did a research update but but I had a couple stragglers, including a pretty important one, namely the Spring 2013 Personnel Psychology, which is free right now!

-  First, an important piece by Bobko and Roth updating d values for Black-White differences on selection measures.  Their updated analysis indicates measures such as biodata and assessment centers may have d values as large as paper-and-pencil tests of cognitive ability.  Personality measures still benefit from small differences.  They include a helpful table that breaks down the values by construct, and they also include a list of factors that can impact d, such as job complexity and range restriction.

- Second, a fascinating study of aberrant personality tendencies and their impact on career outcomes conceptualized using the Five-Factor Model and measured using the NEO PI-R.  More evidence that "dark side" personality traits are an important consideration in predicting career trajectories.


The other that just came out is the February 2013 Journal of Applied Social Psychology:

- First, a look at moderators of the relationship between employee weight and job-related outcomes.

- Second, a study that I think has implications for selection: looking at circumstances under which competitors copy their opponents choices.  I've observed over and over again that when an employee gets a competing offer, suddenly their attractiveness increases.  Perhaps not the same phenomenon, but worth exploring.

-  Next, another study of discrimination, this time age discrimination in within- and between-career job changes.  Results indicated discrimination against older workers was particularly pronounced when older applicants were making between- rather than within-career changes.

- Okay, I'm sensing a theme to this post.  In this study, the authors looked at how the wording of occupational descriptions activates gender stereotypes.

- Finally, something not about discrimination: the authors of this (small N) study found that a perceived aspect of emotional intelligence predicted perceived negotiation success beyond traditional personality traits.

And on that note...until next time!

Research update

It's that time, February research update--and there are some really fascinating studies out there:

First the March 2013 issue of IJSA:

- Procedural fairness is an important aspect of organizational attraction, especially for rejected applicants.

- Does personality type impact perceptions of the selection process?  According to this study and their measures (Bohemian is a personality type?), sometimes.

- Do you use SEM to generate a SED and ultimately score bands? Then you will definitely want to read this piece, which suggests the traditional method is flawed.

- Emotional intelligence continues to be a hot topic.  In this study the authors describe the development of a SJT-based measure.

- Staying on the topic of personality, this study adds to the debate over the utility of constructs vs. facets in predicting performance--this time with conscientiousness predicting police officer performance.

- P-O fit is often used to explain why people apply for certain jobs.  In this study researchers found that values that applicants find attractive are particularly important (as opposed to neutral or aversive aspects).

- I've written about automated reference check systems before.  In fact I've written about the company that the first author of this study is from (SkillSurvey).  What is partially a demonstration of the value of their product also adds to our knowledge of reference checks by finding no demographic differences and the ability to predict involuntary (but not voluntary) turnover.

- How big of a deal is it when it is discovered that an applicant "embellished" on their resume?  Depends on how much you already liked them.

- In this study of applicants in Iran, the authors found that the usefulness of a web-based application system was more important than how easy it was to use.  So basically looks are important but not sufficient.


Next, the January issue of the Journal of Applied Psychology: 

- Construct validity has always been a sticky wicket.  The authors of this study take an interesting approach, using video-based testing to illustrate a method for supporting construct validity.

- This study explores the variance in assessment center ratings stemming from various sources, and the authors argue that existing literature has masked this variance.

-The authors of this study show how the application of a particular bootstrap method applied to meta-analyses increases accuracy.

- Speaking of meta-analysis, the authors of this study demonstrate that the differences between coefficient alpha and composite reliability in actual data sets is quite small.


On to the January issue of the Journal of Applied Social Psychology, which as of the time of this post was available for free!

- Similar to findings in other areas, the impact of negative publicity is stronger than positive publicity on organizational attraction, according to the authors of this study.  The authors also found in these cases, organizations would do well to provide detailed recruiting ads to mitigate the impact.

- Age discrimination seems to be increasing in popularity as a research topic (which makes sense from a demographic perspective).  In this study, the authors found raters discriminated against older and younger applicants, with the former being the least likely to be hired.

- Here's an interesting one: in this study, people of color perceived White women to be, among other things, ditsy, shallow, privileged, and appearance focused.  Stereotypes know no color boundaries.


Let's take a look at the February 2013 issue of the Journal of Personality and Social Psychology:

- First, a fascinating look at how generalizable the five-factor model of personality is across cultures.  Specifically, the authors looked at a largely illiterate, indigenous culture in Bolivia.  They failed to find support for the widely used model, and suggest that the structure of personality may differ across societies.

- Second, a study of sex and gender differences that points out the importance of considering the construct of gender (as opposed to the binary concept of sex) for a variety of behavioral and psychological indicators, including big five personality.


Let's move now to the February 2013 issue of Psychological Science, where:

- In this study, the authors found that age differences in cognitive functioning within cohorts were often as large as those between cohorts, which calls into question the cohort explanation for age differences.

- In a study that I think has implications for personnel assessment, the authors found that raters of M.B.A. applications that had already given several high recommendations were less likely to do so later in the day.  Another bias to watch out for!


Last but not least, there was an article recently in the New York Times about how important referrals are for some organizations and the increased importance this places on social networking sites.  There was a reference to a study done in August of 2012 on source of hire that I hadn't seen before, which found that not only were referrals more likely to be hired, they had longer tenure--which is consistent with other research I've seen.  Unfortunately, no word about performance differences.

That's all for now!

Of Wal-Mart, veterans, and the nature of jobs

I think many of us were somewhat impressed with Wal-Mart's announcement that beginning on Memorial Day, they will offer a job to any honorably discharged veteran that applies within a year of ending active duty.

There's no question that the unemployment rate for U.S. vets is too high--consistently 20-30% above the rate for non-vets.

And it's refreshing to see the private sector step up after years of veteran hiring initiatives coming primarily from the government (which is not the driver of employment, at least not recently).

So kudos to them, it continues their public commitment to hiring veterans, and it's a bold gesture (although I wish the one-year time limitation was relaxed).


But what I found even more interesting than the announcement was the reaction from the veteran support community.  For example, Yana Walton from the Retail Action Project said (from the article linked above), "Workers are going to make sure that the kind of jobs that they’ve announced for veterans are the kind of jobs that workers need.  That means enough hours and living wages and jobs with benefits."

Other workers' rights groups have pointed out that these jobs are likely to include low wages and limited benefits.

Also interesting were the competing stories for why Wal-Mart likes veterans.  Wal-Mart's U.S. CEO, Bill Simon, said veterans have a record of performing under pressure, are quick leaders, and team players.  It's that last claim that is reflected in a response from Dr. Nelson Lichtenstein, a labor historian at the University of California, Santa Barbara, who said, "'They like military people because they have a sense of hierarchy and a commitment to the organization they are in.  And that's important to Wal-Mart.''

Frankly I find both comments to be slightly insulting to veterans, treating them as if they are a homogenous group.  I also find it insulting for anyone to assume that veterans are going to leap to embrace any job, regardless of the duties, pay, and benefits.

Some may see this as looking a gift horse in the mouth.  I look at it as supporting the dignity of workers and those seeking employment.

One of the most interesting debates that has occurred over the last several years of high unemployment has been about the nature of jobs that people want.  Certainly having a job and an income is better than nothing (at least in most cases), but many unemployed aren't interested in any job, they want a job that holds some promise--of good working conditions, of stability, and of promise of future opportunities.

Sadly, that may not be the types of jobs that are being created, at least not in the retail sector.  And it begs a larger conversation about education and skill mismatch, as well as how organizations define qualifications.

I also find this fascinating from an assessment standpoint.  To promise jobs to a single group of individuals without regard for assessing whether the individual applicants possess the qualifications the employer is seeking for a particular position is, well, frankly stupid from an organizational effectiveness standpoint.  This is either a leap of faith on the part of Wal-Mart, or (more likely) a calculated risk given their high turnover rate and the types of jobs veterans are likely to get.


It was with these thoughts rolling around in my brain that I watched my 5-year old's teacher open the door to her kindergarten students the other day.  I wondered what it was like for her day after day--the ups (children learning), the downs (noise)--and what attracted and kept her in the job.  I would guess that most teachers that stay teachers do so because they receive a significant amount of satisfaction from seeing students achieve and change in positive ways.  It's not an easy job, and they don't make a ton of money, so something must be keeping them there.

This made me think about why people apply for jobs at all.  Sure, for some people a job is a job--it almost doesn't matter what it is.  But I think this is probably the minority of job seekers, and it may be decreasing in prevalence.  And the historically (and I would argue misleading) high rate of job satisfaction may be losing its strength as an argument as people ask better questions like whether you would make the same career decision. 


So if more and more people are looking not only for a job, but a meaningful, rewarding one, what can we as assessment professionals do?

For starters, we can do more thinking about how to accurately describe jobs.  Regular readers know I'm a big fan of realistic job previews and that I think organizations generally do a terrible job at letting applicants know what they're getting themselves into.

Second, we can do a better job at assessing people's motivations for applying.  We've all seen the "why are you interested in this job?" question that seems entirely predictable and pointless but often yields surprisingly honest results.  If we proceed on the assumption that job performance and retention is driven at least in part by the match between a person's interests and motivations and what the job offers, we do a pretty sad job of measuring this.  Most of our measures deal with job-related knowledge or skills.  Don't get me wrong--this is really important.  But (in addition to personality) it leaves out a huge factor, namely the motivation for applying.

We should be providing more information, asking more questions, digging deeper, investigating new ways of using technology, and treating applicant motivation with a renewed sense of seriousness.

There may have been a time when measuring motivation to apply was considered secondary or unimportant, but that time has passed.

Research update

 
Cold/wet yet?  Well sit back, have some hot tea, and let's catch up on our research...

Let's start with the biggie: the December issue of IJSA.

-  Juggling selection quality and adverse impact continues to be tricky.  The authors in this article suggest an optimum combination.

- Reflecting results from the turnover literature, this study found a relationship between perceptions of promotion practices and organizational justice and job satisfaction.

- Speaking of justice, the authors of this study found that the relationship between perceptions of distributive justice and intentions to recommend an employer were moderated by applicant affect.

- Job seekers would to well to keep up their psychological well being and self-esteem (easier said than done, right?).

- Back to fairness.  It's reasonable to think that ethnic minority applicants may not perceive video resumes well (due to the increased salience of their minority status).  But at least in this study, that assumption was not supported--although it depended on ethnic identity and language proficiency.

-  Honestly, I'm not one for the pure concept of "multi-tasking": in my experience people perform in serial, not parallel.  But that doesn't stop people from researching the predictive validity of multi-tasking assessments.

- Faking of personality inventories has been one of the hottest topics in assessment for years, but is often framed as dichotomies.  This study looks to bring some needed complexity to the issue using qualitative data.

- We all know it can be challenging to get hiring manager to give up their unstructured interviews of questionable validity.  Interestingly, according to this study, the more a hiring manager has used unstructured interviews, the less open they are to change.  I don't know if this is a causality issue, a mediating variable issue, or more evidence of the inability to accurately judge one's abilities. 

- Need more evidence for discrimination that occurs during resume screening?  Here ya go.

- Understanding why certain individuals perform better during interviews is a needed area for study.  In this article, the authors demonstrate the importance of simply being ability to generate ideas, rather than analyzing the situational requirements.

- This study investigates another area needing more attention: the selection into medical training and education programs.  The authors found strong predictive support for SJTs but they came with lower face validity.

- Last in this issue is another interesting study, this time of 360-ratings of innovation.  Interestingly and unlike a lot of other research on self-perception, self-ratings were lower than overall observer ratings.  However, the situation got more complex when the authors separated and analyzed by level of self-rating.


Next the November issue of Journal of Applied Social Psychology:

- One of the most pervasive (yet bizarrely under-discussed) areas of discrimination in the workplace is age discrimination.  This study illustrates some of the stereotypes held of younger and older workers--by both groups.  (Spoiler alert: you'll find out how accurate some of these are in just a second)

- I've been waiting for this one, partly because I love hearing about how bizarre and non-face valid some puzzle-based interviews are.  This particular study was looking at perceptions of these interviews compared to a behavioral interview.  Results?  The puzzle-based interviews were consistently less popular.  Oh, did I mention that they didn't work as well?


How about the Winter issue of Personnel Psych?

-  Why do certain applicants withdraw from the recruitment process?  This study suggests a relationship with organizational identification.

- Okay, back to stereotypes about older workers.  Whereas the earlier study looked at what the stereotypes are, this one looks at whether they're true.  The answer: no, but for one: older workers are less willing to participate in training and career development.

- Turns out it's not just the unemployed that are frustrated by the job search process--currently employed individuals feel the same in many ways.  Boy it's too bad we don't have a giant shared database that is able to match job demands with worker abilities...wait...


The November issue of Journal of Applied Psychology has a couple gems:

- Evidence for the predictive validity of the external manifestations of personality as well as the associated implicit motives.

-  A reminder that what makes for effective leadership behavior depends on the culture.


The latest issue of Personnel Review has an interesting research article on utility analysis, where the authors reiterate how challenging it can be to communicate UA information (hint: carrot and stick approach may work best).


There are a couple good ones in the November issue of Psychological Science.

- Multiple-choice tests have been beat up in the past for being nothing more than tests of recognition (rather than productive retrieval).  This study presents evidence that refutes that assumption.  Go multiple-choice!

- Conspiracy theorists take note: governments may be less likely to use the assumed relationship between genetic testing and intelligence to pigeon-hole us into tracks.  Why?  Because, at least according to this study, there appears to be little evidence connecting the two.


Okay, this one is pretty cool--in a slightly scary way.  The authors were looking at the impact that virtual avatar attractiveness has on interview ratings.  Turns out our bias toward attractive people is so strong it extends to the virtual world!  Of course maybe I should have seen that coming...I mean, ever read a comic book?  (hat tip)


Still with me? Last but not least, some disturbing new evidence regarding significance testing and potential publication bias (hat tip).  I'm guessing most of you won't be surprised at the finding.


I don't know if I'll have another update before the end of the year, so if I don't, happy holidays to everyone!