Seriously? Just yesterday I did my research update, ending with a note that the December 2014 issue of the International Journal of Selection and Assessment should be out soon.
Guess what? It came out today.
So that means--you guessed it--another research update! :)
- First, a test of Spearman's hypothesis, which states that the magnitude of White-Black mean differences on tests of cognitive ability vary with the test's g loading. Using a large sample of GATB test-takers, these authors found support for Spearman's hypothesis, and that reducing g saturation lowered validity and increased prediction errors.
So does that mean practitioners have to choose between high-validity tests of ability or increasing the diversity of their candidate pool? Not so fast. Remember...there are other options.
- Next, international (Croatian) support for the Conditional Reasoning Test of Aggression, which can be used to predict counterproductive work behaviors. I can see this increasingly being something employers are interested in.
- Applicants that do well on tests have favorable impressions of them, while those that do poorly don't like them. Right? Not necessarily. These researchers found that above and beyond how people actually did on a test, certain individual differences predict applicant reactions, and suggest these be taken into account when designing assessments.
- Although personality testing continues to be one of the most popular topics, concerns remain about applicants "faking" their responses (i.e., trying to game the test by responding inaccurately but hopefully increase the chances of obtaining the job). This study investigates the use of blatant extreme responding, consistently selecting the highest or lowest response option, to detect faking, and looked at how this behavior correlated with cognitive ability, other measures of faking, and demographic factors (level of job, race, and gender).
- Next, a study of assessment center practices in Indonesia.
- Do individuals high in neuroticism have higher or lower job performance? Many would guess lower performance, but according to this research, the impact of neuroticism on job performance is moderated by job characteristics. This supports the more nuanced view that the relationship between personality traits and performance is in many cases non-linear and depends on how performance is conceptualized.
- ...which leads oh so nicely into the next article! In it, the authors studied air traffic controllers and found results consistent with previous studies--ability primarily predicted task performance while personality better predicted citizenship behavior. Which raises an interesting question: which version of "performance" are you interested in? My guess is for many employers the answer is both--which suggests of course using multiple methods when assessing candidates.
- Last but not least, an important study of using cognitive ability and personality to predict job performance in a three studies of Chilean organizations. Results were consistent with studies conducted elsewhere, namely ability and personality significantly predicted performance.
Okay, I think that's it for now!
Showing posts with label Criteria. Show all posts
Showing posts with label Criteria. Show all posts
Monday, October 27, 2014
Saturday, October 25, 2014
Research update
Okay, so it been a couple months, huh? Well, what say we do a research update then.
But before I dive in, I discovered something interesting and important. Longtime readers know that one of my biggest pet peeves is how difficult research articles are to get a hold of. And by difficult I mean expensive. Historically, unless you were affiliated with a research institution or were a subscriber, you had to pay exorbitant (IMHO) fees to see research articles. So imagine my pleasure when I discovered that at least one publisher--Wiley, who publishes several of the research journals in this area--now allows you to read-access for an article for as low as $6. Now that's only for 48 hours and you can't print it, but hey--that's a heck of a lot better than something like $30-40, which historically has been the case! So kudos.
Moving on.
Let's start with a bang with an article from the Autumn 2014 issue of Personnel Psych. A few years back several researchers argued that the assumption that performance is distributed normally was incorrect; and it got a bit of press. Not so fast, say new researchers, who show that when defined properly, performance is in fact more normally distributed.
For those of you wondering, "why do I care?" Whether we believe performance is normally distributed or not significantly impacts not only the statistical analyses performed on selection mechanisms but theories and practices surrounding HRM.
Moving to the July issue of the Journal of Applied Psychology:
- If you're going to use a cognitively-loaded selection mechanism (which in many cases has some of the highest predictive validity), be prepared to accept high levels of adverse impact. Right? Not to fast, say these researchers, who show that by weighting the subtests, you can increase diversity decisions without sacrifice validity.
- Here's another good one. As you probably know, the personality trait of conscientiousness has shown value in predicting performance in certain occupations. Many believe that conscientiousness may in fact have a curvilinear relationship with performance (meaning after a certain point, more conscientiousness may not help)--but this theory has not been consistently supported. According to these researchers, this may have to do with the assumption that higher scores equal more conscientiousness. In fact, when using an "ideal point" model, results were incredibly consistent in terms of supporting the curvilinear relationship between conscientiousness and performance.
- Range restriction is a common problem in applied selection research, since you only have performance data on a subset of the test-takers, requiring us to draw inferences. A few years back, Hunter, Schmidt, and Le proposed a new correction for range restriction that requires less information. But is it in fact superior? According to this research, the general answer appears to be: yes.
Let's move to the September issue of JAP:
- Within-person variance of performance is an important concept, both conceptually and practically. Historically short-term and long-term performance variance have been treated separately, but these researchers show the advantage of integrating the two together.
- Next, a fascinating study of the choice of (and persistence in) STEM fields as a career, the importance of both interest and ability, and how gender plays an important role. In a nutshell, as I understand it, interest and ability seem to play a more important role in predicting STEM career choices for men than for women, whereas ability is more important in the persistence in STEM careers for women.
Let's take a look at a couple from recent issue of Personnel Review:
- From volume 43(5), these researchers found support for ethics-based hiring decisions resulting in improved work attitudes, include organizational commitment.
- From 43(6), an expanded conceptual model of how hiring supervisors perceive "overqualification", which I would love to see more research on.
Last but not least, for you stats folks, what's new from PARE?
- What happens when you have missing data on multiple variables?
- Equivalence testing: samples matter!
- What sample size is needed when using regression models? Here's one suggestion on how to figure it out.
The December 2014 issue of IJSA should be out relatively soon, so look for a post on that soon!
But before I dive in, I discovered something interesting and important. Longtime readers know that one of my biggest pet peeves is how difficult research articles are to get a hold of. And by difficult I mean expensive. Historically, unless you were affiliated with a research institution or were a subscriber, you had to pay exorbitant (IMHO) fees to see research articles. So imagine my pleasure when I discovered that at least one publisher--Wiley, who publishes several of the research journals in this area--now allows you to read-access for an article for as low as $6. Now that's only for 48 hours and you can't print it, but hey--that's a heck of a lot better than something like $30-40, which historically has been the case! So kudos.
Moving on.
Let's start with a bang with an article from the Autumn 2014 issue of Personnel Psych. A few years back several researchers argued that the assumption that performance is distributed normally was incorrect; and it got a bit of press. Not so fast, say new researchers, who show that when defined properly, performance is in fact more normally distributed.
For those of you wondering, "why do I care?" Whether we believe performance is normally distributed or not significantly impacts not only the statistical analyses performed on selection mechanisms but theories and practices surrounding HRM.
Moving to the July issue of the Journal of Applied Psychology:
- If you're going to use a cognitively-loaded selection mechanism (which in many cases has some of the highest predictive validity), be prepared to accept high levels of adverse impact. Right? Not to fast, say these researchers, who show that by weighting the subtests, you can increase diversity decisions without sacrifice validity.
- Here's another good one. As you probably know, the personality trait of conscientiousness has shown value in predicting performance in certain occupations. Many believe that conscientiousness may in fact have a curvilinear relationship with performance (meaning after a certain point, more conscientiousness may not help)--but this theory has not been consistently supported. According to these researchers, this may have to do with the assumption that higher scores equal more conscientiousness. In fact, when using an "ideal point" model, results were incredibly consistent in terms of supporting the curvilinear relationship between conscientiousness and performance.
- Range restriction is a common problem in applied selection research, since you only have performance data on a subset of the test-takers, requiring us to draw inferences. A few years back, Hunter, Schmidt, and Le proposed a new correction for range restriction that requires less information. But is it in fact superior? According to this research, the general answer appears to be: yes.
Let's move to the September issue of JAP:
- Within-person variance of performance is an important concept, both conceptually and practically. Historically short-term and long-term performance variance have been treated separately, but these researchers show the advantage of integrating the two together.
- Next, a fascinating study of the choice of (and persistence in) STEM fields as a career, the importance of both interest and ability, and how gender plays an important role. In a nutshell, as I understand it, interest and ability seem to play a more important role in predicting STEM career choices for men than for women, whereas ability is more important in the persistence in STEM careers for women.
Let's take a look at a couple from recent issue of Personnel Review:
- From volume 43(5), these researchers found support for ethics-based hiring decisions resulting in improved work attitudes, include organizational commitment.
- From 43(6), an expanded conceptual model of how hiring supervisors perceive "overqualification", which I would love to see more research on.
Last but not least, for you stats folks, what's new from PARE?
- What happens when you have missing data on multiple variables?
- Equivalence testing: samples matter!
- What sample size is needed when using regression models? Here's one suggestion on how to figure it out.
The December 2014 issue of IJSA should be out relatively soon, so look for a post on that soon!
Sunday, November 06, 2011
How important is assessment, really?
Prepare for a little blasphemy.
Over the last few months my job--and focus--has changed dramatically. Historically I've been a "testing" guy. Question about job analysis? Item writing? I'm there.
Then, a few years ago, I started managing a team that did a more than assessment--a lot more. In fact, even though assessment is in their job description, the team spends most of their time counseling supervisors on performance management. Of course some of this is because the testing workload is down, but it's also a function of demand for advice in this area.
In July we found out our department's budget was being cut; to the tune of about 10%. We adjusted and tightened our belts, but in the end it wasn't enough and we had to plan for layoffs. I was recruited to be one of the coordinators of said layoff, and thus began the dramatic work shift.
That's all a really long way of saying that my focus lately has not been on recruitment and hiring. I've been thinking a lot more about what keeps people going in difficult times. Sure, the KSAOs they bring to the table are important, but other things raise in importance during times of uncertainty and lack of control.
Which got me to thinking: how important IS assessment really? Even at our best, we can predict only about a third of individual job performance. What's going on with that other 2/3rds?
You're probably familiar with models of job performance, so I won't bore you. Suffice to say that a lot goes into job performance. So that person you hired that aced your assessments? Not guaranteed to be super star. If they end up being supervised by an incompetent manager, their inner greatness may never reach the surface. If they have a death in the family, you better believe their focus is not going to be on work for a while and job performance may not be at maximum.
Let's think about job performance as a pie. Top-notch assessment can predict about a third of that pie. What else is in that pie--and more importantly, how big are the slices? Things like:
- motivation
- supervision style
- role clarity
- co-worker support
- mood
- resources
- performance feedback
- stress
(I tried with no luck to track down a comprehensive path model, maybe one of you can point one out)
Now we can't control all of these things (although as HR professionals we certainly can consult on a lot of them--clear duty statements, supervisor training and accountability, engagement surveys, etc.), but what we CAN do is take the rigor we bring to the study of assessment and apply it to other aspects related to job performance. If you look at HR research outside of assessment I think you'll find that the level of analysis is, shall we say, sometimes lacking.
Don't get me wrong: assessment will always be important. The legal rationale is IMHO the least compelling. Instead, there is proven, substantial, utility in implementing best practices for employee assessment.
But lately I can't help but thinking: Are we spending too much time thinking about--and studying--how we bring people in to an organization, and not enough time thinking about what happens once they get there?
Subscribe to:
Posts (Atom)