Showing posts with label Multiple-choice. Show all posts
Showing posts with label Multiple-choice. Show all posts

Saturday, December 01, 2012

Research update

 
Cold/wet yet?  Well sit back, have some hot tea, and let's catch up on our research...

Let's start with the biggie: the December issue of IJSA.

Juggling selection quality and adverse impact continues to be tricky.  The authors in this article suggest an optimum combination.

- Reflecting results from the turnover literature, this study found a relationship between perceptions of promotion practices and organizational justice and job satisfaction.

- Speaking of justice, the authors of this study found that the relationship between perceptions of distributive justice and intentions to recommend an employer were moderated by applicant affect.

- Job seekers would to well to keep up their psychological well being and self-esteem (easier said than done, right?).

- Back to fairness.  It's reasonable to think that ethnic minority applicants may not perceive video resumes well (due to the increased salience of their minority status).  But at least in this study, that assumption was not supported--although it depended on ethnic identity and language proficiency.

-  Honestly, I'm not one for the pure concept of "multi-tasking": in my experience people perform in serial, not parallel.  But that doesn't stop people from researching the predictive validity of multi-tasking assessments.

- Faking of personality inventories has been one of the hottest topics in assessment for years, but is often framed as dichotomies.  This study looks to bring some needed complexity to the issue using qualitative data.

- We all know it can be challenging to get hiring manager to give up their unstructured interviews of questionable validity.  Interestingly, according to this study, the more a hiring manager has used unstructured interviews, the less open they are to change.  I don't know if this is a causality issue, a mediating variable issue, or more evidence of the inability to accurately judge one's abilities. 

- Need more evidence for discrimination that occurs during resume screening?  Here ya go.

- Understanding why certain individuals perform better during interviews is a needed area for study.  In this article, the authors demonstrate the importance of simply being ability to generate ideas, rather than analyzing the situational requirements.

- This study investigates another area needing more attention: the selection into medical training and education programs.  The authors found strong predictive support for SJTs but they came with lower face validity.

- Last in this issue is another interesting study, this time of 360-ratings of innovation.  Interestingly and unlike a lot of other research on self-perception, self-ratings were lower than overall observer ratings.  However, the situation got more complex when the authors separated and analyzed by level of self-rating.


Next the November issue of Journal of Applied Social Psychology:

- One of the most pervasive (yet bizarrely under-discussed) areas of discrimination in the workplace is age discrimination.  This study illustrates some of the stereotypes held of younger and older workers--by both groups.  (Spoiler alert: you'll find out how accurate some of these are in just a second)

- I've been waiting for this one, partly because I love hearing about how bizarre and non-face valid some puzzle-based interviews are.  This particular study was looking at perceptions of these interviews compared to a behavioral interview.  Results?  The puzzle-based interviews were consistently less popular.  Oh, did I mention that they didn't work as well?


How about the Winter issue of Personnel Psych?

-  Why do certain applicants withdraw from the recruitment process?  This study suggests a relationship with organizational identification.

- Okay, back to stereotypes about older workers.  Whereas the earlier study looked at what the stereotypes are, this one looks at whether they're true.  The answer: no, but for one: older workers are less willing to participate in training and career development.

- Turns out it's not just the unemployed that are frustrated by the job search process--currently employed individuals feel the same in many ways.  Boy it's too bad we don't have a giant shared database that is able to match job demands with worker abilities...wait...


The November issue of Journal of Applied Psychology has a couple gems:

- Evidence for the predictive validity of the external manifestations of personality as well as the associated implicit motives.

-  A reminder that what makes for effective leadership behavior depends on the culture.


The latest issue of Personnel Review has an interesting research article on utility analysis, where the authors reiterate how challenging it can be to communicate UA information (hint: carrot and stick approach may work best).


There are a couple good ones in the November issue of Psychological Science.

- Multiple-choice tests have been beat up in the past for being nothing more than tests of recognition (rather than productive retrieval).  This study presents evidence that refutes that assumption.  Go multiple-choice!

- Conspiracy theorists take note: governments may be less likely to use the assumed relationship between genetic testing and intelligence to pigeon-hole us into tracks.  Why?  Because, at least according to this study, there appears to be little evidence connecting the two.


Okay, this one is pretty cool--in a slightly scary way.  The authors were looking at the impact that virtual avatar attractiveness has on interview ratings.  Turns out our bias toward attractive people is so strong it extends to the virtual world!  Of course maybe I should have seen that coming...I mean, ever read a comic book?  (hat tip)


Still with me? Last but not least, some disturbing new evidence regarding significance testing and potential publication bias (hat tip).  I'm guessing most of you won't be surprised at the finding.


I don't know if I'll have another update before the end of the year, so if I don't, happy holidays to everyone!

Sunday, February 26, 2012

February 2012 research update: Facebook; hope; 3-option items; cognitive ability and racism

So this month the big buzz is over a research article in JASP on using Facebook profiles to judge personality and predict job performance. So let's tackle that one first:

Kluemper, et al. had three trained university students judge Big Five personality factors (using the IPIP) based on over 200 public Facebook profiles of other students. A much smaller sample (56) was used to determine links between evaluator judgments and job performance as rated by supervisors.

So what did they find? Well, among other things:

(1) inter-rater reliability ranged from .48 to .72 which seems low to me but apparently is typical for other-ratings of personality;

(2) two of the other-ratings (emotional stability and agreeableness) significantly correlated with supervisory ratings (around .30), as did one of the self-ratings (extraversion), suggesting to me the two methods might vary in constructs being measured;

(3) the same two other-ratings added incremental validity beyond self-ratings, whereas the opposite was not true.

To their credit, the authors caution readers (those that actually go beyond the mainstream press articles) about using their results to support hiring decisions based on Facebook. In fact, I'd like to quote them:

"Our findings should not be used by organizations as unbridled support for using SNWs [Social Networking Websites] in employment selection. Without more evidence of criterion-related validity and comparability with established employment selection methods, the use of SNW information for hiring purposes is tenuous. In addition to the potential for employment discrimination, there are privacy rights and ethical issues associated with accessing personal information. Clearly, research investigations of such issues lag current informal HR practices."

So bravo to them for researching this issue, and double-bravo (that's a technical term) for cautioning those with Facebook fever.


Let's move to other research out there...

The March issue of IJSA is chalk-full of great research, so let's take a look:

- Much hay is made over the "type" of validity exhibited by cognitive ability tests (don't get me started). To the extent this distinction makes sense to you, you might enjoy Frank Schmidt's argument that ability tests in certain instances can demonstrate content validity. The article is followed by several commentaries and a response.

- Next up is O'Neill, et al. with a critical review of Stevens and Campion's Teamwork-Knowledge, Skills, and Ability Test.

- Reeder, et al. investigate individual differences as they relate to the perception of cognitive ability tests.

- Now here's somethin': Edwards, et al. argue that the three-option multiple choice item is underutilized. This should be read by everyone who has nightmares about writing distractors.

- Hoffman and Meade argue that score differences across assessment center exercises reflect true differences rather than measurement artifact.

- Got hope? Zysberg shows that, through problem-solving-oriented coping, hope is related to success in a selection process.


We already know that people generally aren't very good at accurately describing their skills and abilities, but in the March Psychological Bulletin, Freund and Kasten provide an illuminating meta-analysis indicating that while the relationship between self-estimated and psychometrically measured cognitive ability is modest (.33), it varies depending upon scales and dimensions.


For anyone measuring contextual performance, consider the relationship between role expectations and OCBs as described by Dierdorff, et al.

Speaking of OCBs, Nielsen et al.'s study suggests when measuring OCB expression in a group setting, consider the level of task interdependence.

On the other hand, if you are interested in job performance ratings, be aware that there may be gender bias, and it differs in direction between performance ratings and promotability ratings (with the latter favoring males), according to a recent meta-analysis by Roth, et al.


Last but definitely not least, research by Hodson and Busseri suggest that individuals lower in cognitive ability may be predisposed to exhibit more racism. This suggests that using ability tests may not only increase the validity of your selection process but lower your chances of discriminatory behavior.

****

On a side/editorial note, I find it fascinating and somewhat frustrating that the research energy still seems to be about teasing out major constructs such as cognitive ability and personality. As a practitioner, I gotta tell ya I'm happy when hiring supervisors use any sort of structured assessment beyond their standard interview. I'd love to see more energy behind increasing the validity of the entire selection process. I doubt I'm the only one that feels that way.