Research update

I can't believe it's been three months since a research update.  I was waiting until I got critical mass, and with the release of the September issues of IJSA, I think I've hit it.

So let's start there:

- Experimenting with using different rating scales on SJTs (with "best and worst" response format doing the best of the traditional scales)

- Aspects of a semi-structured interview added incremental validity over cognitive ability in predicting training performance

- Studying the use of preselection methods (e.g., work experience) prior to assessment centers in German companies

- The proposed general factor of personality may be useful in selection contexts (this one was a military setting)

- Evidence that effective leaders show creativity and political skill

- Investigating the relationship (using survey data) between personality facets and CWBs (with emotional stability playing a key role)

- Corrections for indirect range restriction boosted the upper end of structured interview validity substantially

- A method of increasing the precision of simulations that analyze group mean differences and adverse impact

- A very useful study that looked at the prediction of voluntary turnover as well as performance using biodata and other applicant information, including recruitment source, among a sample of call center applicants.  Reuslts?  Individuals who had previously applied, chose to submit additional information, were employed, or were referrals had significantly less voluntary turnover.



Moving on...let's check out the May issue of JAP; there are only two articles but both worth looking at:

- First, a fascinating study of the firm-level impact of effective staffing and training, suggesting that the former allow organizations greater flexibility and adaptability (e.g., to changing financial conditions).

- Second, another study of SJT response formats.  The researchers found, using a very large sample, the "rate" format (e.g., "rate each of the following options in terms of effectiveness") to be superior in terms of validity, reliability, and group differences.


Next, the July issue of JOB, which is devoted to leadership:

- You might want to check out this overview/critique of the various leadership theories.

- This study suggests that newer models proposing morality as an important component of leadership success have methodological flaws.

- Last, a study of why Whites oppose affirmative action programs

Let's move to the September issue of Industrial and Organizational Psychology:

- The first focal article discusses the increasing movement of I/O psychology to business schools.  The authors found evidence that this is due in large part to some of the most active and influential I/O researchers moving to business schools.

- The second is about stereotype threat--specifically its importance as a psychological construct and the paucity of applied research about it.


Coming into the home stretch, the Summer issue of Personnel Psych:

- The distribution of individual performance may not be normal if, as these researchers suggest, "star performers" have emerged

- Executives with high levels of conscientiousness and who display transformational leadership behavior may directly contribute to organizational performance

Rounding out my review, check out a few recent articles from PARE:

- I'm not even gonna attempt to summarize this, so here's the title: Multiple-Group confirmatory factor analysis in R – A tutorial in measurement invariance with continuous and ordinal indicators

- Improving exploratory factor analysis for ordinal data

- Improving multidimensional adaptive testing


Last but not least, it's not related to recruitment or assessment, but check out this study that found productivity increases during bad weather :)

That's all folks!

Research update: September, 2013

Okay, it's mega research update time!

First off, the September IJSA; lots of good stuff, including:

- a constructed response multimedia test for entry-level police resulted in minor ethnic group differences

- panel interviews once again prove their superiority (also: more on interview reliability)

- further analysis of the Hogan Personality Inventory with a Spanish sample

- how to applicants form impressions of person-organization fit?  This study suggests contextual factors may be more important than interview content

- circumplex traits (combinations of personality factors) may predict counterproductive work behaviors better than simple FFM scores

- speaking of CWBs, conditional reasoning tests may not be the best predictor of them

- last but not least, what looks to be a good overview of competency modeling


Next up, the September JAP:

- an interesting, large study of the impact of candidate reactions on test scores, organizational perception, and criterion-related validity

- a study of the dynamics of the job search process and the impact of efficacy and focus

- highlighting certain factors during an interview may reduce discrimination toward pregnant applicants


Next, the Autumn 2013 Personnel Psychology:

- first, an important study of self-efficacy that suggests it is a product of past performance and not necessarily a predictor of future performance (free right now!)

- second, a study indirectly on selection that suggests that age diversity in work groups leads to more emotion regulation


Let's move on to the September JASP:

- okay, this may be a bit of a stretch, but if you're considering interviewing for a position as a dentist or a lawyer, make sure you suit up

- knowledge of service encounters predicts service effectiveness (and is related to conscientiousness)

- can use of biodata instruments result in adverse impact?  This study suggests so, but also suggests that removal of problematic items has no impact on validity


Starting to wrap up, let's move to the October JOB:

- perceptions of the fairness of promotion practices is one of those "bubbling beneath the surface" issues in most organizations.  This study found that perceptions are impacted by having been promoted in the past, organizational commitment, and ego defensiveness.  Good stuff.

- do more creative sales agents produce higher sales?  Perhaps only when there is a high quality of leader-member exchange.

- is validity generalization overgeneralized? (say that five times fast)  These folks seem to think so.


In the home stretch, from the September Psychological Science:

- older employees may have lower average cognitive performance, but it's more consistent

- spatial ability has a valuable role to play in the development of creativity, and can predict things like patents and publications


Second to last, for you stats geeks out there, a study that suggests that t-tests can be used reliably with small samples, thank you very much


Finally, something that has nothing to do with selection but is a nominee for the 2013 HR Tests Coolest Study Award, and something we all are very familiar with: time bandits (no, not the movie).

Research update

Okay, I'm just gonna say it.  There is an insane amount of research out there directly or indirectly related to our field.  I'll be honest, it's a little daunting thinking about reading and processing all of it.

Luckily, you have me to overly simplify it for you so you can incorrectly describe it to others.  So let's jump right in!

First up, the June issue of IJSA:

-  Fairness perceptions matter in web-based selection too!

-  The predictive validity of conscientiousness is moderated by self-enhancement

-  Speaking of conscientiousness, it's a better predictor of performance in routinized jobs than those with complex cognitive requirements

-  Psychological hardiness predicts adaptability in military leaders.  This study is awesome also for showing a negative relationship between SAT scores and West Point performance.

-  Recruiters, listen up: this study found that contingent workers converted to full-time status performed as well as referral and online hires, but not as well as those sourced internally (bookmark this one, there's a dearth of recruitment research)

- Scoring biodata: empirical keying reigns supreme over rational and quasi-rational approaches.  Quasi-rational?  Sounds like me most of the time.

- Personality and job performance: interactions are important beyond main effects


Okay, next, the May issue of JAP:

- Adaptive decision making in military leaders: both the brain and the mind are important (okay you monists out there, chill out)

- Are you a MANOVA (wo)man?  Then read this.

- Proactive leaders set more challenging goals and have higher sales performance--assuming they have the trust of their subordinates.


Next, the April issue of JOB.  Just one study, albeit interesting, in which female evaluators were less likely to recommend hiring or promoting Asian (versus White) applicants into jobs requiring social skills


Speaking of JOB, how about the May issue?

- Have you been calling for more research on calling?  Your wish is granted.

- Curious about the concept of curiosity? (okay, I'll stop)  Looks like it can be predictive of job performance above and beyond traditional cognitive and non-cognitive predictors.  I'm gonna assume it varies with job, but I think the authors are probably right that it will increase in importance over time.

- When selecting for teams that may experience crisis situations (e.g., nuclear power plant crews), consider homogeneity--not mean levels--of positive affect.

- Moderate levels of supervisory structure combined with high levels of consideration lead to lowest levels of CWB's

- The dark side of OCBs

- The light side of OCBs

- Dark side and light side working together in a picture that has nothing to do with selection


Okay, onwards and upwards: one from the April issue of JASP, on generalized self-efficacy, work-related self-efficacy, and job-related outcomes


And one from the May issue:  Sensation seeking and need for structure predict military field exercise performance


How about a little EI research?  Haven't had a lot of that lately.  Here's a piece from the May issue of Journal of Management that found emotion management ability to be a valid predictor of job performance.


Let's look in the May Psychological Science:

- Where we find a fascinating study that demonstrates assessment of profound cognitive abilities at a young age predicts outstanding contributions in many adults

- Another interesting study of how an individual responds to daily stressors (which I would posit are differentially found in various jobs) has long-term consequences for their mental health

- High schoolers with high math and verbal skills are less likely to choose STEM careers than those with math skills but moderate verbal skills

- Okay, check this one out: mindfulness training improved GRE reading comp scores as well as working memory.  Implications for personnel assessment??


Shifting to the May Psychological Bulletin:

- Rorschach, anyone?

- Believing that individuals are malleable, rather than fixed ("implicit theory") predicts goal achievement.  Implications for leaders?


Last but not least, let's hook up with our friends at PARE and see what they're up to:

- Into Excel?  Check out this article on forest plots

- Or maybe factor analysis is your thing.


That's all folks!  Hope your brain is spinning like mine!

May mega research update

It's time once again for the monthly research round-up.  So let's dive right in:

The June International Journal of Selection and Assessment doesn't disappoint; let's take a look:

- More evidence of the link between personality variables and CWBs; this time with concurrent data in China

- Dovetailing nicely with a post I've been working on regarding promotional testing, this research indicates some interesting characteristics of internal test takers

- Why are supervisors open to behavioral interviews but shun discussion of "structure"?  Looks like how we communicate about them plays a big role.

- More research on self-efficacy, this time teasing apart the concept a bit.

- Always a popular topic: applicant reactions to selection mechanisms.  This time with a sample from Saudi Arabia.

- Speaking of applicant reactions...how about another study?  This one comparing U.S. and Vietnamese college students.  By the way, not surprisingly work samples came out a winner in both of these studies.

- Next, a fascinating study of a hidden bonus to UIT: despite the cheating element, it likely increases your candidate pool and eventually performance outcomes

- Speaking of response distortion, here's another study, this time of military cadet selection using personality inventories

- Okay, one more on inflation.  This time a study of Chinese applicants--no difference compared to American samples.

- Back in March I wrote about a study Jeremy Bernerth published in J.A.P.  that got a lot of attention.  This time, Bernerth studied ethnic differences and found minority status was negatively related to credit scores.


Moving on to the summer issue of Personnel Psychology:

-  The "file drawer problem" is the theory that nonsignificant results are less likely to get published.  According to this study, that appears unlikely.  But IMHO looking at all correlations is different than looking at the correlations key to one's hypothesis(es)...

- Back to faking (that may be this post's theme!), can response elaboration reduce faking on biodata items?  This study suggests so.  Although I'm left wondering...what was the impact on validity?

- Speaking of biodata, there are various ways of keying these items.  This research suggests the best method depends on your sample size, although rational keying performed the worst.


How about the May issue of Journal of Applied Psychology?

-  Well this is interesting...Chad Van Iddekinge and his colleagues have provided an updated meta-analysis on the criterion-related validity of integrity tests.  What did they find? Well, the results appear to be less promising than those published previously (e.g., corrected r=.18 for job performance).  Much like SIOP's research journal, this time J.A.P. published several commentaries in response to the study that...well, let's just say a debate ensued about the analysis...

- The Dark Triad.  It sounds like something in a Dan Brown novel.  But in this meta-analysis the authors show that personality characteristics that make up this triad (Machiavellianism, narcissism, and psychopathy) explain some variance in CWBs.

- Why are some people more proactive in seeking career goals than others?  It's an important and under-researched question.  In this study the authors show that part of the explanation lies in "future work selves", or how people's hopes and aspirations as they relate to work.

- Think self-reports of CWBs are biased?  Perhaps not, according to this new study.

- Interested in what causes proactive customer service behavior?  According to this multi-national study, self-efficacy is a key (along with service climate).

- Why do some leaders engage in more self-interested behavior than others?  Perhaps not surprisingly, it appears due in part to the strength of their moral identity.


The May issue of the Journal of Applied Social Psychology has a couple gems...

- Hey, look, turns out being sensitive to your subordinates pays off.  Talk about a lesson that needs frequent repeating...

- And that's it.  Oh, wait, just this little study about using Facebook profiles to predict job performance...that I wrote about before....available in FULL right now...


Okay, getting to the end...The May/June issue of HRM:

- An interesting study of adverse impact in promotion decisions for managers in a Fortune 500 retailer.  The authors compared three methods (top-down assessment, assessment centers, and  multisource appraisal) and the results demonstrate how complex these analyses are!

- Speaking of complex.  Think that successful job postings on the web is just fancy graphics?  Think again--it still involves some classic factors like the labor market, firm reputation, and compensation incentives.  The more things change...

- Identifying future leaders.  There are few other issues that are as important for most organizations.  Yet how exactly to do it eludes many.  These authors propose a model that focuses on four main features: analytical ability, learning agility, drive, and emergent leadership.


Finally, a few from PARE:

- Does item order impact response anxiety?  Not according to this study.

- What's that?  How do we use a new jacknife procedure to eliminating items and improve structural equation modeling?  You're in luck.

- Looks like a lot of research rely on beta weights when interpreting and reporting multiple linear regression results.  But there's so much more...


Happy reading!

March IJSA: So much good stuff

The March 2011 issue of the International Journal of Selection and Assessment is out, and it's a doozie. Check it out:

- Beaty, et al. present evidence that unproctored Internet tests (noncognitive: personality and biodata) generally had similar criterion-related validities across a spectrum of job performance criteria compared to the same administered in a proctored setting. Mmmm....UIT...

- Generally when people re-take a cognitive ability test, they do better. But do they do better on other tests of cognitive ability? Matton, et al. describe a study that looked at just that and found the answer to be "no."

- One advantage that is frequently claimed about personality inventories is that usage results in less adverse impact compared to, say, ability or knowledge tests. But might the AI depend on how hires are made (e.g., top-down, compensatory)? Turns out the answer is yes--at least according to some results by Risavy and Hausdorf.

- Hey, government agencies, still debating whether to put more resources into your career web portal? Maybe this will convince you. Selden and Orenstein show that governments with more usable portals as well as better available content not only attract more applicants per opening, but have less voluntary turnover of new hires.

- With advances in technology and changes to the work environment, clerical jobs have changed a lot over the last 30 years and the old ways of selecting for these jobs (namely g-loaded tests such as perceptual speed and verbal ability) likely need to be re-thought...right? Well, not so much, at least according to a meta-analysis by Whetzel and her colleagues. In fact, the criterion-related validity values met or exceeded those found 30+ years ago. The more things change...

- De Goede, et al. present the results of a study that explores the relationship between P-O fit and organizational websites but include the concept of person-industry fit. One implication: if you're trying to attract a more diverse group of candidates, work on making your portal more attractive.

- We spend a lot of time trying to make sure interviews are loaded with job-relevant content. But how much attention do we pay to the impact of impression management tactics on the part of applicants? Huffcutt's results make a compelling argument that we ignore the latter to our detriment as it may have more to do with interview ratings than the job-relevant content.

- How does one determine managerial potential? Well, it depends who you ask. Thomason, et al. present results that indicate when supervisors are asked, they focus on task-based personality traits (e.g., conscientiousness), whereas peers focus on contextual traits such as agreeableness. Given that leadership is ultimately about achieving things through subordinates, I wonder what we should be paying attention to....hmmm...how about both?

- Thinking about using self-ratings of political skill as part of the application process? I can certainly see situations where this skill may be helpful, but might this method be susceptible to inflation? Not so much, at least according to results from Blickle, et al.

- Last but definitely not least, Carless & Hetherington with some data on the impact of recruitment timeliness on applicant attraction. The longer we make applicants wait, the less attracted to the organization they will be, right? Not so fast. According to this research, it is perceived timeliness that matters, not actual timeliness (hence the importance of communication). In addition, this relationship is partially mediated by job and organizational characteristics.

How to hire an attorney

What's the best way for an organization to hire an attorney with little job experience? What should they look for? LSAT scores? Law school grades? Interviewing ability? A multi-year project that issued its final report in 2008 gives us some guidance. And while the study focused on ways law schools should select among applicants, it's also instructive for the hiring process. (By the way, individuals looking for personal representation may find the following interesting as well.)

Recall that the formalization of the "accomplishment record" approach occurred in 1984 with a publication by Leaetta Hough. She showed, using a sample of attorneys, that scores using this behavioral consistency technique correlated with job performance but not with aptitude tests or grades, and showed smaller ethnic and gender differences.

But in my (limited) experience, many hiring processes for attorneys have consisted of a resume/application, writing sample, and interview. Is that the best way to predict how well someone will perform on the job?

Assessment research would strongly point to cognitive ability tests being high predictors of performance for cognitively complex jobs. This is at least part of the logic of hurdles like the Law School Admissions Test (LSAT), a very cognitively-loaded assessment. When you're at the point of hire, however, LSAT scores are relatively pointless. Applicants have--at the very least--been through law school, and may have previous experience (such as an internship) you can use to determine their qualifications.

So what we appear to have at the point of hire is a mish-mash of assessment tools, relying heavily on un-proven filters (e.g., resume review) followed by a measure of questionable value (the writing sample) and the interview, which in many cases isn't conducted in a structured way that would maximize validity.

So what should we do to improve the selection of attorneys (besides using better interviews)? Some research done by a psychology professor and law school dean at UC Berkeley may offer some answers.

The investigators took a multi-phase approach to the study. The first part resulted in 26 factors of lawyer effectiveness--things like analysis and reasoning, writing, and integrity/honesty. In the second phase they identified several off-the-shelf assessments they wanted to investigate for usefulness, and they developed three new assessments--a situational judgment test (SJT), a biodata measure (BIO), and other measures, including optimism and a measure of emotional intelligence (facial recognition). In the final phase, they administered the assessments online to over 1,000 current and former law students and looked at the relationship between predictors and job performance (N for that part of about 700, using self, peer, and supervisor ratings).

Okay, so enough with the preamble--what did they find?

1) LSAT scores and undergraduate GPA (UGPA) predicted only a few of the 26 performance factors, mainly ones that overlapped with LSAT factors such as analysis and reasoning, and rarely higher than r=.1. Results using first-year law school GPA (1L GPA) were similar.

2) The scores from the BIO, SJT, and several scales of the Hogan Personality Inventory predicted many more dimensions of job performance compared to LSAT scores, UGPA, and 1L GPA.

3) The correlations between BIO and SJT and job performance were substantially higher-- in the .2-3 range compared to LSAT, UGPA, and 1L GPA. The BIO measure was particularly effective in predicting a large number of performance dimensions using multiple rating sources.

3) In general, there were no race and gender subdifferences on the new predictors.

These results strongly suggest that when it comes to hiring attorneys with limited work experience, organizations would be well advised to use professionally developed assessments, such as biodata measures, situational judgment tests, and personality inventories, rather than rely exclusively on "quick and dirty" measures such as grades and LSAT scores. Yet another proof of the rule that the more time spent developing a measure, the better the results.


On a final note, several years back I did a small exploratory study looking at the correlation between law school quality and job performance. I found two small to moderate results: law school quality was positively correlated with job knowledge, but negatively correlated with "relationships with people."

References:
Here is the project homepage.
You can see an executive summary of the final report here.
A listing of the reports and biographies is here.
The final report is here.

Lessons from NYC Fire case - part 2

Part 2 of 2

Last time I discussed five important lessons we can take away from recent rulings in the Vulcan v. City of New York case. In this post I'll review the remaining lessons and also discuss the relief order.

----

6) The city failed to provide sufficient evidence that the exam(s) tested for a sufficient number of the critical KSAs. They also failed to explain why they chose not to measure several KSAs identified as critical.

Lesson: the courts do not require employers to measure every single critical KSA. But there is an expectation that employers attempt to measure a sufficient number that represent a significant portion of the job requirements. In this case, that included non-cognitive abilities such as resistance to stress, teamwork, and conscientiousness, that were not measured.

7) The city failed to adequately consider how to measure a significant number of essential KSAs. While some of their concerns were valid (e.g., structured interviews for all applicants would be an operational nightmare), there are many different forms of testing that should have been considered, including situational judgment tests (SJTs) and biodata, which can be used to measure non-cognitive components.

Lesson: triers of fact expect employers to be up on the various assessment methods available and be able to explain why they chose not to use certain ones. This includes tests that are relatively easy to develop (e.g., SJTs) as well as ones that require substantial resources and statistical expertise (e.g., biodata).

8) The city failed to conduct a reading level analysis on the exams to ensure that it was not "pointlessly high." The plaintiff introduced evidence suggesting the reading level was above 12th grade; in addition, it appeared to exceed the reading level of materials at the academy.

Lesson: never forget that every assessment method is in some sense measuring additional KSAs beyond those you intend. For written exams, reading comprehension is always a requirement (barring accommodation). It's quite easy to conduct a reading level analysis (MS Word has it built in) to ensure that the level is reasonable and matches other job-related material.

9) The city failed to show that the cutoff scores (pass points) established for the exams were based on adequate rationale, namely "the necessary qualifications for the job of entry-level firefighter." Instead, the cutoff scores were based on operational need (the number of job openings expected). This is particularly important in multiple-hurdle selection processes such as in this case, where a failure on one exam component precludes an applicant from participating in the rest of the (potentially compensatory) assessment process.

Lesson: ultimately applicants have to pass the test(s) to be considered for employment. Cutoff scores should be established using the expertise of both SMEs and test developers and should be based on the minimum competency levels required upon entry to the job. At a minimum (and I would not rely solely upon this), the scores should be analyzed to identify any logical "break-points."

----

After ruling for the plaintiffs on both the adverse impact and disparate treatment claims, the judge issued a relief order on 1/10/10. In it, he imposes several things, including the following:

1) The city must develop a new testing procedure for entry-level firefighter in conjunction with the relevant parties. Following the development of the test, there will be a hearing to determine if this test should be used rather than the current test (developed in 2007 and not at issue in this litigation).

2) The court shall develop a process by which the approximately 7,400 applicants covered by this case can file a claim for monetary relief.

3) The city will identify 293 black candidates on the eligibility list and offer them priority hiring. (No quotas are being imposed, although the judge leaves this possibility open)

4) Retroactive seniority for those hired.

In addition, several other issues are up for debate, including the appointment of a special master or monitor, standards that will be relied upon in constructing the new exam, and the need for additional relief.

---

So what did we learn from all this? If you follow--fairly closely--best practices when developing and administering exams, you will be on solid ground defending them. If you don't, and your exam has a discriminatory effect, you may be called on it--and it's not a pleasant process. I'll leave you with this quote from the January ruling on disparate treatment:

"The history of the City's efforts to remedy its discriminatory firefighter hiring policies can be summarized as follows: 34 years of intransigence and deliberate indifference, bookeneded by identical judicial declarations that the City's hiring policies are illegal."

Want better prediction? Gather more data.

That's the bottom line from a study in the November 2009 issue of the Journal of Applied Psychology.

Oh & Berry looked at how adding personality ratings from peers and supervisors added incremental validity to self-ratings using a five-factor model measure. What were the results? Increases of 50-74% in operational validity across personality facets. They also looked at differential prediction of task and contextual performance (unfortunately those results weren't reported in the abstract). Bottom line? If you're using a personality assessment for promotions, strongly consider gathering data from co-workers.

Speaking of self-presentation, in the same issue Barrick et al. report the results of a meta-analysis of how self-presentation tactics (e.g., appearance, non-verbal behavior) impact interview ratings and later job performance. Results? "What you see in the interview may not be what you get on the job and...the unstructured interview is particularly impacted by these self-presentation tactics." An important reminder of how who the candidate seems to be impacts your assessment, and another reason to collect multiple sources of data.

There are a number of other great articles in this issue, such as:

How Major League Baseball CEO personalities impact important outcomes (like, um, winning).

How SJT and biodata measures add to the prediction of college student performance.

How personality scale validities change over time among a group of medical students.

Differences among letters of recommendation in academia between genders.

September '09 IJSA

The September 2009 issue of IJSA (International Journal of Selection and Assessment) is chalk full of good stuff. Let's dive in.

1) An important update of the "guidelines and ethical considerations for assessment center operations"--a must for anyone interested in the appropriate use of assessment centers.

2) Speaking of assessment centers, here's a meta-analysis of how they correlate with cognitive ability and personality, as well as the proper way to weight the results.

3) Speaking of cognitive ability, curious about the correlation between ability and faking? Check out this large-sample study of faking on a biodata measure.

4) Worried about what your applicants think of your selection method? Frame it as select in (accept) rather than select out (reject).

5) Want to make sure your raters are rating accurately? You may want to re-think stocking your panel with agreeable people (sounds like a lot of fun for the exam analyst!).

6) Before you put the finishing touches on your new online job application system, make sure you pay attention to its features, user friendliness, and efficiency. I like to think of this as "Googley."

7) Looking for a measure of person-job fit that relates equity of contribution to reward? Check this out.

That's all for now!

Hybrid tests and the June '09 IJSA

What do you get when you combine a structured interview with a performance assessment? Perhaps some sort of hybrid with pieces from both sides. In the June 2009 issue of the International Journal of Selection and Assessment we find out more.

Morgeson et al. describe the development of a "performance interview" that combines a structured interview with an on-site performance demonstration. Essentially this involved going to the relevant work area (this study was for parts manufacturers) and asking a series of questions to determine promotability, such as "How do you set up this machine?" It's fascinating stuff, and it worked (using concurrent measures), although it might be challenging to use for less observable performance measures. For more details, check out the in press version here; the recipe book starts on page 12, and check out the example on page 41.

What else is in the issue? Take a look:

Predicting managerial readiness in Chinese workers

Is inflation in personality inventories necessarily a bad thing?

Do occupations tend to have their own personality? (yep)

Leadership effectiveness: Self- versus other-ratings (check out who tends to inflate)

CWBs: The organization plays a role

Biodata continues to shine (this time in healthcare organizations)

Is handwriting analysis popular among European employers? Not so much.

Can you predict military performance using personality inventories? Seems so.

Job experience v. personality measures in a small sample

Biodata in the news

I don't write about biodata inventories very often. Not because they don't work--quite the opposite--but because like personality inventories, they require special knowledge and resources to develop. They also don't tend to be in the news nearly as much as other types of tests like interviews and personality inventories. So I was pleased to see a recent article in the mainstream press about how the FBI uses biodata inventories as part of its hiring process for Special Agents.

For those of you unfamiliar with this type of test, they look at first glance like a job knowledge-type multiple choice exam. The example given in the article is as follows:

To what extent have you enjoyed being given a surprise party?
A. Not at all
B. To a slight extent
C. To a moderate extent
D. To a great extent
E. I have never been given a surprise party

Like a personality inventory, a common reaction to this type of question is sometimes "What does this have to do with hiring someone?" But herein lies the beauty--and complexity--of biodata inventories: each item on the exam has been statistically linked to job performance. So unlike most other types of tests, items are shown to predict performance before being given to applicants.

Other benefits? They are resistant to faking so are great candidates for Internet administration and they typically result in smaller group differences than cognitive tests.

So why don't more organizations use them? A few reasons:

1) They require large sample sizes to develop
2) They require a fair amount of statistical expertise
3) They look a little funky ("why are you asking about my surprise party??")

Those are substantial hurdles for many organizations. But given the rush to online assessment and screening, we may see their usage increase.

When Hiring a Surgeon, Should We Ask If They Play Video Games?

You may have seen this news article today, about a study published in the most recent issue of the Archives of Surgery. The study found a strong relationship between video game experience/skill and performance on a surgical skills test among a sample of 33 residents and attending physicians at Beth Israel Medical Center in New York.

How did the authors come to this conclusion? They had the participants play some games, fill out a survey about their game playing experience, and take a simulation test (interestingly called "Top Gun").

Correlation, causation, whatever

The authors note that "video game skill correlate[d] with laparoscopic surgical skills." But here's where everything goes to heck and those of us who have taught statistics hang our head in shame. One of the fundamental lessons in statistics is that correlation is not causation--just because two things relate to one another doesn't mean one CAUSES the other. My favorite example of this is the positive correlation between ice cream cone purchases and crime rate. Does this mean ice cream cone purchasing causes crime? Or, um, maybe there's a third variable (like heat)?

The news outlets, feeding off the study authors, picked this up and ran with it, suggesting that playing video games somehow causes you to be a better surgeon--or even that kids that play video games could be somehow preparing themselves for a high paying career.

But here's the thing: this study did not have unexperienced video gamers play a game then perform surgery. It's purely point-in-time. All this study showed is that folks that are good at video games are good at the surgical task. Makes sense--they involve some of the same skills. But that's all we can really say. We can't suggest that people play more video games if they want to be better surgeons (and to their credit the news articles generally point this out).

Biodata pops up again

On the other hand, this type of information could be useful for another purpose--biodata. Google has been in the news recently (discussed here and elsewhere) for re-vamping its hiring practices to include biodata. So as part of a hiring practice, questions could be asked about hobbies, one of which could be video games. But you certainly wouldn't want this to be your entire selection process.

On the other hand, with so many kids playing video games these days, is there really going to be a lot of variance in experience 10 years from now for us to use in separating out applicants? Maybe by then we can ask applicants if they've completed Expert level on Surgeon Wars.

More on Google hiring

Another day, another article about Google hiring practices.

I've posted about this before--about how Google recently changed its processes to focus on fewer interviews, triangulate on candidate KSAs, and expand the scope of predictors beyond GPA and college diplomas.

I wish I was as excited about this new article, but truth be told I have some concerns about the messages they're sending.

- Laszlo Bock, their VP of "People Operations" had this to say about interviews:

"Interviews are a terrible predictor of performance."

Hmmmm....I seem to recall reading some evidence that refutes this (granted, it's looking like we need to take another look at the validity coefficients). If by "terrible" Bock is implying that biodata is far superior to structured interviews, I'd like to see what data he's looking at. Perhaps he meant the data Google collected showed poor correlations between their interview scores and subsequent job performance?

- This led to this rather unfortunate quote from the article author:

"...academic research suggests that...grades and interviews...are not an especially reliable way of hiring good people."

Again, research suggests otherwise. Yes, support for using grades to predict performance is limited at best. And yes, unstructured interviews are not only unreliable but unwise from a legal perspective. But structured interviews with well-designed questions and rating scales can be very effective.

That said, I still believe Google is setting the proper example for modern assessment in several ways:

- By focusing on what the data shows. Google collected a substantial amount of data that allowed it to look carefully at which background characteristics predicted job performance--both task-based and citizenship behaviors.

- By tailoring assessment to the needs of the position. Separate "surveys" were developed for different job groups (e.g., engineering, sales, finance, HR).

- By not rushing to embrace point-based T&Es in automated systems but instead taking the time to go with something far better (biodata).