When Hiring a Surgeon, Should We Ask If They Play Video Games?

You may have seen this news article today, about a study published in the most recent issue of the Archives of Surgery. The study found a strong relationship between video game experience/skill and performance on a surgical skills test among a sample of 33 residents and attending physicians at Beth Israel Medical Center in New York.

How did the authors come to this conclusion? They had the participants play some games, fill out a survey about their game playing experience, and take a simulation test (interestingly called "Top Gun").

Correlation, causation, whatever

The authors note that "video game skill correlate[d] with laparoscopic surgical skills." But here's where everything goes to heck and those of us who have taught statistics hang our head in shame. One of the fundamental lessons in statistics is that correlation is not causation--just because two things relate to one another doesn't mean one CAUSES the other. My favorite example of this is the positive correlation between ice cream cone purchases and crime rate. Does this mean ice cream cone purchasing causes crime? Or, um, maybe there's a third variable (like heat)?

The news outlets, feeding off the study authors, picked this up and ran with it, suggesting that playing video games somehow causes you to be a better surgeon--or even that kids that play video games could be somehow preparing themselves for a high paying career.

But here's the thing: this study did not have unexperienced video gamers play a game then perform surgery. It's purely point-in-time. All this study showed is that folks that are good at video games are good at the surgical task. Makes sense--they involve some of the same skills. But that's all we can really say. We can't suggest that people play more video games if they want to be better surgeons (and to their credit the news articles generally point this out).

Biodata pops up again

On the other hand, this type of information could be useful for another purpose--biodata. Google has been in the news recently (discussed here and elsewhere) for re-vamping its hiring practices to include biodata. So as part of a hiring practice, questions could be asked about hobbies, one of which could be video games. But you certainly wouldn't want this to be your entire selection process.

On the other hand, with so many kids playing video games these days, is there really going to be a lot of variance in experience 10 years from now for us to use in separating out applicants? Maybe by then we can ask applicants if they've completed Expert level on Surgeon Wars.

More on Google hiring

Another day, another article about Google hiring practices.

I've posted about this before--about how Google recently changed its processes to focus on fewer interviews, triangulate on candidate KSAs, and expand the scope of predictors beyond GPA and college diplomas.

I wish I was as excited about this new article, but truth be told I have some concerns about the messages they're sending.

- Laszlo Bock, their VP of "People Operations" had this to say about interviews:

"Interviews are a terrible predictor of performance."

Hmmmm....I seem to recall reading some evidence that refutes this (granted, it's looking like we need to take another look at the validity coefficients). If by "terrible" Bock is implying that biodata is far superior to structured interviews, I'd like to see what data he's looking at. Perhaps he meant the data Google collected showed poor correlations between their interview scores and subsequent job performance?

- This led to this rather unfortunate quote from the article author:

"...academic research suggests that...grades and interviews...are not an especially reliable way of hiring good people."

Again, research suggests otherwise. Yes, support for using grades to predict performance is limited at best. And yes, unstructured interviews are not only unreliable but unwise from a legal perspective. But structured interviews with well-designed questions and rating scales can be very effective.

That said, I still believe Google is setting the proper example for modern assessment in several ways:

- By focusing on what the data shows. Google collected a substantial amount of data that allowed it to look carefully at which background characteristics predicted job performance--both task-based and citizenship behaviors.

- By tailoring assessment to the needs of the position. Separate "surveys" were developed for different job groups (e.g., engineering, sales, finance, HR).

- By not rushing to embrace point-based T&Es in automated systems but instead taking the time to go with something far better (biodata).