Saturday, October 16, 2010

Q&A with Piers Steel: Part 2

Last time I posted the first part of my Q&A with Piers Steel, co-author of a recent piece in Industrial and Organizational Psychology (that I wrote about here) on synthetic validity and a fascinating proposition to create a system that would greatly benefit both employers and candidates. Read on for the conclusion.

Q4) Describe the system/product--what does it look like? For applicants? Employers? Governments?

A4) How do we do it? Well, that’s what our focal article in Perspectives on Science and Practice was about. Essentially, we break overall performance into the smallest piece people can reliably discern, like people, data, things (note: our ability to do this got some push back from one reviewer – that is, he was arguing we can’t tell the difference if people are good at math but not good at sales and vice-versa – it is a viewpoint that became popular because researchers assumed that “if it ain’t trait, it’s error”). We get a good job analysis tool that assesses every relevant aspect of the job, such as job complexity. We get a good performance battery, naturally including GMA and personality. We then have lots of people in about 300 different jobs take the performance battery, have their performance on every dimension as well as overall assessed to a gold standard (i.e., train those managers!), and have their jobs analyzed with equal care with that job analysis tool. From that, we can create validity coefficients for any new job simply by running the math. It is basically like validity generalization plus a moderator search, where once we know the work, we can figure out the worker. Again, read the article for more details, but this was basically it.

Once built all employers need to do to get a top-notch quality selection is describe their job using the job analysis tool and then as fast as electrons fly through a CPU, you get your selection system, essentially instantly. It is several orders of magnitude better in almost every way from what we have now from almost every criterion.

Q5) What are the benefits--to candidates, employers, and society?

A5) Everyone had a friend who struggled through life before finding out what they should have been doing in the first place. Or changed a job for a new company only to find they hated it there. Or never found anything they truly excelled at and just tried to live their lives through recreational activities. Everyone has experienced lousy service or botched jobs because the employee wasn’t in a profession that they were capable of excelling in. Everyone has heard of talented people who were down and out because no one recognized how good they really were.

Synthetic validity is all about making this happen less. How much less is the real question. If we match people to jobs and jobs to people wonderfully now, then perhaps not at all. But of course, we know that presently it's pretty terrible.

Now synthetic validity won’t be able to predict people’s work future perfectly, but it will do a damn sight better job than what we have now. Also, the best thing about synthetic validity is that it is going to start off good and then get better every year. Because it is a consolidated system, incremental improvements, “technical economies,” are cost effective to pursue and once discovered and developed, they are locked in every time synthetic validity is used.

Right now, we have a system that can only detect the largest and most obvious of predictors (e.g., GMA) because of sample size issues, but can’t pursue other incremental predictors because they aren’t cost effective for just one job site. By the very nature of selection today, we are never going to get much better. As I mentioned, nothing major has changed in 50 years and nothing major will change in the next 50 if we continue with the same methodology. Synthetic validity is a way forward. With synthetic validity, the costs are dispersed across all users, potentially tens of millions, making every inch of progress matter.

So, what we will get? Higher productivity. If synthetic validity results in just a few thousand dollars of extra productivity per employee each year, multiply that by 130 million, the US work force. Take a second to work out the number – it’s a big number.

Also, people should be happier in their jobs too, creating greater life satisfaction. They will stay in their jobs longer, creating real value and expertise. Similarly, unemployment will go down as people more rapidly find new work appropriate to their skills. In fact, I can only think of one group that won’t like this – the truly bad performer. They are the only group that wouldn’t want better selection.

Q6) Finally, what do you need to move forward?

A6) So far, no one I know is doing this. There are some organizations who think they are doing synthetic validity, though it is really just transportability and they aren’t interested in pursuing the real thing. Partly, I think it is because the real decision makers don’t know about synthetic validity or don’t understand it. I could do more to communicate synthetic validity, though I have done quite a bit already. I have sent a few press releases, received a dozen or two newspaper interviews (Google it), contacted a few government officials on both sides our border, and pursued a dozen or so private organizations. Part of my reason to do this interview here is to try to get the word out. So far, all I got back was a few “interesting” but no actual action.

I used to think this lack of pursuit was because synthetic validity was so hard to build, requiring 30,000 people -- but we know a lot more now. In the Perspectives article, McCloy pointed out we could allow ourselves to use subject matter experts to estimate some of the relationships. That won’t be as good as if we gathered the data ourselves, but we could get something running real quick, though later we would upgrade with empirical figures. Consequently, the reason why this isn’t built isn’t because it is too difficult. Also, the payoff would eventually be cornering the worldwide selection and vocational counseling market. I am not sure what that is worth but I imagine you could buy Facebook with change left over for MySpace if you wanted to. The value of it then isn’t the problem either.

I am coming to the conclusion that despite the evidence, to most people it is just my word as an individual. I’m a good scientist, winner of the Killam award for best professor at my entire University, but it still isn’t enough. You need the backing of a professional association and so far ours [SIOP] hasn’t yet taken a stand. As a professional organization, we should be promoting this, using the full resources of our association. I admit that I am “a true believer,” but this seems to be one of the bigger breakthroughs in all the social sciences in the last 100 years. Alternatively to the backing of a professional association, we need a groundswell where hundreds of voices repeat the message. I will do my bit but hopefully I will have a lot of company.

If you think I am overstating the case regarding synthetic validity, show me where I’m wrong. We handled all the technical critiques and issues in the Perspectives article. Right now, you have to make the argument that “human capital” doesn’t matter, that being good or bad at your work doesn’t matter. And if you try to make that case, I don’t think you are the type of person who would be even worth arguing with.


I'd like to thank Dr. Steel for his time and energy. I truly hope this idea sees the light of day. If you are interested in moving this forward, leave a comment and I can put you in touch with him.

8 comments:

Catherine Claridge said...

I've been following this study for a few years. As a Recruiter myself it's very exciting and I'd love to keep abreast of the latest updates on Mr. Steel's efforts. Nothing seems to have been posted online since this article. I wonder if there have been any new developments in his goal to make this idea a reality?

BryanB said...

Catherine - great question. I followed up with Piers and he's been up to quite a bit. In addition to presenting at this year's SIOP conference on synthetic validity, he's still pursuing the concept he described here but it's slow going. He understands that at least one consulting outfit is pursuing something similar but it's on a similarly slow track.

So bottom line: he's still very much interested and pursuing the idea, but it be a while before we see the fruits of his labor.

Piers said...

Still at it by the way. Our industry partner got taken over and the project was dropped, being in the initial stages. We decided then that we had enough expertise and good bring the costs down low enough to do this as a research grant. We got a perfect rating on our SSHRC grant from the external reviewers but rejected by the committee (sigh!). This year we are apply to multiple places. It really shouldn't be this hard, but I find I can't turn my back on it. Amazing, really, for something that would benefit the world in terms of noticeable increments to GNP for almost nothing.

BryanB said...

Dr. Steel - thanks so much for the update! Glad you're keeping at it, and much luck to you! Definitely a worthy proposal.

Piers said...

Still no luck by the way after four years. Our latest effort was to interest SHRM's research grant, but their reply was"

"Reviewers questioned the practical implications of the results. Although reviewers understood the overarching goal of synthetic validity, they were not clear how this would translate to helping an individual HR professional."

This, by the way, was our opening paragraph:

"Synthetic validity enables the efficient, quick, and accurate matching of people to their ideal jobs and jobs to their ideal employees. In their review article on synthetic validity, Steel, Huffcutt, and Kammeyer-Mueller (2006) summarized it as a process that has “the capability of creating and justifying a selection procedure on the basis of a job analysis alone. That is, once you have described the work, you know on what basis workers should be hired” (p. 16). The benefit over traditional selection systems is significant: Synthetic validity promises to create systems faster, at lower cost, and at higher quality. Moreover, these systems are likely to be more legally defensible and more easily administered. What once took months and tens of thousands of dollars with concurrent and predictive validation procedures commonly used today, would take minutes and cost just a fraction with synthetic validity."

A few smaller companies appear to be inching their way towards this. I think it might be my role to have laid the groundwork. You wouldn't think it would be this difficult but it is.

BryanB said...

Unfortunate but sadly not surprising. I think about this idea quite a bit. I wonder if you would have better luck hooking up with SIOP or the feds. Or a private testing company.

Piers said...

Finally we got the SSHRC grant! From the worst to the highest rated SSHRC grant in one year (with absolute no change in the grant). Working with Colin Lee, a post doc, to make it happen. We are making really good progress. Have a working methodology and finishing a simulation at a national level. About one year away now and nothing can really stop us.

BryanB said...

Excellent!! Please keep us updated on how things progress! Very exciting.