Evidence-based I/O: Where are we? More importantly, who?

The March 2011 issue of Industrial and Organizational Psychology contains two excellent focal articles. One, which I plan on writing about in the future, is about how the field needs to spend more time studying the experience of working rather than treating workers as objects. It's one of the best, most thought-provoking articles I've read in this journal.

But today I'll focus on the first article, which is about the evidence-based practice (or lack thereof) of I/O psychology, by Briner and Rousseau (B&R). This topic is obviously near and dear to me given this blog so I had a lot of reactions during my read of it. I'll share some of those with you today.

B&R argue that the practice of I/O psychology is not strongly evidence based--at least not in the sense that other professions (e.g., medicine) are becoming. This may surprise you given the history behind I/O psychology and its strong grounding in sound research methods. But we're not talking about the quality of research (although as some of the commentaries point out, this is a relevant question), rather how well we're doing putting research findings into practice.

The authors point out that there are many "snake-oil" peddlers out there who claim to have evidence for what they do. That the concept of evidence based is not used or well known in I/O circles. Even more fundamentally, we have no data on what percentage of I/O practices are based on solid scientific evidence. They point out that part of the problem is that the latest research findings and research summaries aren't accessible--a point I obviously agree with given the purpose of this blog. They even provide a tool (systematic reviews) that they suggest could help us get there.

Yet it's one particular point, referenced by both focal and commentary authors but not treated in depth, that I'd like to focus (okay, rant) on. I'll use a quote from B&R to get us started: "It is not always obvious to practitioners, certainly not the least experienced or less reflective, how exactly to apply the principles identified in such research." This comes as close as any to, IMHO, the real issue: who uses I/O research in organizations.

Before I go too much further, I'd like to do something that I don't believe any of the authors did: acknowledge my bias in approaching this topic. I'm someone who has an advanced degree in I/O (a Masters, which paraphrasing the focal authors puts me slightly above the village idiot), but who works in the trenches assisting supervisors and managers alongside many who have little if any formal education in either I/O or HR. This obviously impacts the issues I see as important and my take on them. 'Nuff said.

Now, back to my point. I believe the authors in this issue fail to recognize a very important point: that although I/O research is used by I/O consultants and academics, and to a lesser extent by mid-level managers, there's a large group who is going unnoticed: HR practitioners. It is these individuals who I would argue in most organizations are the primary "users"--whether they know it or not. These are the folks who supervisors count on to provide expertise related to a wide variety of HR/IO issues, including recruitment and selection. Yes, many organizations use consultants who have formal training in I/O, but I think we can all agree that in terms of the number of day-to-day decisions that get made related to HR, we're primarily talking about supervisors and HR practitioners. And only after recognizing this point do answers to the question "how do we make I/O practice more evidence-based?" become more clear.

Let me throw out a couple questions just to stimulate you a bit more:

1) Why is this one of only a handful of "publications" devoted to making research more accessible to HR practitioners? (and heck, I don't even know how many of you are practitioners in the first place!) And why is it left up to "independents" such as myself, rather than professors? (Dennis, you are a blessed exception)

2) Why does SHRM, with its enormous resources and user base, focus on things like HR strategy and leadership, while failing to focus on evidence based practice? (and while I'm ranting, Sir Richard Branson as keynoter? really?)

3) Why is SIOP seemingly only now beginning to make attempts to make research more accessible (e.g., with its randomly updated blog and thankfully soon-to-be published Science You Can Use series)?

I would argue it's all because we do a horrible job of focusing on the true users of I/O research. We engage in high-level debates about criterion-related validity but fail to gather even basic information like who is responsible for making each type of HR decision.

And unlike evidence-based medicine, we have a particular responsibility to ensure that HR/IO decisions are made using the best science. Because the consumers of the research aren't all individuals with advanced training (e.g., doctors). They're supervisors who are under the gun to make an effective, and ass-covering, decision. They're HR analysts who ended up there not because they love the topic but because they were looking for a promotion.

So if the goal is to increase the number of people-related decisions in the workplace that are based on the best evidence, in addition to tackling the issues identified in this issue (such as publication bias and accessibility), we need to do a much better job of understanding our customer base and tailoring our efforts toward them. After all, I/O psychologists generally do not control organizational practices.

So here, in no particular order, are my own recommendations for helping ensure the "best" (defined here are based on science, not things like, oh, I dunno, organizational politics) people decisions get made:

1. Make basic research more accessible--and by this I mean affordable. And by this I mean free. Or cheap. Fifteen dollars for a research article? Really? How many of those are you selling, exactly, publishers? Somebody, I don't know who know, needs to get on this.

2. Start addressing the elephant in the room: that many of those providing guidance to supervisors (i.e., internal HR), not to mention supervisors themselves, are doing so based on absolutely zero research. Who are these people? How are they trained? What do they know? We know very little, because we don't study them (with some exceptions). Imagine if evidence-based medicine tried to progress without understanding anything about doctors.

3. In a similar vein, spend more time understanding those who are ultimately responsible for people decisions and have to live with them: supervisors. Particularly first-line ones. Why--and more importantly how--do they make decisions related to particular HR issues? To what extent is speed the single most important decision criterion?

4. Identify the "high value" people decisions (e.g., selection, harassment prevention) and their current practice. Use this to figure out where the biggest gap is and where we should be channeling resources. Without a baseline it's hard to figure out where we should be focusing.

5. For Pete's sake, let's start putting our "blessing" on certain practices. SIOP, don't be afraid to endorse things. By remaining "objective", you're de facto taking the stance of, "hey, figure it out yourself, or hire a high-priced consultant." There's a thought behind products earning an "energy star."

6. Let's have an industry-approved training curriculum and certification. Why does SHRM offer the PHR and SPHR...and that's pretty much it? We need training programs for practitioners that are created and reviewed by people steeped in the research but able to translate.

7. Let's identify "best of" practices as well as "worst of." Why are most awards for individual researchers? Why not organizations that demonstrate particularly effective practices? Why not have a professional equivalent of the Razzies for just the opposite? ("and now...the award for worst interview question of 2011 goes to...")

Okay, I think that's the end of my rant. Let me be clear, the proposition that people decisions should be based on the best available science is hard to argue with. I applaud all of the authors for thinking deeply about this topic. I just think we need to get in the trenches a little more. Because if we don't, the trend that's already apparent (toward automation/speed and away from thoughtful, research-based decisions) will accelerate. Leaving lots of people trained in I/O but few who understand--and are interested in--their services.

Side note: this article did point out a couple books I'm considering adding to my library, including Locke's handbook on organizational behavior (probably still too much for the average consumer in this age of Twitter, but on the right road), and Locke, et al.'s book on evaluating HR programs.