Research update + happy anniversary...to me!

Two things this time: we've got a lot of research to go over, and then a bit of a celebration! First the research.

The September issue of the Journal of Applied Psychology is out. Let's see what it has to offer:

- Using performance ratings as an assessment or a criteria? You'll want to look at Ng, et al.'s study of leniency and halo errors among superiors, peers, and subordinates of a sample of military officers.

- Speaking of criteria, you may be interested in Northcraft et al.'s study of how characteristics of the feedback environment influence resource use among competing tasks. Interesting stuff.

- Okay, let's turn to something more traditional. Berry, et al. look at correlations between cognitive ability tests and performance among different ethnic groups. Not surprising to those of you familiar with the research, the largest difference found was between White and Black samples.

- Another traditional (but always interesting) topic: designing Pareto-optimal selection systems when applicants belong to a mixture of populations. Check out De Corte, et al.'s piece. Oh, you might be interested in the in-press version.

- Dr. Lievens (a co-author on the previous study) has been busy. He and Fiona Patterson collaborate on a study of the incremental validity of simulations, both low fidelity (SJTs in this case) and high fidelity (assessment centers), beyond knowledge tests. Yes, both had incremental validity, and interestingly ACs showed incremental validity beyond SJTs. Check out the in press version as well.

- Wondering whether re-testing degrades criterion-related validity or impacts group differences? You're in luck because Van Iddekinge, et al. present the results of a study of just that. Short version? Re-testing actually did a lot of good.

- I know what you're thinking: "Might Lancaster's mid-P correction to Fischer's exact test improve adverse impact analysis?" Check out Biddle & Morris' study for an answer.

- And now that you've had your fill of that statistical analysis, you find your mind wandering to effect size indices for analyzing measurement equivalence. I'm right there with ya. So are Nye & Drasgow.


Let's turn now to the October issue of the Journal of Personality and Social Psychology because there are a few articles I think might interest you...

- First, Lara Kammrath with a fascinating study of how people's understanding of trait behaviors influence their anticipation of how others will react.

- Speaking of fascinating, George, et al. present the results of a 50-year longitudinal study of personality traits predicting career behavior and success among women. It makes you realize again how much has changed since the 1960s!

- I tell ya, this issue is chalk full of goodness. Carlson et al. demonstrate that people can make valid distinctions between how they see themselves and how others see them--potentially informing the debate on personality inventories.

- Lastly, a piece by Specht et al. on how personality changes over the life span and why this might be. Fascinating implications for using personality inventories for selection.

Bonus article: remember how I mentioned using performance ratings above? Well you might be interested in an article by Lynn & Sturman in the most recent Journal of Applied Social Psychology where they found that restaurant customers sometimes rated the performance of same-race servers as better than those of different races--but it depended on the criterion.



FINALLY, I'm proud to announce that this blog has officially been going strong for five years. My first post (now incredibly hard to read) was in September of 2006. Back then the only other similar blog was Jamie Madigan's great (but now sadly defunct) blog, Selection Matters. My first email subscriber (from DDI if you're curious) came on a month later. Now I have almost 150 email subscribers and at least a couple hundred more who follow the feed. Around 3,000 individuals visit the site each month from over a hundred countries/territories (U.S., India, and Canada are 1-2-3). It's a labor of love and I thank you for reading!

One last time: It's all subjective

While reading news about a court decision recently I was struck again by how the U.S. Court system continues to make a false and largely unhelpful distinction between "objective" and "subjective" assessment processes, and they're certainly not the only ones. Presumably this is meant to highlight how some are based on judgment while others are free from it.

One last time: it's all subjective.

I challenge you to come up with a single aspect of the personnel assessment process that is not based on judgment. Not only that, "degree of judgment" is not a useful metric in determining the validity of a selection process--for many reasons including but not limited to whose judgment is being used.

Here is a sample of assessment components that are based on judgment:

- how to conceptualize the job(s) to be filled

- how to study the job(s)

- how to recruit for the job(s)

- which subject matter experts to use

- how to select the KSAs to measure

- how to measure those KSAs

- how items are created or products are selected

- how to set a pass point, if used

- how to administer the assessments

- how to score the assessments

- how to combine assessments

- how to make a final selection decision

- what type of feedback to give the candidates

And this is just the tip of the iceberg. The entire process is made up, like all complex decisions, of smaller decisions that can impact the selection process (what software do I use to analyze the data? Is this one KSA or two?).

So what WOULD be helpful in describing how an assessment process is developed and administered? I can think of a few yardsticks:

1. Extent to which processes and decisions are based on evidence. To what extent is the assessment and selection process based on methods that have been shown scientifically to have significant utility?

2. Degree of structure. To what extent are the assessment methods pre-determined? How flexible are the processes during the assessment?

3. Multi-dimensionality. How many KSAs are being measured? Is it sufficient to predict performance?

4. Measurement method. How many assessment methods are being used? Do they make sense given the targeted KSAs?

5. Transparency. Is the entire process understandable and documented?

I'm not inventing the wheel here. I'm not even reinventing it. I'm pointing out how most cars have four of them. It's obvious stuff. But it's amazing to me that some continue to perpetuate artificial distinctions that fail to point out the truly important differences between sound selection and junk.

August Research update

Okay, past time for a research update.

- The September issue of IJSA has several articles on job discrimination, including pieces by Anderson, Anseel, and Patterson & Zibarras. Looks like the focus is not only on actual job discrimination but perceived job discrimination.

- Krause, et al. describe assessment center practices in South Africa

- Ree and Carretta get serious about incremental validity versus unique prediction

- What do you need to be a top notch recruiter? Well if your measure of job performance is revenue, don't dismiss emotional intelligence, according to Downey, et al.

- Chen, et al. help us understand how applicant personality interacts with impression management tactics to influence interviewer perceptions

- What type of student succeeds at a campus recruitment drive? Gokuladas studied nearly 600 engineering graduates in South India and found that success correlated with engineering GPA and English proficiency. Interestingly female graduates outperformed their male counterparts for jobs in the software services industry.

- Trying to predict counterproductive work behaviors? Bowling et al. suggest that the interaction between conscientiousness, agreeableness, and neuroticism is important and complex.

- One of the focal articles in the September 2011 Industrial and Organizational Psychology is all about individual psychological assessment--something I don't write about very often. Silzer and Jeanneret focus on executive assessment but cover a lot of ground on the broader topic, including why IPA has its critics and how we can advance the science. It is, as with all the focal articles, followed by several commentaries.

- Son Hing, et al. present a fascinating study of meritocracy and how it differs from other hierarchy-legitimizing ideologies

- Stats fans: Bonett and Wright present a method for obtaining the proper sample size for obtaining a certain confidence interval when conducting multiple regression.

- van Vianen, et al. describe a study that contributes to our understanding of the important of person-organization and person-supervisor fit

Sorry there haven't been more updates lately. New baby + potential layoffs at work = little time for blogging!

How can assessment help in a downturn?

Exam workload is down at my job. Not "pack up your bags and go home" down, but we're running significantly fewer exams than we were 5-6 years ago. And I doubt we're the only organization in this situation.

Some employers are hiring. But many organizations are still running very lean, and the public sector in particular continues to face budget cuts and layoffs.

So if you have staff with skills in assessment that aren't being used to their full capacity (a dangerous recipe for several reasons, budgetary and morale-wise), what's an organization to do?

Well, personnel assessment is all about measuring people. So anything you do in your organization that involves measuring people hypothetically has a built-in staff with the competencies you need to get the job done.

Let's look at some examples of how assessment can help organizations during a downturn:

1) Help determine who to let go (if possible). If the organization is able to use competency levels to determine who to keep, this is obviously the preferred route over, say, letting go those with the most (or least) seniority. The tests should measure KSAs relevant for the critical work the organization needs done now--and in the future.

2) Help internal employees objectively assess their skill levels--both those being let go as well those staying who wish to enhance their career mobility and stability. A well-designed and scored assessment should be able to give employees some insight into their strengths and areas for improvement.

3) Help organizations get a better sense of their talent. Sometimes called a "talent inventory", assessment can be used strategically to help organizational leadership conduct workforce planning and identify areas of skill discrepancies. This includes succession planning. What percentage of key leadership positions in your organization could be backfilled tomorrow--successfully? What skills do up-and-comers need to develop to help them get ready for the next step? Assessment can help answer that.

4) For those that are hiring, many hiring supervisors are still receiving large volumes of applications. This is where assessment shines--help them identify who are truly the most qualified for the position.

5) Use this opportunity to make sure your competency models and/or job analyses are complete and updated. Jobs change--are you still using a description from 10 years ago? Are you still hiring based on outdated duty statements?

Now let's think a bit more broadly, beyond traditional personnel assessment into some other areas where the skills your team has developed are just as relevant:

6) Engagement surveys. Getting a better sense of the attitudes and emotions of your workforce helps you do a variety of things, including avoiding turnover of high performers, targeting organizational sub-units that need improving, and identifying under-performing supervisors. It also--when done properly--gives your employees a sense of voice, which can be key in times of anxiety and uncertainty. Just make sure you do something with the results.

7) Organizational change initiatives. Assessments can be used for a variety of purposes during change initiatives, such as measuring the "pre" and "post" states, identifying key sources of resistance, keeping track of success measures, and accurately measuring outcomes.

8) New product implementation and user satisfaction surveys. Do you have key pieces of technology you've implemented recently? How is that working out for the users? Do the users have additional ideas for products or services that would help them get their jobs done and be innovative? I'm thinking of technology here, but you can see how we could go beyond that.

9) Team building. Most people in organizations depends on others to get their jobs done thoroughly. This can be invigorating or frustrating--do you have a good measure of team satisfaction? Are there assessments that can help team members interact more effectively? You betcha.

10) Program evaluation. Whether formative, summative, theory-based, or some other type, assessment can help identify needs, clarify paths, and determine whether money and time invested is giving the organization the outcomes it hand in mind. It can also help them uncover unanticipated consequences.

11) Entrance and exit surveys. When resources are scarce, it's even more important to maximize the return on the investment we make in hiring and make sure those entering the organization have the resources they need. On the other end, capturing good data from those leaving the organization can help identify key areas of weakness or provide insight for hiring the next time.

I'm sure I missed a few. But you get the idea. Any time you have a slow down in a part of the organization, use it as an opportunity to expand the scope of your HR strategy. Test your ability to be flexible and innovative. Whatever you do, don't waste your resources.

July Research Update

Okay, it's past time for a research update. Let's see what's going on out there:

- Looking to get a better salary offer? Research by Thorsteinson suggests that you shouldn't be afraid to aim high...

- Trying to hire for creativity? You'll be interested in Madjar et al.'s research that indicated that depending on the type of creativity you're after (i.e., radical or incremental) both personal and environmental factors play a role. For example, if you're after radical creativity, look for a willingness to take risks and career commitment.

- You stats guys and gals out there will want to check out Johnson et al.'s study that investigated common method variance (CMV). Specifically, they found that by applying different types of remedies for CMV, they altered the relationship between two variables (core self-evaluation and job satisfaction). Something to consider when investigating criterion-related validity.

- Speaking of core self-evaluation, Chen disputes the inclusion of several factors in the concept, and argues that problems exist with its convergent and discriminant validity.

- Observer ratings of personality are hot. And Oh, Wang, and Mount get into the action with their study that found (as other have) that observer ratings yield higher predictive validities than self-reports. Now if we only had an easily-accessible database of observer ratings...

- Do hiring managers really discriminate against obese individuals? Yep, as Agerstrom and Rooth show. Specifically, scores on the implicit association test predicted whether hiring managers were likely to invite obese applicants for an interview.

- Looking to increase the number of staff that exhibit organizational citizenship behaviors? (ya know, things like helping out a co-worker when they don't have to) You might look first to the supervisor, as Yaffe and Kark demonstrate.

- A recent study that found that more women on a team increased group performance garnered some press, but the real story was the predictive role played by social competence (which women happened to score higher on). The authors pondered whether social competence could be trained, thus increasing group performance. Well, a new study by Kotsou et al. seems to suggest just that, showing that a group-format intervention increased emotional competence in adults (it also resulted in lower cortisol secretion and better subjective and physical well-being).

- Speaking of trying to maximize team performance, a study by Humphrey et al. found that both short- and long-term team performance was highest when variance in conscientiousness scores was lowest but variance in extroversion was highest. What does this mean? It suggests that people perform best when around others with similar levels of conscientiousness but perform better when working with people that vary in their levels of extroversion. Fascinating, and something to consider when hiring for or building teams.

- And speaking (again) of emotional competence, Seal et al. describe the development of a new measure of social and emotional development.

- Finally there is Hee et al. with an important study of prejudice. Specifically, the authors found that prejudice against out-group members increases with in-group size and perceptions of homogeneity among both the in-group and out-group. In addition to validating the importance of intergroup contact, this research suggests prejudice may be reduced when dealing with small groups and increasing an understanding of the differences within each group. Makes a lot of sense.


Last but not least, for you IPAC members don't forget that presentations from the just-concluded conference (which I heard was a rousing success) are starting to be posted in the members-only portion of the website. Good stuff!

Is Google+ what we've been waiting for?

No doubt by now you've heard about Google+ ("Google plus"). It's essentially Google's latest stab at trying to topple Facebook as the global social networking leader. While Facebook holds a lot of promise in the way of recruiting, selection, and other core HR functions, its use has been sporadic due to a number of issues. Can Google+ give us the functionality we've always wanted--and stick around long enough to be a major player?

If you haven't heard of Google+ (or haven't read enough), you can get more information here or here or here or here. Or heck, just watch their intro video. By some estimates it already has around 10 million users, which is pretty amazing considering it only came out a few weeks ago, although it's not anywhere near Facebook which claims it has 750 million.

But let's not get ahead of ourselves. Let's first review why Facebook isn't the holy grail we once thought it could be.

At first blush, a public social networking site holds a heck of a lot of promise. It allows organizations to learn more about potential candidates, things beyond a test score. It allows people to network, potentially increasing the speed and efficiency of information sharing. And it allows applicants to learn more about organizations--faster and more informally than a career webpage.

This all sounds good in theory. And some organizations have made Facebook work for them, by using things like fan pages. But for many, the promise has never been fulfilled. Let's look at the main reasons why, and how Google+ may be the answer.

The main problem is that on Facebook, there's just one you. When you post something on Facebook, it goes to all your connections. Friends, family, co-workers, acquaintances--everyone, unless they've "hidden" you. (Yes, you can post to a smaller group but it's a pain and who actually does this?) This has several implications:

- You watch what you say. Do really want to say the same thing to your friends that would to your family? Do you really want to post that picture for everyone to see?

- It increases "lurkers", who simply read but never post, thus not being true participants in the online social interaction.

- This hyper-openness serves as a barrier to some users, who are simply uncomfortable sharing their life with people.

- You have friend request anxiety. You dread getting an email that your boss or someone you don't really know wants to be "friends" on Facebook. If you say no you feel like a jerk, if you say yes then you're openness goes down a notch. What fun is that?

Google+ deals with all this by allowing you to create something called "circles" where you add individuals to certain groups (e.g., friends, colleagues, family) which thereby allows you to post things to only those groups. It even allows you to view your own profile as if you were someone else (IMHO one of the most innovative features).

What are the implications of these circles? There are many:

1. People are more likely to create profiles. Along with the increased attention to security attracting new people, once it gets out that you can manage your social identity much better (and easier) with Google+, it's more likely that someone will create a profile to begin with.

2. Individuals will be more likely to accept invitations to connect from organizations and recruiters. Why? Because they'll be able to manage what information gets shared with those individuals.

3. The information employers have access to will be less--but of higher quality. Assuming people think before posting (yes I'm giving people the benefit of the doubt here), while there may be less information on a profile for an employer to view, it will be more relevant. Instead of posts about children or parties, it will be opinions or accomplishments--things that might actually be job related and much less likely to get employers into legal hot water.

4. It could help with referrals, as individuals will feel more comfortable sharing information about jobs--or their own interest in jobs--without fear of what their management might think.

5. It could give potential applicants a more realistic job preview. With the concern about tanking your job lessened, people will be more likely to be open about the good--and the bad--things about where they work.

You can envision how Google+ has broader applicability for organizations. It allows organizations to create employee-only groups. It allows employees to create informal social groups--or more formal interest groups. It adds another way for colleagues to share knowledge. And it helps create that intangible bond that connects co-workers in a way that meetings and off-sites never can.

So there is quite a bit of promise, but there are still many questions. Could Facebook add design elements to mimic aspects of Google+? Absolutely (and I strongly suspect they are in the process of doing just that). Could Google fail to attract enough followers to its new site to make it the killer app that Facebook is? Sure; in fact there's a good example of this in Orkut (although it is quite popular in Brazil and India). Are there examples of many social networking sites that have flared and fizzled? You bet (heard of MySpace?). And the juggernaut that is Facebook is not to be ignored.

But there are reasons to believe that this could be the real deal. Google spent a lot of time testing this thing out and appears to be listening intently to users on issues from design to privacy--something Facebook has been grilled about for as long as I can remember. And I didn't even touch on the other features of Google+, such as real-time group video chat.

The bottom line is when it comes to websites, most of us are followers. All it takes is your friends and colleagues to start posting somewhere else (heck, it's just another bookmark), and before you know it Facebook could start looking a lot like another casualty in the hyper-competitive web wars. Fortunately, organizations will be the better for it.

Should you hire more women for your teams?

Should you hire more women for the teams within your organization? You might think so after reading an article in the June 2011 Harvard Business Review. It's an interview with the authors of some research that came out last year in Science. In fact this hiring strategy has even been suggested based on this research.

But let's take a deeper look.


The takeaways from the HBR article (and the published study) suggest:

- there is a "collective intelligence" factor (c) that is related to team success

- this (c) factor out-predicts team success compared to the average team intelligence score, the highest intelligence score among the team members, or other logical factors such as group cohesion and satisfaction

- the (c) factor is primarily related to the average social sensitivity of the team members, the equality of distribution of turn-taking during team conversation, and...(drumroll please) the proportion of females in the group

In the studies, the authors had nearly 700 individuals (one assumes students? the subjects aren't described) participate in teams of two to five on a variety of tasks, such as completing puzzles, brainstorming, and negotiating. At the end of the session they had them complete the criterion task--in the first study a video game of checkers against a computer opponent, in the second, an architectural design task.

So what did they find? As Kai Ryssdal would say, let's do the numbers:

- There did seem to be some general factor that predicted a significant amount of variance in the criteria (43% and 44% respectively).

- (c) seems to be related, at least to a small amount, with both average individual intelligence (r=.15) and maximum member intelligence (r=.19), but the authors stress higher correlations with average social sensitivity of group members (r=.26), variance in the number of speaking turns (r=-.41), and proportion of females in a group (r=.23), although the latter was largely a result of the women scoring much higher on the measure of social sensitivity, which the authors stress came out on top in terms of unique prediction power.

- the instrument used to measure social sensitivity, the "Reading the mind in the eyes" test, has subjects identify the emotion being displayed by a set of eyes (reminiscent of some emotional intelligence tests I've seen). It would be interesting to see how well other measures of social sensitivity (e.g., body language, tone of voice) predicted team decision making, and tie this with other research that has shown emotional intelligence measures predicting team performance.

- The standardized regression coefficients (betas) for (c) were .51 and .36 for the two criteria, substantially above average member intelligence (.08, .05) and maximum member intelligence (.01, .12).

- (c)'s relationship with performance on the various tasks in Study 1 varied pretty widely, from .38 to .86. This, combined with the differential prediction of the criteria, suggests (c) as conceptualized may be more useful for predicting performance on certain group tasks. It's worth noting that the lowest correlation was with brainstorming--a task that requires less team interaction.

- The authors do not say what instrument was used to measure individual intelligence. This may or may not matter.


There are some important lessons here:

1. As is often the case, the farther we get from the actual publication, the more important it is to view the interpretation with caution. In this case, I believe some writers have over-emphasized and over-played the "flashy" result (more women on team -> better decisions) and failed to consider things like effect sizes or relationships among variables. What I'm more interested in is why the women scored higher.

2. From this research the concept of a collective intelligence factor does seem promising (and has been the subject of other recent popular publications). In reality this line of research is old as well as thriving, and includes such well-researched concepts as groupthink as well as several lines of research around what makes an effective team.

3. It is important to remember that job performance is multi-faceted. We know this from (among other things) previous research that has shown intelligence tests do a better job predicting task performance than contextual performance, where non-cognitive tests are at their best (this fact has interesting implications for the study that is the subject of this post). The results of this study remind us to carefully consider what behaviors we're hiring for.

4. It's studies like this that, when improperly analyzed, muddy the waters of our profession. Using this research to say you should hire more women is like saying you should hire more Whites than Blacks because they tend to score higher on intelligence tests. Aside from the obvious discriminatory intent, this is just plain bad decision making: it over-emphasizes differences at the group level and assumes that you have clear evidence that intelligence tests are highly correlated with performance in the job you are hiring for (and similarly valid tests with smaller mean group differences are unavailable).

I have to give the authors credit for going beyond gender as a causal factor in predicting team performance and looking for root relationships, and for not leaping to conclusions like organizations should hire more women, but instead focusing our attention on the implications for team development (they suggest electronic collaboration tools may increase collective intelligence).

This type of press is great for getting us to talk more about what matters. Let's just make sure when we do so we start with the research and consider all the important angles.

New emotional intelligence meta-analysis: Now with 65% more studies!

Emotional intelligence (EI) continues to be a hot topic in the I/O and HR communities. Some are big fans, some (including me) are more skeptical.

Last year's article in IOP by Cherniss, et al. and the accompanying commentaries provided a great overview of the current situation, with the bottom line that while many seem to agree on a definition of EI, measurement is all over the place, with different methods measuring different things and a distinct lack of convergent and discriminant validity.

EI has been shown to have important relationships with a variety of criteria, but many researchers and practitioners have been left wondering, what exactly is being measured?

In the latest Journal of Organizational Behavior, O'Boyle et al. attempt to shed light on the situation, primarily by looking at whether measures of EI add predictive validity above and beyond cognitive ability and five-factor personality measures.

You may remember a similar study published back in early 2010 by Joseph and Newman. So why the need for this one? The current authors point out that in addition to updating the data and using more current estimates of other relationships, "our data set includes 65 per cent more studies that examine the relationship between EI and job performance, with an N that is over twice as large." I don't know why I found this so funny--there's nothing wrong with trying to get a better handle on things--I guess it just reminded me of packaging on dishwasher detergent or something. In case you're curious, their final sample included 43 effect sizes relating EI to job performance.

ANYWAY, what did they find? In the words of the authors: "We found that all three streams of EI correlated with job performance. Streams 2 and 3 incrementally predicted job performance over and above cognitive intelligence and the FFM. In addition, dominance analyses showed that when predicting job performance, all three streams of EI exhibited substantial relative importance in the presence of the FFM and intelligence." [Stream 1 = ability measures, Stream 2 = self report, Stream 3 = mixed models]

Let's do the numbers: the correlation between EI and job performance varied depending on which "stream" of EI research was analyzed, from about .24 to about .30. So if you're considering using a measure of EI, any of the reputably-built measures should add validity to your process (and in fact these numbers may be low since the job performance measures were primarily task-based). Since the numbers were pretty similar, it seems they support a similar construct, right? Not so fast.

When they looked at how the streams related to personality measures and cognitive ability, they were all over the place. In their words, "For all six correlates (FFM and cognitive ability), we found significant Q-values indicating that the three streams relate to other personality and cognitive ability measures differently...These differences in how the EI streams related to other dispositional traits provide a contrasting perspective to the assertion that the various measures of EI assess the same construct."

What about the incremental validity? It ranged from pretty much nothing (stream 1) to a respectable .07 (stream 3) with stream 2 in the middle at .05. The stream 1 measures seem to have much more overlap with cognitive ability measures, and since ability measures dominate the prediction of performance, well...there you go.

So does that mean employers should avoid stream 1 measures (e.g., MSCEIT)? Again, not so fast. The authors point out that this type of EI measure may be more resistant to social desirability and faking effects--it primarily seems to lose its value when you're already using a cognitive ability measure. (Although again, we're talking task performance here)

All of which leaves me still wondering: what exactly is being measured? More to come I suppose; after all, EI is a newbie in the history of I/O constructs. The authors themselves point out that "much more work is still needed" on the construct validaty of EI.

On the other hand, the more practical among you may be thinking, "who cares? it works!" And to them, I would only say, remember what Kurt Lewin said: "There is nothing so practical as a good theory." Without the theory...

You can see an in-press version here. It's good stuff.

Oh, and on a totally unrelated point, I highly recommend upgrading your Adobe Reader to version X if you haven't already. Highlighting with bookmarking at the click of a button.

Mega research update

Time for a big research update. There's a lot to catch up on, so I'll go quickly and give you just enough for you to follow up on if you're interested.

- More support for the attraction-selection-attrition model

- A 25-year review of the study of leadership outcomes

- Faces with stereotype-relevant features are more open to prejudice

- Early socioeconomic experience plays big role in future risk-taking behavior

- Taking the perspective of another may be key to reducing racial bias

- Generational differences in Big 5 personality factors

- How to improve management research

- Pros and cons of using social networking sites to make HR decisions

- Research directions for talent management

- Organizational branding and "best employer" surveys

- Using social networking sites for hiring decisions (if we just had a single database...)

- How computer adaptive testing aids in delivering unproctored internet tests

- How broad dimension factors may improve assessment centers

- Practical intelligence predicts success as an entrepreneur

- Psychological capital (efficacy, hope, optimism, resilience) is related to job performance

- Expressions of personality factors varies with the situation

Last but not least, this gem, a meta-analysis of the efficacy of simulation games for instruction. Relevance for us? As active engagement went up, so did learning. Lessons? Consider interactive simulations for recruitment and selection, but make sure the viewer is truly involved and not just a spectator.

Can an applicant be TOO qualified? And...does it matter?

I've worked with people that seemed overqualified--you probably have too. They brought a bazooka when all that was needed was a squirt gun. Cognitively speaking.

They work amazingly fast, are incredibly innovative, but often seem bored or unhappy with their jobs. It's made me wonder what organizations can do with these folks to (1) get the most out of their skills, and (2) help them be satisfied with their work life.

Of course it's not just me who has ruminated about this issue. Recruiters and screeners are often faced with this question: do we take a chance on someone who seems to have much greater education, training, or competencies than the job calls for? Are they likely to simply turn around and leave? This issue takes on increased importance in this day and age with higher unemployment and more people simply looking for *A* job.

There have even been lawsuits about this issue. You may remember the Jordan v. New London case, where a law enforcement applicant sued the City for age discrimination after they failed to hire him because he scored too high (yes, you're reading that right) on a cognitive ability exam. (BTW, the City won)

But despite this case and recruiter perceptions, we're still left with a legitimate question: can an applicant be too qualified in ways that matter for an organization?

In the June 2011 issue of Industrial and Organizational Psychology, Erdogan, et al. provide an overview of this issue and argue that this situation deserves more attention than the scant it has received so far--particularly in comparison to other I/O topics.

What the research does seem to indicate is this: individuals that are overqualified (either objectively through, for example, having "too much" education, or through subjective perceptions) have more negative job attitudes. Specifically, they have been shown to have:

- lower job satisfaction
- lower life and career satisfaction
- lower organizational commitment
- higher turnover intentions

This makes intuitive sense. If you believe your qualifications far out-strip those required for the job, you're likely to feel underutilized and under-challenged, which in turn is likely to make you feel unsatisfied and cause you to look elsewhere.

What about actual turnover? Are overqualified applicants actually more likely to leave? This is one of the primary research questions and there is some research that supports this. However, the authors point out several problems with this line of thinking, including its cross-occupational nature.

In addition, the authors argue (and I agree) that more important than turnover is job performance. I think we'd all take a chance on someone that would have a huge positive impact, even if their stay was short.

So what about the performance of the overqualified; what does the research say? To quote the authors:

"There is...a small, but growing literature suggesting that overqualified individuals perform their jobs better than their less-qualified coworkers...whether turnover is good or bad for the organization needs to be considered within the context of how well individuals are performing their jobs." [emphasis added]

The authors also point out several additional advantages to hiring overqualified workers:

- they may be prime candidate for future roles, including leadership positions

- even if their tenure is short, their impact may be significant

They also point out that this is the tip of the iceberg of this research: much remains to be done in terms of defining overqualification, in studying how recruiters make these determinations, situational factors, measurement, and other important issues.

So bottom line: yes, there may be downsides to hiring someone that appears to be overqualified--but how certain are you that this is the case? And even if it's true, would the benefits outweigh the risks?

If organizations choose to go with someone they perceive as overqualified, it becomes even more important to communicate openly and frequently with this applicant about job expectations, as well as possibilities. And management needs to realize that they may be getting more than their money's worth.


(Interestingly, the other focal article in this issue has to do with performance management, and how difficult it is to get right. One of the points the authors make loud and clear is it's important for supervisors to offer frequent, informal feedback. The common thread between the two articles: the importance of engaged, communicative supervisors who have an accurate view of their team's competencies.)

Recipe for losing a lawsuit

Ingredients

One large, diverse candidate pool
One cognitive ability OR physical agility test
One protective services job (police or fire if in season, otherwise corrections)

Optional: large, aggressive employee union
Optional: history of litigation


Instructions

1. Begin by deciding what type of exam--cognitive ability or physical agility--you feel like giving; don't worry about performing a job analysis first as these are time consuming and boring. If you must, select a small sample of current employees (preferably the poor performers) and provide minimal instruction. Don't worry about whether they are "true" job experts, and whatever you do, don't link tasks to KSAs--everyone hates doing it.

2. Select what you will be measuring. Base your decision on what you feel like, or whatever's easiest. Usually this is just doing what you did last time.

3. Have untrained analysts prepare the exams. Because anyone can do hiring, select whoever has time on their hands. Optional: search the Internet for a test that catches your eye. Rule of thumb is one question per content area (if you have more than one question, you're wasting applicant time).

4. Make sure the reading level of the exam is graduate school-level. After all, isn't reading an important part of any job? And don't you want the best?

5. Next, choose weighting of your exam components either randomly or based on gut feeling. When in doubt, place the largest weight on the test that is most related to cognitive ability.

6. Select a pass point. It should either be: (a) 70 percent; (b) based on administrative convenience; or (c) chosen at random.

7. Administer the exam, preferably with limited advertising. If you must advertise, give applicants a very short amount of time to prepare--after all, this isn't grade school. Do not pre-test the exam--if you've followed these instructions, it should be fine.

8. Score the exam--if you can, avoid "right/wrong" questions and go with ones where you can personally judge the quality of the answer. Don't worry about a boring "benchmark"--you know a good response when you see it.

9. Keep all scoring results and details regarding the process to yourself. Candidates don't need to know (and won't understand).

10. Make final selection decisions. Do not administer yet another test before making your selection, or if you must because of boring rules, make it an unstructured interview. Ask lots of questions like, "If you were a book, what would your title be?" If you have any women or minorities, ALWAYS ask questions about their ability to perform the job.

11. Do not document any of this process. Everyone involved will be with the organization for a long time, and people have really good memories.

Above all: have fun! After all, it's just people's livelihood.


A good example of how this type of thing plays out (albeit in a less horrific manner) is the recent decision in Easterling v. Connecticut Dept. of Corrections.

And for those of you that want to read more about how the law applies to selection, look for an upcoming IPAC monograph written by yours truly!

Shameless plug: The 2011 IPAC conference

And now for something completely different. A man with a marketing video.

It's getting toward the end of May, so if you haven't made up your mind about attending this year's premier event for practitioners of assessment and selection methods, you might want to do it soon.

Details: July 17-20, Washington, D.C. Dupont Hotel. So many great presentations I can't even begin to summarize. Check out the details here.

If that doesn't convince you, I doubt amateurish marketing tactics will work, but since you're still reading and I have you captive...

IPAC Conference 2011 on Prezi

IJSA v.19 #2: Personality, personality, personality (and more)

The June 2011 issue of the International Journal of Selection and Assessment (IJSA, volume 19, issue 2) is out. And it's chalk full of articles on personality measurement, but includes other topics as well, so let's jump in! Warning: lots of content ahead.

- O'Brien and LaHuis analyzed applicant and incumbent responses to the 16PF personality inventory and found differential item functioning for over half the items (but of those only 20% were in the hypothesized direction!).

- Reddock, et al. report on an interesting study of personality scores and cognitive ability predicting GPA among students. "At school" frame-of-reference instructions increased validity and, even more interesting, within-person inconsistency on personality dimensions increased validity beyond conscientiousness and ability.

- Fein & Klein introduce a creative approach: using combinations of facets of Five-Factor Model traits to predict outcomes. Specifically, the authors found that a combination (e.g., assertiveness, activity, deliberation) did as well or better in predicting behavioral self-regulation compared to any single facet or trait.

- Think openness to experience is the runt of the FFM? Mussel, et al. would beg to disagree. The authors argue that subdimensions and facets of openness (e.g., curiosity, creativity) are highly relevant for the workplace and understudied--and demonstrate differential criterion-related and construct validity.

- So just when you're thinking to yourself, "hey, I'm liking this subdimension/facet approach), along comes van der Linden, et al. with a study of the so-called General Factor of Personality (GFP) that is proposed to occupy a place at the top of the personality structure hierarchy. The authors studied over 20,000 members of the Netherlands armed forces (fun facts: active force of 61,000, 1.65% of GDP) and found evidence that supports a GFP and value in its measurement (i.e., it predicted dropping out from military training). Unsurprisingly, not everyone is on the GFP bus.

- Next, another fascinating study by Robie et al. on the impact of the economy on incumbent leaders' personality scores. In their sample of US bank employees, as unemployment went up, so did personality inventory results. Faking or environmental impact? Fun coffee break discussion.

- Recruiters, through training and years of experience, are better at judging applicant personality than laypersons, right? Sort of. Mast, et al. found that while recruiters were better at judging the "global personality profile" of videotaped applicants as well as detecting lies, laypeople (students in this case) were better at judging specific personality traits.

- Last one on the personality front: Iliescu, et al. report the results of a study of the Employee Screening Questionnaire (ESQ), a well-known covert, forced-choice integrity measure. Scores showed high criterion-related validity, particularly for counterproductive work behaviors.

- Okay, let's move away from personality testing. Ziegler, et al. present a meta-analysis of predicting training success using g, specific abilities, and interviews. The authors were curious whether the dominant paradigm that g is the single best predictor would hold up in a single sample. Answer? Yep. But specific abilities and structured interviews were valuable additions (unstructured interviews--not so much), and job complexity moderated some of the relationships.

- Given their popularity and long history, it's surprising that there isn't more research on role-players in assessment centers (ACs). Schollaert and Lievens aim to rectify this by investigating the utility of predetermined prompts for role-players during ACs. Turns out there are advantages for measuring certain dimensions (problem solving, interpersonal sensitivity). Sounds promising to me. Fortunately you can read the article here.

- What's the best way to combine assessment scores into an overall profile? Depends who you ask. Diab, et al. gathered information from a sample of adults and found that those in the U.S. preferred holistic over mechanical integration of both interview and other test scores, whereas those outside the U.S. preferred holistic for interview scores only.

- Still with me? Last but not least, re-testing effects are a persistent concern, particularly on knowledge-based tests. Dunlop et al. looked at a sample of firefighter applicants and found the largest practice effects for abstract reasoning and mechanical comprehension (both timed)--although even those were only two-fifths of a standard deviation. Smaller effects were found for a timed test of numerical comprehension ability and an untimed situational judgment test. For all four tests, practice effects diminished to non-significance upon a third session.

Hiring HR professionals: What are we thinking?

When you hire someone for your Accounting department, what do you look for? Accounting experience, undoubtedly, but presumably you look for someone with some college-level accounting training as well as basic competencies such as facilities with numbers, conscientiousness, etc.

What about IT support? Again, in most cases you're probably looking for experience with specific hardware or software or general support experience, but in many cases you're searching that resume for formal education/training in IT-related topics.

Connection? For many organizational "support" functions, we look not only for experience but educational experiences that would give the individual a grounding in the basics of the field and (hopefully) train their mind to recognize historical developments as well as connections between concepts.

So why is that when we hire for HR, another support function, our brains fall out our ears and we seem to focus primarily on past experience? This weakness seems common in the public sector but I'm guessing the private sector is not immune.

Phrased another way: Why don't more organizations place value on formal HR education when hiring?

I'm not suggesting that one needs a degree in HR to be good at it, although I do think it limits people. What I'm concerned about is the apparent lack of importance placed on these degrees and what that says about the profession.

Is it because formal HR educational programs don't exist? Nope. According to the College Board, over 350 schools exist with a major in HRM.

Is it because formal education in HR isn't as important for job performance as experience? I'm not aware of any research that shows this to be true (if you are, please enlighten me).

No, I suspect the following:

1) Many HR leaders themselves do not have formal educational training in HR therefore they tend not to think of it as a screening tool (or place much value in it).

2) Similarly, there is a lack of knowledge about HR educational programs--what they offer, the advantage of having gone through one, and how to connect to the school.

3) There are relatively few candidates out there that apply for HR vacancies that have a relevant degree (either as a pure function of the number of individuals that have a degree in HR or because many applicants believe anyone can do HR).

4) HR is still seen as largely transactional and/or not a critical business function, therefore the qualifications sought have more to do with customer service than they do formal training. (I believe this is a large reason why HR outsourcing is easy to contemplate for many executives)

5) Many are simply passing through HR. Many incumbents do not see HR as a "career", but rather a stopping point on their way to...something else. But much like Lightning McQueen (or Doc Hollywood if you prefer), they find they have a hard time leaving, either because they come to like it or they find they're not as employable as they thought.

6) The professional HR organizations and HR publications focus on anecdotes, opinion, and news bits rather than formal study and analysis. SHRM is not SIOP.

So why do I care about this topic? Because I see HR stagnating until it truly becomes a profession and not a loose collection of people who vaguely care about things relating to people management. And part of becoming a true profession is placing formal structure around the path from education to employment.

I'm also concerned because of the relationship between I/O and HR. Ultimately much of what is researched in I/O gets practiced through HR, and there is a close relationship in many people's minds--in fact I would wager most managers haven't the foggiest idea what the difference is. So what impacts HR ultimately impacts I/O.

Maybe it's just not there yet. Maybe I need to be patient. HR's a relatively new field and maybe it just needs time to develop, and to figure out questions like its relationship to I/O.

But given what I've seen, I'm not feeling optimistic. I see HR shops being outsourced or automated, resulting in more IT skills being required than knowledge about research on human behavior. Inevitably this will lead many organizations to lose out on important efficiencies they could be gaining (not to mention improvements in the work environment).

What can be done? I don't have all the answers, just some suggestions:

1) A wider promotion of the value of formal HR education. SHRM, I'm looking at you, as well as the other HR professional organizations.

2) More research on the connection between formal HR education and job performance.

3) Effort on the part of HR leaders to at least consider the potential importance of HR education when hiring for their teams.

4) More effort on the part of HR leaders to establish connections to schools that offer HR degrees and begin programs like internships and formal recruiting.

5) More organizational support (e.g., tuition reimbursement) for staff to obtain HR degrees.


To read more about this issue, I highly recommend starting with the 2007 piece by Sara Rynes and her colleagues.

Hat tip to this HR Examiner article, which helped me crystallize something that's been bothering me for a long time.

Tech tools: Brainshark and Greenshot

Brainshark and Greenshot. Sounds like a kids' cartoon about a pair of superheroes.

But no, this post is all about a pair of simple tools that you can use in a variety of ways to enhance your recruitment and selection efforts and just plain make your life easier.

Brainshark is a remarkably simple, and free, tool you can use to create slideshows or videos with audio in a matter of minutes. Did I mention it's free?

Simply upload your file, add audio (if it's a PowerPoint) using either your computer microphone or phone, name your slides, and you're good to go! You can see a simple example of a promotional spot I whipped up below--remember, my specialty is information filtering, not multimedia! Make sure to check out the "Contents" menu to the right of the fast-forward button.



This took maybe thirty minutes to develop and record, and I used my computer's built-in microphone, hence the less-than-stellar audio quality. But I think you get the idea and see some of the possibilities:

- realistic job preview
- job advertising
- instructions for applying

And so on. For $10 or $20 a month, respectively, you can upgrade to Brainshark Pro or Brainshark Pro Trainer, which includes options like private sharing, capturing leads, testing, and integration with LMS. Just pretty darn cool all around, I say.


The next tool is simpler but no less useful on a day-to-day basis. I know I'm not the only one out there who likes SnagIt. It's an easy way to take screenshots of parts of the screen and quickly add borders, arrows, or other accents. And it's very reasonably priced.

But I'm all about the free stuff whenever possible, which is why I was pleased to learn about Greenshot, a scaled-down tool that gives you pretty much what you'd get in SnagIt, if a little less snazzy. Simply load the software and whenever you press Print Screen, instead of taking a picture of the whole screen you can specify the region. I use this all the time for showing others what I'm talking about, presentations, and user guides. You can also capture just the window, or the whole screen.

One feature unique (to my knowledge) to Greenshot is "obfuscate", which allows you to blur parts of the picture (e.g., name, SSNs) you may wish to hide. See the screenshot below for an example where I obfuscated part of the blog post title:



The one feature that I'd still use SnagIt for is capturing a rolling webpage that includes the links. Very handy. But other than that, Greenshot would do ya just fine.

So there you have it, two simple tools that have the potential to add tremendous value to your life. Hope you enjoy.

Hat tip to my colleagues at CODESP for turning me on to Brainshark, and my friends at Biddle for Greenshot.

Research update: Political skill, stereotype threat, and NFL players

A few research articles for us...

First up, several articles from the latest issue of Human Performance:

Lee & Dalal demonstrate in a policy-capturing study that performance "troughs" exceed "peaks" in their influence on performance ratings.

Next, a fascinating study by Meurs et al. where they show how political skill (or networking ability) moderates the relationship between the HEXACO factor of sincerity and task performance. In other words, for individuals high on political skill the authors found a positive relationship between sincerity and task performance (and a negative relationship for those low on the skill).

Are you recruiting highly educated graduates? Then you'll want to read Jaidi, et al.'s piece. In it, they describe a study where recruitment advertising and positive word of mouth related positively to job pursuit intention and behavior. Somewhat surprisingly, on-campus presence related negatively to these measures.

If you like football and/or physical ability tests, you'll be interested in the study by Lyons, et al. of NFL players. In it, they demonstrate that collegiate game performance out-predicted physical ability tests administered during the NFL Combine when looking at future NFL performance. And unlike physical ability, past performance remained a consistent predictor across four years of performance, although the criterion coefficients deteriorated over time, similar to what we find with cognitive ability scores.

Finally, over in the Journal of Applied Social Psychology, Nadler & Clark report the results of research on stereotype threat. The results of their meta-analysis indicated that attempts to nullify stereotype threat (e.g., by dismissing it or disguising the task) resulted in a moderate improvement in score (d=.52) for both African Americans and Hispanic Americans, and there appeared to be no difference between the groups in terms of the effect.

Sidenote: those of you with an interest in HR technology and talent management might want to check out the six sessions being streamed live from Bersin & Associates' IMPACT 2011 conference on April 27th and 28th.

Are executive recruiters discriminating against non-Whites?

You might think so if you just glanced at a recent study by Dreher, et al. in the Journal of Management. But what if we read the study...are the results more complicated?

Here's the background: The authors distributed a survey to over 13,000 U.S.-based BlueSteps subscribers. BlueSteps is an "executive career management service"--essentially a place where $100k+ job seekers and executive recruiters can connect, seemingly similar to TheLadders. It is maintained by the Association of Executive Search Consultants (AESC).

Due to various reasons (including the survey being blocked as spam), the researchers ended up with a final sample size of 572. So keep that in mind when interpreting the results. Also, nearly 90% of the respondents were male, and around the same number of respondents were White. So the non-white, non-male sample was relatively small.

What the researchers found was that white male respondents were significantly more likely to report being contacted by executive recruiters. And it wasn't because of factors like individual education or work experience--these were controlled for. Interestingly, further analysis revealed that race appeared to be the driving force; there was no difference in contact between female and non-White male respondents.

They also found that switching employers ("pursuing an external labor market strategy") resulted in a compensation premium only for White male respondents. Not surprisingly, those who reported receiving more contacts from search firms had greater compensation. But White males who had received the most number of contacts reported the highest level of compensation compared to other demographic groups.

Here's how the authors summarized their results: "...our study suggests that the White male advantage associated with external job change is, to a meaningful degree, sensitive to the processes and practices of executive search firms."

The authors do point out that the small size of the non-White male sample may have something to do with the results, and they caution readers about generalizing the results to other populations. However, they also state: "...the executive search industry would likely benefit from in-depth internal reviews regarding a variety of diversity-oriented issues. It is in the search firm’s likely best interest to present a diverse slate of candidates to its clients; as such, search firms would benefit from pursuing internal studies designed to determine if and why female and minority male managers and executives are underrepresented in their databases (or in databases of the BlueSteps variety)."

Is this an example of blatant racial discrimination? It's suggestive but not conclusive. The authors even write "we are not suggesting that anything sinister is going on or that search firms are intentionally discriminating against women and non-White males." But the fact that a large percentage of top search professionals appear to be White males should factor into the conversation. It may not be intentional, but that doesn't mean it's not discrimination. More research (e.g., controlled laboratory studies) would help us answer this question.

A little of this, a little of that

I've got a hodgepodge of things for you this time, some research mixed with some other interesting things.

First, the March issue of Journal of Applied Psychology, which has been out for a while and I've been a little slow to get to:

- Becker and Cropanzano on the (non-linear) relationship between job performance and voluntary turnover (people tend to skedaddle when they're on a performance skid).

- Podsakoff, et al. with a fascinating study of the impact of OCBs on interviewee ratings. Turns out they made quite an impact, particularly for the higher-level position, and particularly when candidates demonstrated low levels of OCB (e.g., helping, loyalty). Raters were probably surprised that a situation that so clearly calls for impression management failed to elicit it.

- McDaniel, et al. with a great piece on SJTs. The authors describe two adjustments that can be made to traditional SJTs that improve validity, reduce Black-White mean differences and score elevation due to coaching, and reduce total length. That all sounds pretty good to me, and you can read an in press version here.

- Last but not least, Swider, et al. report the results of a study on job search effort and voluntary turnover. Job embededness appears to play an important rule, as does job satisfaction and the availability of alternatives.

Speaking of McDaniel, he and his colleagues have written an article for an upcoming issue of Industrial and Organizational Psychology with the provocative title of "The Uniform Guidelines are a detriment to the field of personnel selection." SIOP members should be sure to consider submitting a commentary, and even if you're not a member, you should check out the in press version; it's a good read.

Speaking of adverse impact...I attended a fascinating webinar sponsored by PTC/MC on Wednesday where Kenneth Yusko of Marymount University described the development of the Siena Reasoning Test, which uses a slightly different question type (along with some other techniques) to reduce d but maintain criterion-related validity. Provocative stuff, and one of the the holy grails of personnel assessment. Which probably explains why Yusko and his colleagues are being presented with the 2011 M. Scott Myers award at this year's SIOP conference. Interested? Check it out yourself. You can also flip through slides from a similar presentation at the 2008 IPAC conference.

Speaking of IPAC, have you registered for the July conference in D.C.? It's shaping up to be another great one--just check out some of the pre-conference workshops.

Finally, if you're up for a little heated discussion, head on over to ERE, where Wendell Williams laments about the increasing number of people who claim to be "experts" in assessment but who lack the chops. He particularly calls out poorly informed bloggers. Hey...wait a minute...

Seeing through the candidates' eyes

"A person-centric work psychology will look at the world through the eyes of the worker. It will take work in the raw, all the sights and sounds and tasks and people that run by the worker everyday. The worker experiences all of it, not demarcated parts of it. She organizes and partitions to be sure, but it is her organization that must interest us." - Weiss & Rupp

"Before you criticize someone, you should walk a mile in their shoes. That way when you criticize them, you are a mile away from them and you have their shoes." - Jack Handy

The two quotes above represent different ways of looking at someone else. The first, from a recent journal article, suggests we benefit from studying the world as others see it. The second, while humorous, suggests an alternative--and more common--worldview, where people are really just a means to our ends. Unfortunately in the world of recruitment and selection, all too often we're stealing shoes rather than practicing (and studying) empathy.

Fortunately there's been quite a bit of talk lately about seeing things from the perspective of the applicant, particularly in the recruiting world. Joe Murphy, among others, has done a lot of writing and thinking about this. And in the latest issue of I/O Psychology, Howard Weiss and Deborah Rupp bring our attention to the fact that I/O research tends to treat people as objects. (e.g., a collection of KSAs, a test score). They also point out that much of I/O research is done for the "collective purpose", driven explicitly to serve organizational needs, rather than beginning with the person. It's one of the best written, most thought provoking pieces I've read in a while. And it convinced me that we need to spend more time studying the experience of working; or for our purposes, the experience of applying for a job.

There is a lot of emphasis these days on reaching out to prospective applicants via social media. I'd like to respectfully submit that for many organizations this is putting the cart before the horse. In fact it may cause the horse to trample you. Because we don't even do a very good job communicating with EXISTING applicants. So you could very likely be trying to build a brand or reputation that's already been sullied. Yeahh..good luck with that.

And think about how important it is to treat applicants right. Not only can you get easily razzed on Facebook, you're dealing with people at one of their most vulnerable times. Think about when you really NEED an organization and how it impacts your sensitivity. What state of mind are you in during a medical emergency? When your car needs to be fixed? When you REALLY need a plumber? Looking for a job, whether because you're unhappy in your current one or, even worse, because you don't have one, puts people in a vulnerable, and sensitive, state of mind.

Do we see things from the eyes of our applicants? Do we treat them as customers? Because in many cases they literally are, and if they're not they're connected to people that are.

If we do see them as customers, then why...

Do we not get back to them, even with a simple automated acknowledgement, when they apply?

Do we not give them any feedback about their performance on assessments?

Do we structure our processes and career portals around what makes our lives easier, not theirs?

Do we wring our hands about how hard it is to weed through piles of applications, when most ads do a horrible job of serving their purpose (attracting the RIGHT applicants) and we give people very few tools to use to self screen (e.g., realistic job previews)?

In our defense, there is research done in this area (e.g., perception of selection tools and websites, organizational attraction), but there are many more opportunities, as pointed out in the commentaries. For example:

- What does it feel like to be recruited? To be passed over?

- What does the experience of taking a particular assessment for a particular opportunity feel like? Exciting? Frustrating? Confusing? Engaging?

- What are applicants attending to while taking a test? (we assume it's just the test)

- Does having fun while being assessed add to the organization's perception, or lower test anxiety?

- How does an applicant describe their assessment process to others?

If we want to explicitly tie these to organizational objectives, we could take the results of this research and answer questions like: How do candidate/applicant perceptions translate into socialization, turnover, attitudes, and job performance?

From a broader perspective, this article reinforces out that most organizations need to adopt much more of a systems perspective of the selection process. And there are other implications. Perhaps if we knew more about the experience of working, it would help us understand job performance better. And that would help explain the large amount of variance that goes unaccounted for by formal testing.

To some extent, none of this is new. Recruitment and selection research is all about understanding applicants. But it's done so through the lens of the organization--in other words, it looks at what applicants bring to the table. It's a slight shift to think about studying applicants as individuals and focusing on their experience. But doing so might help both organizations and individuals find a better match.

Evidence-based I/O: Where are we? More importantly, who?

The March 2011 issue of Industrial and Organizational Psychology contains two excellent focal articles. One, which I plan on writing about in the future, is about how the field needs to spend more time studying the experience of working rather than treating workers as objects. It's one of the best, most thought-provoking articles I've read in this journal.

But today I'll focus on the first article, which is about the evidence-based practice (or lack thereof) of I/O psychology, by Briner and Rousseau (B&R). This topic is obviously near and dear to me given this blog so I had a lot of reactions during my read of it. I'll share some of those with you today.

B&R argue that the practice of I/O psychology is not strongly evidence based--at least not in the sense that other professions (e.g., medicine) are becoming. This may surprise you given the history behind I/O psychology and its strong grounding in sound research methods. But we're not talking about the quality of research (although as some of the commentaries point out, this is a relevant question), rather how well we're doing putting research findings into practice.

The authors point out that there are many "snake-oil" peddlers out there who claim to have evidence for what they do. That the concept of evidence based is not used or well known in I/O circles. Even more fundamentally, we have no data on what percentage of I/O practices are based on solid scientific evidence. They point out that part of the problem is that the latest research findings and research summaries aren't accessible--a point I obviously agree with given the purpose of this blog. They even provide a tool (systematic reviews) that they suggest could help us get there.

Yet it's one particular point, referenced by both focal and commentary authors but not treated in depth, that I'd like to focus (okay, rant) on. I'll use a quote from B&R to get us started: "It is not always obvious to practitioners, certainly not the least experienced or less reflective, how exactly to apply the principles identified in such research." This comes as close as any to, IMHO, the real issue: who uses I/O research in organizations.

Before I go too much further, I'd like to do something that I don't believe any of the authors did: acknowledge my bias in approaching this topic. I'm someone who has an advanced degree in I/O (a Masters, which paraphrasing the focal authors puts me slightly above the village idiot), but who works in the trenches assisting supervisors and managers alongside many who have little if any formal education in either I/O or HR. This obviously impacts the issues I see as important and my take on them. 'Nuff said.

Now, back to my point. I believe the authors in this issue fail to recognize a very important point: that although I/O research is used by I/O consultants and academics, and to a lesser extent by mid-level managers, there's a large group who is going unnoticed: HR practitioners. It is these individuals who I would argue in most organizations are the primary "users"--whether they know it or not. These are the folks who supervisors count on to provide expertise related to a wide variety of HR/IO issues, including recruitment and selection. Yes, many organizations use consultants who have formal training in I/O, but I think we can all agree that in terms of the number of day-to-day decisions that get made related to HR, we're primarily talking about supervisors and HR practitioners. And only after recognizing this point do answers to the question "how do we make I/O practice more evidence-based?" become more clear.

Let me throw out a couple questions just to stimulate you a bit more:

1) Why is this one of only a handful of "publications" devoted to making research more accessible to HR practitioners? (and heck, I don't even know how many of you are practitioners in the first place!) And why is it left up to "independents" such as myself, rather than professors? (Dennis, you are a blessed exception)

2) Why does SHRM, with its enormous resources and user base, focus on things like HR strategy and leadership, while failing to focus on evidence based practice? (and while I'm ranting, Sir Richard Branson as keynoter? really?)

3) Why is SIOP seemingly only now beginning to make attempts to make research more accessible (e.g., with its randomly updated blog and thankfully soon-to-be published Science You Can Use series)?

I would argue it's all because we do a horrible job of focusing on the true users of I/O research. We engage in high-level debates about criterion-related validity but fail to gather even basic information like who is responsible for making each type of HR decision.

And unlike evidence-based medicine, we have a particular responsibility to ensure that HR/IO decisions are made using the best science. Because the consumers of the research aren't all individuals with advanced training (e.g., doctors). They're supervisors who are under the gun to make an effective, and ass-covering, decision. They're HR analysts who ended up there not because they love the topic but because they were looking for a promotion.

So if the goal is to increase the number of people-related decisions in the workplace that are based on the best evidence, in addition to tackling the issues identified in this issue (such as publication bias and accessibility), we need to do a much better job of understanding our customer base and tailoring our efforts toward them. After all, I/O psychologists generally do not control organizational practices.

So here, in no particular order, are my own recommendations for helping ensure the "best" (defined here are based on science, not things like, oh, I dunno, organizational politics) people decisions get made:

1. Make basic research more accessible--and by this I mean affordable. And by this I mean free. Or cheap. Fifteen dollars for a research article? Really? How many of those are you selling, exactly, publishers? Somebody, I don't know who know, needs to get on this.

2. Start addressing the elephant in the room: that many of those providing guidance to supervisors (i.e., internal HR), not to mention supervisors themselves, are doing so based on absolutely zero research. Who are these people? How are they trained? What do they know? We know very little, because we don't study them (with some exceptions). Imagine if evidence-based medicine tried to progress without understanding anything about doctors.

3. In a similar vein, spend more time understanding those who are ultimately responsible for people decisions and have to live with them: supervisors. Particularly first-line ones. Why--and more importantly how--do they make decisions related to particular HR issues? To what extent is speed the single most important decision criterion?

4. Identify the "high value" people decisions (e.g., selection, harassment prevention) and their current practice. Use this to figure out where the biggest gap is and where we should be channeling resources. Without a baseline it's hard to figure out where we should be focusing.

5. For Pete's sake, let's start putting our "blessing" on certain practices. SIOP, don't be afraid to endorse things. By remaining "objective", you're de facto taking the stance of, "hey, figure it out yourself, or hire a high-priced consultant." There's a thought behind products earning an "energy star."

6. Let's have an industry-approved training curriculum and certification. Why does SHRM offer the PHR and SPHR...and that's pretty much it? We need training programs for practitioners that are created and reviewed by people steeped in the research but able to translate.

7. Let's identify "best of" practices as well as "worst of." Why are most awards for individual researchers? Why not organizations that demonstrate particularly effective practices? Why not have a professional equivalent of the Razzies for just the opposite? ("and now...the award for worst interview question of 2011 goes to...")

Okay, I think that's the end of my rant. Let me be clear, the proposition that people decisions should be based on the best available science is hard to argue with. I applaud all of the authors for thinking deeply about this topic. I just think we need to get in the trenches a little more. Because if we don't, the trend that's already apparent (toward automation/speed and away from thoughtful, research-based decisions) will accelerate. Leaving lots of people trained in I/O but few who understand--and are interested in--their services.

Side note: this article did point out a couple books I'm considering adding to my library, including Locke's handbook on organizational behavior (probably still too much for the average consumer in this age of Twitter, but on the right road), and Locke, et al.'s book on evaluating HR programs.