Top 10 Assessment and Recruitment Research of 2013

So I'm gonna try something a little different this year.  I'm going to present the research from 2013 that I think has the best chance of fundamentally changing research directions, has the biggest implication for practice, or is just plain interesting.

So without further ado, and in no particular order, here are my choices for the Best Assessment and Recruitment Research of 2013:

1) Murphy, et al. call into question one of the fundamental assumptions of test development: that judgments of subject matter experts have a direct relationship to test utility.

2) Kim, et al. demonstrate a real value of age diversity in work groups: better emotional regulation.

3) Ghumman and Barnes with a simple but elegant study that demonstrated how important sleep is in preventing a persistent thorn: prejudicial assessments by raters.

4) Konradt, et al. showed that perceptions of fairness matter in web-based assessments too.

5) Personality research continued to dominate in 2013, and one of the best studies was by Shaffer and Postlewaite.  In it, they demonstrate that conscientiousness is best used as a predictor of performance in highly routinized jobs.

6) Mrazek et al. focused on a topic near and dear to my heart: mindfulness.  They showed that training in mindfulness increased GRE scores.  The implication for employment testing is clear, we just need more research in that direction.

7) Early in the year, Bobko and Roth gave us one of those "I better print this out" articles, showing that assessment methods historically assumed to result in lower levels of adverse impact, like biodata and work samples, may be more prone to d than we thought.  Side note: this article is still free in its entirety!

8) Kuhn, et al. presented the results of an elegantly simple experiment illustrating that the impact of a minor resume embellishment depended on the pre-existing perception of the applicant.

9) Discrimination, sadly, knows no demographic boundaries.  In this study, Conley found that non-Whites described White women as attractive, blonde, ditsy, shallow, privileged, sexually available, and appearance focused.

10) In a study of M.B.A. program admission judgments that has implications for employment selection, Simonsohn and Gino found that as the day progressed, fewer applicants were rated as highly recommended if several were recommended earlier in the day.

There were, of course, many other well-done and interesting studies in 2013, but these were some of my favorites.  Here's to a productive, stimulating, and successful 2014!

Research update

Well, it's that time of year again.  No, not the holidays.  No, not winter (or summer, depending on where you are!).  Research update time!  And I think you will agree with me that there is a lot of interesting research being reported, on traditional topics as well as emerging ones.

First, the November issue of JOB:

- Do transformational leaders increase creative performance and the display of OCBs?  Well, that may depend on how much trait affectivity they had to begin with. A reminder to not make blanket statements like "X type of leadership causes Y type of behavior."

- There is seemingly endless debate about the utility of personality inventories.  This study reminds us--again--that in assessment research there are few simple answers.  The authors describe how a particular combination of personality measures correlated with task performance among professional employees, but not non-professionals.  (yes, I said task performance)


Next, the Winter issue of Personnel Psychology (free right now!), much of which is devoted to corporate social responsibility (CSR):

- Do perceptions of CSR drive job pursuit intentions?  It may depend on the applicant's previous justice experiences and their moral identity.

- Oh, and it may also depend on the extent to which applicants desire to have an impact through their work.

- There is a debate in the assessment center literature about whether competency dimensions are being measured or if it's purely a function of the assessment type.  This study suggests that previous research has been hamstrung by a methodological artifact and that measured properly, assessment centers do in fact assess dimensions.


Let's switch to the November issue of the Journal of Applied Social Psychology:

- Engagement is all the rage, having seemingly displaced the age-old concept of job satisfaction (we'll see).  This study reminds us that personality plays an important role in predicting engagement (so by extension our ability to increase engagement may be bounded).

- Here's another good one and it's related to internal motivations.  The authors developed an instrument that helps organizations measure the "perception of the extant motivational climate."  What does that mean?  As I understand it, it's essentially whether most people are judging their performance against their peers or their own internal standards.  It seems the latter may result in better results, such as less burnout.

- On to something more closely tied to assessment: letters of recommendation (LORs).  There's surprisingly little research on these, but this study adds to our knowledge by suggesting that gender and racial bias can occur in their review, but requiring a more thorough review of them may reduce this (I don't know how likely this is for the average supervisor).

- Finally, a study looking at the evaluation of job applicants who voluntarily interrupted their college attendance.  Unfortunately this does not appear to have been perceived as a good thing, and the researchers found a gender bias such that women with interrupted attendance had the lowest evaluations.


Next, the November issue of Industrial and Organizational Psychology, where the second focal article focus on eradicating employment discrimination.  This article looks pretty juicy.  I haven't received this one yet in the mail, so I may have more to say after digesting it.  There are, as always, several commentaries following the focal article, on topics including background checks, childhood differences, and social networks.


Okay, let's tackle the 800-pound gorilla: the December issue of IJSA:

- Are true scores and construct scores the same?  According to this Monte Carlo study, it seems how the scales were constructed makes a difference.

- Can non-native accents impact the evaluation of job applicants?  Sure seems that way according to this study.  But the effect was mediated by similarity, interpersonal attraction, and understandability.

- Here's a fascinating one.  A study of applicants for border rangers in the Norwegian Armed Forces showed that psychological hardiness--particularly commitment--predicted completion of a rigorous physical activity above and beyond physical fitness, nutrition, and sensation seeking.

- Psst....recruiters...make sure when you're selling your organization you stay positive.

- Spatial ability.  It's a classic KSA that's been studied for a long time, for various reasons including its tie to military assessments and the finding that measures can result in sex differences.  But not so fast, spatial ability is not a unitary concept.

- Another study of assessment centers, this time in Russia and using a consensus scoring model.

- And let's round it out with one that should rock some worlds: the authors presents results that suggest that subject matter expert judgment of ability/competency importance bore little relation to test validity!  Okay, I'm really curious about what the authors say about the implications, so if anyone reads this one, let us know!


Last but not least, the November issue of the Journal of Applied Psychology:

- Another on personality testing, this one underlining the important distinction between broad and narrow traits.  This is another article I'm very curious about.

- Here's on one leadership: specifically, on the impact of different power distance values between leader and subordinates on team effectiveness

- And another on nonnative speakers!  This one found discriminatory judgments made against nonnative speakers applying for middle management positions as well as venture funding.  Interestingly, it appears to be fully mediated by perceptions of political skill--a topic that is hot right now.

- Okay, let's leave on a big note.  This meta-analysis found an improvement in performance prediction of 50% when a mechanical combination of assessment data was used rather than a holistic (judgment-based) method.  BOOM!  Think about that the next time a hiring supervisor derides your spreadsheet.

Until next time!

Will robots replace assessment professionals?

Technology and assessment have had a close relationship for years.  From the earliest days of computers, we were using them to calculate statistics, store items, and put applicants into spreadsheets.

Over time as computers advanced, we used them for more advanced tasks, such as multiple regression, applicant tracking, and computer-based testing.

With the advent of the Internet, a whole new area of opportunity opened for us: web-based recruitment and testing.  People began "showing off for the world" by creating personal webpages, commenting on articles, writing blogs, and living their lives through online social networks.  We developed Internet testing, allowing applicants to examine more conveniently.  And new forms of assessment opened up, such as advanced simulations.

We now find ourselves evolving yet again to take advantage of another significant technology advance: the social web.  As millions and billions of people began living their lives publicly on the web, they began developing a web identity and leaving footprints all over the place.  It was only a matter of time before recruiters (historically some of the first in HR to embrace technology) figured out how to harvest this information.  

One of the hottest trends now in HR technology is scouring the web to seek out digital footprints and making this information readily available to recruiters.  It's the latest iteration of Big Data applied to HR, and it's a creative way to make Internet recruiting more efficient.  Companies like Identified, TalentBin, Gild, and Entelo offer solutions that purport to lay qualified applicants at your doorstop, without all the hassle of spending hours manually searching the web.  They claim an additional benefit of targeting passive job seekers, who are obviously more challenging to attract.

But just exactly how big of an evolutionary step is this?  How big of a solution will this be?  Will this next evolutionary step result in us working ourselves out of a job?

I don't think so.  And let me explain why.

Fundamentally, assessment is about measuring--in a valid, reliable way--competencies key for successful performance in a field, job and/or organization.  Assessment can be performed using a number of different methods, the biggest ones being:

- Ability testing.  Measuring things like critical thinking, reading comprehension, and physical agility.  These tests seek the boundaries of individuals, the maximum they are capable of demonstrating related to a variety of constructs.  When properly developed and used, these tests have been shown to be highly predictive of performance, although some can result in adverse impact.

- Interviews.  One of the oldest forms of assessment and still probably the most popular.  Like ability tests, interview questions can seek "maximum" performance (i.e., knowledge-based), but they can also be used to probe creativity (i.e., situational) as well as gain a better understanding of someone's background and accomplishments (i.e., behavioral).  Interviews have also been shown to be valid predictors of performance, although they rely heavily on potentially unrelated competencies such as memory and verbal skills.

- Knowledge testing.  SAT or GRE anyone?  Multiple-choice tests have been around a long time, and with newer technologies like computer adaptive testing, don't show any signs of going away any time soon.  While these used to be quite common in employment testing, they have fallen out of favor in many places, which is odd given that they too have been shown to be successful predictors of performance (I suspect it is due to their "unsexy" nature and the fact that they require a significant amount of time to prepare)

- Personality inventories.  While these haven't been used nearly as much as the others above, there is an enormous interest in measuring personality characteristics related to job performance.  While they sometimes suffer from a lack of face validity (although contextualizing them seems to help), they have been shown to be useful, and typically demonstrate low adverse impact.

- Applications.  Also extremely popular, and the most relevant for this topic.  The assumption here is that qualifications and (like behavioral questions) past accomplishments predict future performance.  There is potential truth here, but as we know relying on applications (and resumes) is fraught with risks, from irrelevant content to outright lies.


An important thing that all of these assessment types have in common is that they are employer-generated.  One of the fundamental changes society has seen in the last ten years is an enormous shift to user-generated content at the grass roots level.  Anyone can have a blog, regardless of qualifications, and many of questionable veracity are read more than those written by people who actually know what they're talking about.  Content has become, if it wasn't already, king/queen.

But therein lies the fundamental challenge for aggregating digital footprints/content for use in assessment.  Relying on user-generated content, whether from social networks, blogs, comments, or other sources, is predicated on the assumption that qualified candidates are leaving digital versions of themselves.  In places that you have access to.  And that it is accurate.  And predicts performance.  This may work decently in certain industries, like IT, where it may be nearly universal--and expected--that professionals live their lives publicly on the web.  But for many people in many different professions, they may have neither the time nor the inclination to reveal their qualifications online.  In contrast, you can always test someone's ability, and a significant advantage of ability testing is it gives candidates an opportunity to demonstrate what they can do even if they haven't had the chance to do it yet.

I should note that using this information for recruitment is a different--but related--animal.  In this context, concerns about replacing tried-and-true assessment methods are moot.  However, we should carry the same concerns about content generation, both frequency and veracity.

As I've said before, taking technology to its logical endpoint would result in a massive database of everyone on the planet and their competency levels.  This database would empower users to generate and control their content, but allow organizations the widest possible field of qualified candidates.  At this point I'm aware of only one thing that comes close, and honestly I don't see anything approaching this scope anytime soon, particularly with more and more concerns over digital privacy.

Which leaves us...where exactly?  Will robots replace assessment professionals?  Not anytime soon.  At least not if we want hiring to work.  But we should be active observers of these trends, looking both for opportunities as well as pitfalls.  We shouldn't fear technology, but rather the way it's used.  Any important endeavor that requires human analysis should use technology as an assistive tool, not a sexy replacement.

I also want to give props to these companies for taking advantage of user-generated content.  It's a much more efficient way of assessing (i.e., it doesn't require applicants to in some sense double their efforts by completing a separate assessment).  And it's not surprising that these companies have sprouted up, given the trend in HR to automate user-initiated activities that lend themselves to automation, such as leave requests, benefit changes, and training.  But importantly, the science of whether digital footprints predict real-world job performance is in its infancy.  With something as important--operationally as well as legally--as hiring, we have to be careful that our addiction to technology doesn't outstrip our evidence that it works.

2013 HR Tech Conference: #giantexpo #echochamber #wrongfocus?

I just finished attending the 2013 HR Technology conference in Las Vegas, something I've wanted to do for many years.

It was at a great venue (Mandalay Bay), well organized, and full of industry experts.  And vendors.  Lots and lots and lots and lots of vendors in an enormous expo hall.  I'm pretty sure the city I was born in could have fit in there.

But I'm getting ahead of myself.  For anyone interested in recruiting products or HCM suites, this is the place to go to get educated.  Or sold to.  Ok, a little of both.

What about assessment, you ask?  Well, Hogan had a presence...I saw PAN there...but the floor was dominated by the big HCM vendors like SucessFactors/SAP, Oracle, and Workday (shown at the top, who had a great booth btw).

While sitting in a milieu of thought leaders like Don Tapscott, above on the jumbotron (one of the highlights--check out this video he talked about in the context of ease of collaboration driving social change), industry experts, and product VPs naturally results in some learning through osmosis, I came away feeling, to misquote Flynn Rider, "kinda like the color brown."  It took me a while to figure out why I felt that way, but it came from a place of personal disconnect.

See, much like putting lipstick on a pig doesn't change the fact that it's still a pig, you have to put gas in that fancy new car.  What the hell am I talking about?  Lemme 'splain.  Getting the right people into your organization is critically important.  You have to have the right ingredients to make that fancy cake.  You have to get the right people on the bus.  You have to have coal to make diamonds.  Pick your fav metaphor.

But that highly skilled, friendly, conscientious person you just hired ain't gonna go very far--certainly not as far as they could--without a focus on what happens after they start.  And it's here that I didn't see nearly as much tech innovation as we need.

Creating recruiting and employee tracking tools for HR pros is great.  We need those tools.  Particularly ones that don't make you go cross-eyed trying to figure out the GUI.  But here's the thing:  HR is a support function.  We exist to catalyze other functions.  Our product is a higher performing organization, and we get that through working with our business units.  This doesn't mean HR's not important or strategic, that's just reality.  So tech products should, at the end of the day, really be about program supervisors and managers. They don't have to be USED by them (although that would be ideal), but the goal should be to make their lives easier.

But you would think attending this conference that attracting the right people and getting their information into your HCM was the be-all and end-all of HR management.  Is recruitment and hiring the most important step of HR?  Quite possibly.  But it sure as heck ain't the only issue on my mind as an HR manager.

Let me share with you some of the biggest people challenges I've seen on a daily basis in organizations:

- getting employees to come to work (seriously)

- getting employees to do their jobs

- reducing bureaucracy so it doesn't take five years to hire or fire someone

- getting employees to work as a team and squashing negative gossip

- getting supervisors to pay attention to their people and, ya know, document it every once in a while

- getting HR staff passionate about HR

- getting HR and line supervisors to work TOGETHER to solve people problems

How much focus did I see at the conference on the above? Very, very, very little.  Maybe I was looking in the wrong places.  Maybe I was dazzled by the big sponsors.  Tell me I'm wrong.

Heck, maybe technology can't solve these problems.  Maybe these are examples of things that take face-to-face communication and won't be solved by "analytics" (by the way, if you attend this conference it is law that you use this phrase in at least every other sentence).

But I suspect there's a lot more innovation to be had here.  Ways of connecting HR and supervisors.  Getting HR jazzed about what they do.  Keeping employees engaged, or at least as much as we can.  THAT'S what I want to see.  That's what I hope to see in the coming years.  I fervently hope that we move away from talking about a talent shortage and focus more on making sure our houses are in order.     

Am I being unfair?  Most likely.  Were there products there that focused on things beyond recruitment, CRM, and core HR?  Sure, including a cool one that facilitates volunteerism.  But they felt like an afterthought.  Maybe the market gets what it demands, but I suspect it's just what's sexy.  Recruitment is sexy.  Progressive discipline?  Not so much.  But which one takes up more of our time?  Just like you don't want to hire too many pigs, you better buy some gas for that car or you won't get very far down the road.  That's what I need--more fuel.

So that's it, that's my perspective.  Next time I promise it'll be back to focusing on recruitment and selection.

By the way, a lot of people at this conference really need to have their hashtags and @ signs taken away, I think they're addicted.  At the very least they need to watch the Jimmy Fallon/Justin Timberlake vid.

Research update: September, 2013

Okay, it's mega research update time!

First off, the September IJSA; lots of good stuff, including:

- a constructed response multimedia test for entry-level police resulted in minor ethnic group differences

- panel interviews once again prove their superiority (also: more on interview reliability)

- further analysis of the Hogan Personality Inventory with a Spanish sample

- how to applicants form impressions of person-organization fit?  This study suggests contextual factors may be more important than interview content

- circumplex traits (combinations of personality factors) may predict counterproductive work behaviors better than simple FFM scores

- speaking of CWBs, conditional reasoning tests may not be the best predictor of them

- last but not least, what looks to be a good overview of competency modeling


Next up, the September JAP:

- an interesting, large study of the impact of candidate reactions on test scores, organizational perception, and criterion-related validity

- a study of the dynamics of the job search process and the impact of efficacy and focus

- highlighting certain factors during an interview may reduce discrimination toward pregnant applicants


Next, the Autumn 2013 Personnel Psychology:

- first, an important study of self-efficacy that suggests it is a product of past performance and not necessarily a predictor of future performance (free right now!)

- second, a study indirectly on selection that suggests that age diversity in work groups leads to more emotion regulation


Let's move on to the September JASP:

- okay, this may be a bit of a stretch, but if you're considering interviewing for a position as a dentist or a lawyer, make sure you suit up

- knowledge of service encounters predicts service effectiveness (and is related to conscientiousness)

- can use of biodata instruments result in adverse impact?  This study suggests so, but also suggests that removal of problematic items has no impact on validity


Starting to wrap up, let's move to the October JOB:

- perceptions of the fairness of promotion practices is one of those "bubbling beneath the surface" issues in most organizations.  This study found that perceptions are impacted by having been promoted in the past, organizational commitment, and ego defensiveness.  Good stuff.

- do more creative sales agents produce higher sales?  Perhaps only when there is a high quality of leader-member exchange.

- is validity generalization overgeneralized? (say that five times fast)  These folks seem to think so.


In the home stretch, from the September Psychological Science:

- older employees may have lower average cognitive performance, but it's more consistent

- spatial ability has a valuable role to play in the development of creativity, and can predict things like patents and publications


Second to last, for you stats geeks out there, a study that suggests that t-tests can be used reliably with small samples, thank you very much


Finally, something that has nothing to do with selection but is a nominee for the 2013 HR Tests Coolest Study Award, and something we all are very familiar with: time bandits (no, not the movie).

Hiring interviews: You're probably doing them wrong

I'll have a research update for you as soon as I get enough content, but today I wanted to give you a preview of a talk I'll be giving in San Francisco on September 23rd titled "Hiring interviews: You're probably doing them wrong.  A motivational talk."  The topic was inspired somewhat by Google's semi-recent announcement regarding their internal research on the issue.

The sad truth is that even though interviews are one of the most popular forms of personnel assessment, they are often done wrong.  Not necessarily through any ill intent, but because of two main factors:

(a) they're harder to develop than many people think

and

(b) most people think they're great interviewers


On the first point, interviews are deceptively simple.  Many people assume that if they can talk to other people, they can interview someone.

Wrong and wrong.

Interviewing isn't talking to people.  Well, okay, I suppose literally it is.  But interviewing really is about MEASURING people.

If I asked you to use a set of measuring spoons to give me 1 tsp of sugar, you would have no problem, right?

But what if I asked you to use a Halloway P36 spectrometer* to measure photon radiation?  You might need some help.  It's all about what you're measuring.


On the second point, research has well established that people are generally very bad at accurately reporting their skill levels--across a wide variety of disciplines.

To make matters worse, research has also established that interviewers tend to get addicted to bad interviews: when things turn out poorly, they tend to blame outside factors rather than the interview format.

But there is good news.

Specifically, we know how to do interviews the right way.  Namely by structuring them.  What does this mean?  There are several key features.  Here are a few:

1.  Use high quality question formats.  This means behavioral, hypotheticals, knowledge-based, and background questions.  Not puzzle questions or gimme questions.

2.  Be consistent.  Each candidate should be asked the same questions in the same order (with limited variations for follow-up if needed).

3.  Use a detailed rating scale.  When there is no criteria to compare answers to, scoring tends to be inconsistent, reducing the utility of the whole process.

4.  Base the questions on an analysis of the job.  I probably should have made this #1.  Everything you do in your selection process, including interviews, should be driven by the key knowledge, skills, abilities, and other characteristics required for good job performance.

5.  Train the interviewers.  Because of the wide variety of biases that plague interviewers, it's critical that they be aware of these tendencies and guard against them.


These are just some of the ways we can avoid bad interviews.  Of course another strategy to increase your success is to look beyond interviews at things like tests that more accurately mirror the job (i.e., performance tests).  But even following these guidelines can radically improve your results.

In other words, there is hope.

* I made this up.  If this is a real instrument I'm better than I thought!

Recruitageddon

 

Recently at a certain employer, let's call it the Department of Inertia (DOI), a mandate was passed down from on high to "do more recruiting."  The ensuing chaos that ensued, from HR to line manager, is worth discussing.

First, some background.  DOI had historically generally used the "post and pray" approach to recruiting.  Meaning they put job ads on their careers page and assumed (hoped?) that qualified candidates would come running.

In general there was very little formal establishment of relationships with schools or professional groups.  Branding was minimal and uniform.  Unique attractants weren't highlighted.  Advertisements were about as exciting as reruns of Gilligan's Island.  Wait, less exciting than that, Gilligan had a rockin' hat.

If a bad candidate pool resulted, the assumption was bad timing.

One day, the Executive Office passed down a mandate that all recruitments must henceforth include...ya know... recruiting.  Specifically a "recruitment plan."  (I know, earth-shattering, right?)

Here's what happened after that, not necessarily in this order:

- Program shared services staff freaked out.  What's a recruitment plan?  What does that mean?  Can we see one? Can you make one for us?

- Line supervisors freaked out.  What's this about recruiting?  Aren't we already advertising?  Who can do that for me?

- HR was deluged with calls seeking clarification. Since no clear instruction had been given, they scrambled to provide consultation.

- The one half-time recruiter in HR (budget cuts, ya know) quickly became overwhelmed.  Because most of the other analysts were transactional or focused on performance management, recruitment expertise was lacking.  (Don't even get me started about resource balancing between selection and discipline)

- Slowly, ever so slowly, a strategy coalesced to provide short- and long-term advice to programs on how to best recruit, particularly for hard-to-fill positions.  Discussions, long overdue, about things like LinkedIn, craigslist, and Dice, were had.

- Everyone breathed a sigh of relief and, much to everyone's shock, the world did not end.


Lessons?

- Recruiting never dies*.  I don't care if there's a depression and the unemployment rate is 20%.  Branding is always important.  Attractive but accurate job ads are always important.  Your careers page is always important.

- HR consultants should have recruiting as a core competency.  If your customer gets you on the phone, you should be able to speak half-intelligently about LinkedIn.

- Times change.  One year the focus may be budget cutbacks, but the next things may turn around and suddenly you're hiring again.  HR and shared services providers need to be flexible and agile, and not assume the future will look like the present.  In fact, absent a time machine, I can pretty much guarantee you it won't.

- Supervisors should have to demonstrate they've given recruiting half a second of thought.  I don't care if they think they know who they want.  Ingrain high-quality recruiting and assessing into your organizational culture and hold people accountable for ensuring it.

- Recruitment should be a core focus of any executive office.  Talent is the one thing standing between you and sustainable success.  To the extent that you care about your organization succeeding.

- Negative is stronger than positive.  Left to their own devices, hiring supervisors naturally gravitate toward avoiding poor performers rather than attracting great ones.  HR plays a key role in sustaining the focus on finding the diamonds in the rough.

- If you're going to mandate "more recruitment", give folks a little direction on what you mean, how success will be measured, and when you expect results.


Those of you in HR leadership positions know that not overreacting and keeping a cool head are key competencies for success as a manager.  Sudden changes in the direction of organizational HR practices are no exception.  It may hit a little closer to home, but that makes your calm and logical approach all that much more important.  Rise to the occasion.


* This may have made a better title for this post.  Although I almost went with recruitapocalypse, which now sounds like a dinosaur.

Good leaders leave big talent footprints

When my dad passed away a few years ago, I was proud to see how many of his previous co-workers came.  Several of them mentioned how he personally chose and mentored them and what an impact he had on them. These were people that worked for him 10, 20, 30 years ago.

At the time, I remember marveling at the impact that one person can have on others.  And how important leadership is, both personally and organizationally.


Leadership is one of the oldest topics in organizational behavior, yet remains as popular today as ever.  Why?  Many reasons:

. Leaders are important.  They have an enormous impact on those they influence or control, and this remains the case even in today's more decentralized organizational approaches.

. Leadership is hard.  To be successful, most leaders need to display an amazing array of behavior, such as flexibility, analytical strength, assertiveness, listening ability, creativity, patience, drive, and vision.

. Many leaders are bad.  I consistently see rates of around 25-50% for successful leaders.  Which isn't too surprising when you think about the previous point--how many people are capable of exhibiting and balancing all of this behavior?

. Successful leadership can be hard to measure.  Surveys are of course one way to measure, but they have inherent problems such as response rate. Productivity is a good one as well, but that can be defined in many ways and is also influenced by a wide variety of environmental factors.

. Leaders can be hard to identify, because of the aforementioned factors.  A simple interview won't cut it when it comes to assessing for a leadership role.  Even assessment centers are only so effective when it comes to predicting performance.

All this had me thinking...what happens in an organization or work unit when you have an effective leader?  Specifically, what is the impact to those around them?  Even more specifically, what has been their "talent footprint"?

If you're a good leader, what type of lasting impact do you have on the people around you?  

I have some ideas, and If you think about bosses you've had that were exemplary, chances are you'll recognize some of these indicators:

- Good leaders personally recruit and hire the best.  They have a clear vision of what competencies are required for a position and they have a clear-headed approach to seeking out, attracting, and rigorously assessing to ensure a good match.  When you ask your best performers who recruited and hired them, nine times out of ten they'll identify one of your best leaders.

- Good leaders promote their subordinates.  They support their team members, mentor them, and prepare them for promotional opportunities.  They make it clear that if you succeed, you have a career path in the organization.

- Or help subordinates promote elsewhere.  In situations where a promotional position doesn't exist at the time, good leaders help star employees seek fame and fortune elsewhere--and wish them well.  On the flipside, bad leaders see star performance transfer elsewhere for no increase in pay or responsibility.

- Good leaders get boomerangs.  Because they support and mentor their employees--even helping them get promotions elsewhere when necessary--they're more likely to be a return destination by said employees in the future.

- Good leaders are followed when they move.  They are so well thought-of that high performers are willing to follow them to new positions, units, or organizations even though it means big changes and possibly no pay increase.

- Good leaders attract internal star performers.  When you ask your high performers where they want to work, they indicate units overseen by good leaders.

So assuming you're with me on this, how do you measure these factors when hiring for a leadership position?  That's where things get a little tricky.

For internal candidates, you should be able to gather history through observation and discussion.  In fact--frankly--you should be able to quickly identify these individuals if you're paying attention.

For external candidates, there are at least a couple ways:

1) Ask them.  Yep, you can integrate this into your interview process.  Ask questions such as, "Which high performers in your organization have you personally recruited and hired?"  "Which have you personally promoted?"  "Which have returned to work for you after working somewhere else?"

2) Integrate this into your reference checking process.  Find out who the high performers are that worked under your applicant and ask them questions that get at the points above.  This has the additional benefit of potentially identifying applicants you weren't even considering!


There is at least one important caveat to all this: it's unlikely good leaders will leave this footprint in a short amount of time.  And if a good leader is entering into a bad environment or one that needs to be turned around, it will likely take years for the impact to be felt.

So like all assessment methods, this approach will work best in certain situations, like executive-level selection.  But it's something to add to your tool belt as an assessment professional.

And something to think about in your group, organization, and--for those of you that lead--your daily work life.

Are we overlooking the obvious? Use part-time positions to supercharge your recruiting

We spend a lot of time trying to come up with creative approaches to reach qualified candidates:

In-your-face advertising.

Search engine optimization.

Social media.

Gamification.

Good for us.  We should always be looking at ways of streamlining human resource matching.

But like assessment (interview questions should be high fidelity? WAH?), sometimes we leave some good stuff behind when we blaze into the future.

Exhibit A: Work schedules.  In my experience (primarily public sector), this is one of the last bastions of stubbornness for many hiring supervisors.  They'll consider more modern solutions like telecommuting and BYOD before reducing a full-time position to part-time.  There's something burned into their brains that their employees MUST be there 8-5.

And yet...and yet...this small decision may be having the single largest negative impact on their recruiting success.  Why?  Because of the large number of qualified candidates they're missing out on.

Whenever we do recruiting surveys, work/life balance is at or near the top, particularly for cognitively demanding jobs (e.g., attorney).  What does this phrase mean to people?  From the conversations I've had with folks, 99% of the time they mean fewer working hours.  And it's not just job-hunters.  In fact, at least one study found that 20% of full-time workers would prefer to be part-time.  Another study found that 47% of mothers prefer part-time work.

There's even evidence that organizations get more bang for their buck with part-time employees.  And it's not just a single study.  Here's another one.

Why?  Could be employees are thankful for the arrangement and are more engaged.  Could be they feel guilty (or protective) and work hard to keep their schedule.

Not to mention the fact that many people continue working even after they've left the physical work site (whether they should be or not), the fact that productivity and safety practices in many jobs is temporally uneven and may drop off curvilinearly with longer hours, and you really start to blur the lines of "paying for time".  In fact, absent jobs that are directly tied to hours worked (e.g., production, reception), for many jobs paying people for the number of hours they work is positively...archaic.  Scott Adams captured the insanity perfectly.

Oh, and did I mention that the same evidence regarding performance suggests employees with reduced work schedules may have higher job satisfaction?  And that it may increase your ability to retain skilled workers?  And if you can't compete on salary, compete on something people probably care about more--having enough time to live and enjoy their non-work lives.  Think about what it will do to your reputation as a destination employer!

Obviously this strategy isn't going to work in every situation.  But it's something that should always be considered when competing for talent.  And for those out there that still cling to notions of mandatory 40-hour weeks, consider opening your minds.  You, your organization, and your employees will be better for it.

Another (smaller) research update

Seems like this always happens, I post a research update and then Personnel Psychology comes out with their latest issue.

Oh well!  At least you won't have to read as much this time  And while we're at it, let's take a look at the latest Journal of Applied Social Psychology.  But first, Personnel Psych:

- The relationship between task performance and citizenship behavior: more complicated than you might think

- Propensity scoring: a statistical technique that may improve our ability to make causal inferences from quasi-experimental designs

- Can creative employees drive higher organizational performance?  Well that seems to depend on how open to risk your organization is...

- It's easier said than done, but those looking for a job should try to stay positive: it influences how quickly you're likely to land your next position

- Last but not least, a small correction to a study posted earlier on the advantage of contextualizing personality inventory items


Okay, on to June 2013 JASP, which is a special issue on prejudice:

-Trying to reduce bias in your selection process? Make sure your raters get enough sleep.

- Discrimination against people based on their weight: it starts early!  (there's another study on weight discrimination that looked at its impact on perceived social status)

- Discrimination against men perceived to be Muslim increased among Western observers when subjects were presented in traditional dress.

- Those of Black/White biracial descent were less likely to be perceived as minorities and thus less appropriate receipts of affirmative action

- Okay, I like this one in particular, and it happens to be the last one I'll describe.  Researchers showed that competency perceptions of an African-American man hired varied depending on whether it was under "affirmative action" or a "diversity initiative"--the latter resulting in higher ratings.  This was particularly true for conservative, White raters.  Just goes to show how important words are, as any political consultant will tell you.

Fin.

Research update

Okay, I'm just gonna say it.  There is an insane amount of research out there directly or indirectly related to our field.  I'll be honest, it's a little daunting thinking about reading and processing all of it.

Luckily, you have me to overly simplify it for you so you can incorrectly describe it to others.  So let's jump right in!

First up, the June issue of IJSA:

-  Fairness perceptions matter in web-based selection too!

-  The predictive validity of conscientiousness is moderated by self-enhancement

-  Speaking of conscientiousness, it's a better predictor of performance in routinized jobs than those with complex cognitive requirements

-  Psychological hardiness predicts adaptability in military leaders.  This study is awesome also for showing a negative relationship between SAT scores and West Point performance.

-  Recruiters, listen up: this study found that contingent workers converted to full-time status performed as well as referral and online hires, but not as well as those sourced internally (bookmark this one, there's a dearth of recruitment research)

- Scoring biodata: empirical keying reigns supreme over rational and quasi-rational approaches.  Quasi-rational?  Sounds like me most of the time.

- Personality and job performance: interactions are important beyond main effects


Okay, next, the May issue of JAP:

- Adaptive decision making in military leaders: both the brain and the mind are important (okay you monists out there, chill out)

- Are you a MANOVA (wo)man?  Then read this.

- Proactive leaders set more challenging goals and have higher sales performance--assuming they have the trust of their subordinates.


Next, the April issue of JOB.  Just one study, albeit interesting, in which female evaluators were less likely to recommend hiring or promoting Asian (versus White) applicants into jobs requiring social skills


Speaking of JOB, how about the May issue?

- Have you been calling for more research on calling?  Your wish is granted.

- Curious about the concept of curiosity? (okay, I'll stop)  Looks like it can be predictive of job performance above and beyond traditional cognitive and non-cognitive predictors.  I'm gonna assume it varies with job, but I think the authors are probably right that it will increase in importance over time.

- When selecting for teams that may experience crisis situations (e.g., nuclear power plant crews), consider homogeneity--not mean levels--of positive affect.

- Moderate levels of supervisory structure combined with high levels of consideration lead to lowest levels of CWB's

- The dark side of OCBs

- The light side of OCBs

- Dark side and light side working together in a picture that has nothing to do with selection


Okay, onwards and upwards: one from the April issue of JASP, on generalized self-efficacy, work-related self-efficacy, and job-related outcomes


And one from the May issue:  Sensation seeking and need for structure predict military field exercise performance


How about a little EI research?  Haven't had a lot of that lately.  Here's a piece from the May issue of Journal of Management that found emotion management ability to be a valid predictor of job performance.


Let's look in the May Psychological Science:

- Where we find a fascinating study that demonstrates assessment of profound cognitive abilities at a young age predicts outstanding contributions in many adults

- Another interesting study of how an individual responds to daily stressors (which I would posit are differentially found in various jobs) has long-term consequences for their mental health

- High schoolers with high math and verbal skills are less likely to choose STEM careers than those with math skills but moderate verbal skills

- Okay, check this one out: mindfulness training improved GRE reading comp scores as well as working memory.  Implications for personnel assessment??


Shifting to the May Psychological Bulletin:

- Rorschach, anyone?

- Believing that individuals are malleable, rather than fixed ("implicit theory") predicts goal achievement.  Implications for leaders?


Last but not least, let's hook up with our friends at PARE and see what they're up to:

- Into Excel?  Check out this article on forest plots

- Or maybe factor analysis is your thing.


That's all folks!  Hope your brain is spinning like mine!

Test validity: A fairy tale

Once upon a time there was a field called Industrial and Organizational Psychology.  Its researchers and practitioners dealt with a myriad of magical issues ranging from individual differences and behavior to organizational structures.

Within this field, there was a specialty called Personnel Psychology.  It dealt with narrower--but no less mysterious--issues such as defining and designing jobs and, most relevant for us, finding and hiring the right people.  The I/O psychologists and HR professionals that quested for these answers often found themselves on dangerous missions like battling Monsters of Doubt (i.e., first-line supervisors).

These adventurers had two main weapons at their disposal when fighting these monsters: the Carrot of Truth and the Stick of Pain.

When invoked, the Carrot of Truth, fashioned deep in the Mines of Correlation, caused monsters to realize that hiring the right people was the right thing to do for their realm.  It increased productivity and morale, customer satisfaction, and organizational flexibility.  It also allowed supervisors to spend more time leading and less time dealing with gremlins (i.e., employees with performance problems).

The Stick of Pain used an opposing form of magic but was sometimes equally effective.  It attempted to slay the monsters using a peculiar power called The Law.  The Law frightened monsters because it meant they could experience emotional pain and suffering, and--more importantly--fewer bags of gold.

For a long time, Personnel adventurers used both of these weapons to slay all kinds of monsters, on high mountains and in dungeons.  But over time, the adventurers discovered something: the Stick of Pain was becoming less and less effective.

It wasn't that the Stick was powerless.  It's just that its magic didn't seem to frighten the monsters as much.  The monsters saw their gold piling up and didn't feel the sting of suffering as they once had.  And they started developing an addiction to carrots all on their own.

So there came a day where the adventurers and the monsters met on the battlefield and came to an agreement.  No longer would the adventurer wield the Stick of Pain.  And in return, the monsters pledged to respect the Carrot of Truth.  They forged an eternal partnership and lived happily ever after.

The End.


Okay, so I've taken a little artistic license with my blog post today.  But hopefully you see where I'm going.

Back in the old days (ya know, like the 80s), employers were faced with a foreboding world of testing, with the Civil Rights Act and cases like Griggs vs. Duke Power looming large over their assessment programs.  I/O psychologists were brought in to help organizations navigate the complicated world of employment testing, which required an appreciation of statistics and the law alike.  Large awards and settlements brought C-level attention, and regulatory agencies like the EEOC and DOJ served in ongoing oversight roles, requiring employers to clean up their act with procedural requirements that could be burdensome.

Nowadays, I/O psychologists are as likely to be valued for their ability to crunch "big data" to detect employee behavior trends as their ability to conduct thorough job analyses (not that the two are mutually exclusive).  Lawsuits regarding testing are infrequent compared to issues like wages and hours, harassment, and terminations.  The selection cases that do come up are as likely to involve disabilities as adverse impact due to cognitive loading.

Sure, we have the occasional big case that gets attention.  But the bottom line is over the years the "stick" has become much less effective as an argument for sound assessment than the "carrot."

Smart employers like Google have started crunching the numbers and realized the true business value of defining the right competencies for jobs.  They're doing so not because they're afraid of litigation, but because they see more clearly the direct line between best practices in selection that we've been preaching for years--i.e., focusing on valid assessment results--and the bottom line.


So where does that leave the stick (i.e., fear of lawsuits)?  Is it time to put it away along with phrenology and T&Es (woops, that slipped in)?

I don't think so.  Organizations will always be subject to legal scrutiny when their selection processes have adverse impact and the right person talks to the right attorney.  Personnel psychologists and HR professionals should always have a healthy respect for the legal climate we operate in, and not forget that "job related and consistent with business necessity" isn't fictional gibberish.

But what it does mean is that because organizations are paying attention to their assessments, they are more likely to yield valid results and be more free of illegal bias.  That means management's quest and the selection professional's quest are more likely to converge, with a lot more cooperation.

And hopefully a lot fewer monsters.

Presentations from 2013 ATP Conference

Many of the presentations from the 2013 Association of Test Publishers conference are now available and I strongly recommend checking them out.  Strong themes include gamification, global assessment, item development, and test security, just to name a few.

I found the one from HumRRO on creating a virtual role-play fascinating.  If you haven't already seen it, I strongly recommend you check out their demo at www.humrro.org/simdemo.html.  The username and password are both humrrodemo.  A small taste of what's possible in assessment!

Hat tip.

Research update addendum

Okay, I know I just did a research update but but I had a couple stragglers, including a pretty important one, namely the Spring 2013 Personnel Psychology, which is free right now!

-  First, an important piece by Bobko and Roth updating d values for Black-White differences on selection measures.  Their updated analysis indicates measures such as biodata and assessment centers may have d values as large as paper-and-pencil tests of cognitive ability.  Personality measures still benefit from small differences.  They include a helpful table that breaks down the values by construct, and they also include a list of factors that can impact d, such as job complexity and range restriction.

- Second, a fascinating study of aberrant personality tendencies and their impact on career outcomes conceptualized using the Five-Factor Model and measured using the NEO PI-R.  More evidence that "dark side" personality traits are an important consideration in predicting career trajectories.


The other that just came out is the February 2013 Journal of Applied Social Psychology:

- First, a look at moderators of the relationship between employee weight and job-related outcomes.

- Second, a study that I think has implications for selection: looking at circumstances under which competitors copy their opponents choices.  I've observed over and over again that when an employee gets a competing offer, suddenly their attractiveness increases.  Perhaps not the same phenomenon, but worth exploring.

-  Next, another study of discrimination, this time age discrimination in within- and between-career job changes.  Results indicated discrimination against older workers was particularly pronounced when older applicants were making between- rather than within-career changes.

- Okay, I'm sensing a theme to this post.  In this study, the authors looked at how the wording of occupational descriptions activates gender stereotypes.

- Finally, something not about discrimination: the authors of this (small N) study found that a perceived aspect of emotional intelligence predicted perceived negotiation success beyond traditional personality traits.

And on that note...until next time!

Research update

It's that time, February research update--and there are some really fascinating studies out there:

First the March 2013 issue of IJSA:

- Procedural fairness is an important aspect of organizational attraction, especially for rejected applicants.

- Does personality type impact perceptions of the selection process?  According to this study and their measures (Bohemian is a personality type?), sometimes.

- Do you use SEM to generate a SED and ultimately score bands? Then you will definitely want to read this piece, which suggests the traditional method is flawed.

- Emotional intelligence continues to be a hot topic.  In this study the authors describe the development of a SJT-based measure.

- Staying on the topic of personality, this study adds to the debate over the utility of constructs vs. facets in predicting performance--this time with conscientiousness predicting police officer performance.

- P-O fit is often used to explain why people apply for certain jobs.  In this study researchers found that values that applicants find attractive are particularly important (as opposed to neutral or aversive aspects).

- I've written about automated reference check systems before.  In fact I've written about the company that the first author of this study is from (SkillSurvey).  What is partially a demonstration of the value of their product also adds to our knowledge of reference checks by finding no demographic differences and the ability to predict involuntary (but not voluntary) turnover.

- How big of a deal is it when it is discovered that an applicant "embellished" on their resume?  Depends on how much you already liked them.

- In this study of applicants in Iran, the authors found that the usefulness of a web-based application system was more important than how easy it was to use.  So basically looks are important but not sufficient.


Next, the January issue of the Journal of Applied Psychology: 

- Construct validity has always been a sticky wicket.  The authors of this study take an interesting approach, using video-based testing to illustrate a method for supporting construct validity.

- This study explores the variance in assessment center ratings stemming from various sources, and the authors argue that existing literature has masked this variance.

-The authors of this study show how the application of a particular bootstrap method applied to meta-analyses increases accuracy.

- Speaking of meta-analysis, the authors of this study demonstrate that the differences between coefficient alpha and composite reliability in actual data sets is quite small.


On to the January issue of the Journal of Applied Social Psychology, which as of the time of this post was available for free!

- Similar to findings in other areas, the impact of negative publicity is stronger than positive publicity on organizational attraction, according to the authors of this study.  The authors also found in these cases, organizations would do well to provide detailed recruiting ads to mitigate the impact.

- Age discrimination seems to be increasing in popularity as a research topic (which makes sense from a demographic perspective).  In this study, the authors found raters discriminated against older and younger applicants, with the former being the least likely to be hired.

- Here's an interesting one: in this study, people of color perceived White women to be, among other things, ditsy, shallow, privileged, and appearance focused.  Stereotypes know no color boundaries.


Let's take a look at the February 2013 issue of the Journal of Personality and Social Psychology:

- First, a fascinating look at how generalizable the five-factor model of personality is across cultures.  Specifically, the authors looked at a largely illiterate, indigenous culture in Bolivia.  They failed to find support for the widely used model, and suggest that the structure of personality may differ across societies.

- Second, a study of sex and gender differences that points out the importance of considering the construct of gender (as opposed to the binary concept of sex) for a variety of behavioral and psychological indicators, including big five personality.


Let's move now to the February 2013 issue of Psychological Science, where:

- In this study, the authors found that age differences in cognitive functioning within cohorts were often as large as those between cohorts, which calls into question the cohort explanation for age differences.

- In a study that I think has implications for personnel assessment, the authors found that raters of M.B.A. applications that had already given several high recommendations were less likely to do so later in the day.  Another bias to watch out for!


Last but not least, there was an article recently in the New York Times about how important referrals are for some organizations and the increased importance this places on social networking sites.  There was a reference to a study done in August of 2012 on source of hire that I hadn't seen before, which found that not only were referrals more likely to be hired, they had longer tenure--which is consistent with other research I've seen.  Unfortunately, no word about performance differences.

That's all for now!

Of Wal-Mart, veterans, and the nature of jobs

I think many of us were somewhat impressed with Wal-Mart's announcement that beginning on Memorial Day, they will offer a job to any honorably discharged veteran that applies within a year of ending active duty.

There's no question that the unemployment rate for U.S. vets is too high--consistently 20-30% above the rate for non-vets.

And it's refreshing to see the private sector step up after years of veteran hiring initiatives coming primarily from the government (which is not the driver of employment, at least not recently).

So kudos to them, it continues their public commitment to hiring veterans, and it's a bold gesture (although I wish the one-year time limitation was relaxed).


But what I found even more interesting than the announcement was the reaction from the veteran support community.  For example, Yana Walton from the Retail Action Project said (from the article linked above), "Workers are going to make sure that the kind of jobs that they’ve announced for veterans are the kind of jobs that workers need.  That means enough hours and living wages and jobs with benefits."

Other workers' rights groups have pointed out that these jobs are likely to include low wages and limited benefits.

Also interesting were the competing stories for why Wal-Mart likes veterans.  Wal-Mart's U.S. CEO, Bill Simon, said veterans have a record of performing under pressure, are quick leaders, and team players.  It's that last claim that is reflected in a response from Dr. Nelson Lichtenstein, a labor historian at the University of California, Santa Barbara, who said, "'They like military people because they have a sense of hierarchy and a commitment to the organization they are in.  And that's important to Wal-Mart.''

Frankly I find both comments to be slightly insulting to veterans, treating them as if they are a homogenous group.  I also find it insulting for anyone to assume that veterans are going to leap to embrace any job, regardless of the duties, pay, and benefits.

Some may see this as looking a gift horse in the mouth.  I look at it as supporting the dignity of workers and those seeking employment.

One of the most interesting debates that has occurred over the last several years of high unemployment has been about the nature of jobs that people want.  Certainly having a job and an income is better than nothing (at least in most cases), but many unemployed aren't interested in any job, they want a job that holds some promise--of good working conditions, of stability, and of promise of future opportunities.

Sadly, that may not be the types of jobs that are being created, at least not in the retail sector.  And it begs a larger conversation about education and skill mismatch, as well as how organizations define qualifications.

I also find this fascinating from an assessment standpoint.  To promise jobs to a single group of individuals without regard for assessing whether the individual applicants possess the qualifications the employer is seeking for a particular position is, well, frankly stupid from an organizational effectiveness standpoint.  This is either a leap of faith on the part of Wal-Mart, or (more likely) a calculated risk given their high turnover rate and the types of jobs veterans are likely to get.


It was with these thoughts rolling around in my brain that I watched my 5-year old's teacher open the door to her kindergarten students the other day.  I wondered what it was like for her day after day--the ups (children learning), the downs (noise)--and what attracted and kept her in the job.  I would guess that most teachers that stay teachers do so because they receive a significant amount of satisfaction from seeing students achieve and change in positive ways.  It's not an easy job, and they don't make a ton of money, so something must be keeping them there.

This made me think about why people apply for jobs at all.  Sure, for some people a job is a job--it almost doesn't matter what it is.  But I think this is probably the minority of job seekers, and it may be decreasing in prevalence.  And the historically (and I would argue misleading) high rate of job satisfaction may be losing its strength as an argument as people ask better questions like whether you would make the same career decision. 


So if more and more people are looking not only for a job, but a meaningful, rewarding one, what can we as assessment professionals do?

For starters, we can do more thinking about how to accurately describe jobs.  Regular readers know I'm a big fan of realistic job previews and that I think organizations generally do a terrible job at letting applicants know what they're getting themselves into.

Second, we can do a better job at assessing people's motivations for applying.  We've all seen the "why are you interested in this job?" question that seems entirely predictable and pointless but often yields surprisingly honest results.  If we proceed on the assumption that job performance and retention is driven at least in part by the match between a person's interests and motivations and what the job offers, we do a pretty sad job of measuring this.  Most of our measures deal with job-related knowledge or skills.  Don't get me wrong--this is really important.  But (in addition to personality) it leaves out a huge factor, namely the motivation for applying.

We should be providing more information, asking more questions, digging deeper, investigating new ways of using technology, and treating applicant motivation with a renewed sense of seriousness.

There may have been a time when measuring motivation to apply was considered secondary or unimportant, but that time has passed.