Where there's a will, there's a way: OPM shows how to do UIT the right way

On August 10 and 12, PTC-NC was privileged to have Dr. Patrick Sharpe from the U.S. Office of Personnel Management (OPM) deliver a presentation about USA Hire, part of a suite of online platforms that has allowed OPM to revolutionize the way they deliver assessment services to their diverse customers.

The idea to ask Dr. Sharpe to present occurred to me when I read an article in the Washington Post on April 2 about USA Hire titled, "For federal-worker hopefuls, the civil service exam is making a comeback".   It provides an overview of what OPM has managed to accomplish with its partner, PDRI, in the area of unproctored internet testing (UIT).  Start there if you want to learn more because it includes some examples of the items--although significantly absent is an example of the excellent video avatar-based assessment used for things like situational judgment.

Dr. Sharpe did an excellent job painting the picture for the audience of how much work was involved in the project, and how important things like stakeholder communication and contract management were to ensuring the project was a success.  He then showed us a demo version of USA Hire, where he lead us through what it looks like from the applicant's perspective as they proceed through a series of competency-based assessments.  The item formats range from the traditional (e.g., reading comprehension multiple-choice) to the modern (avatar-based SJT) to the groundbreaking--at least for the public sector (forced-choice non-cognitive assessment).

Here are some of the key points I took away:

- The technology is just a part of successfully putting an UIT program together, you have to step back and look first at what you're trying to accomplish.  For example, are you interested in whole person assessment (as OPM is) or simply focusing on certain KSAs?

- USA Hire is the culmination of years of research and analysis, and traces its history back 20-30 years within the federal government.   Translation : don't jump into UIT without careful planning.

- Start with the basics when delivering UIT: make sure the customer has a solid job analysis foundation before jumping to the assessment platform

- Getting a larger, more influential, customer successfully implemented can cause others to jump on board

- Realize that, particularly in a decentralized testing environment, you may still end up with a hybrid of different testing approaches following the roll-out of UIT, and this includes T&Es.  But the best way to move the practice is to show what success looks like.

- Consider carefully whether you want to build, buy, or lease the technology.  There are benefits and drawbacks to each.

- Starting with a pilot can be a great way to test the system (no pun intended), and also demonstrate the potential to stakeholders.

- Collaboration between assessment professionals, HR specialists, and vendors is critical.

- Don't underestimate the importance of change management.  Fears (e.g., about losing control) come easily and have to be addressed head-on.

- Organizational and system readiness is very important.  Part of the reason this effort was successful is because hiring organizations were fed up with the extremely low utility (and perception) of point-based T&Es.

For someone passionate about assessment and technology, the presentation was educational and motivational.  I walked away, as did others, with a new-found optimism for what sufficient will, resources, and tenacity can accomplish.  It's seductive to focus on what can't be done in the public sector, so to hear and see what can be done reemphasizes the importance of leadership--both in HR and at the top of the organization.

Unproctored internet testing has been talked about for so long, but to see it in action, in a research-based way in the public sector, is truly inspirational.  Truly where there is a will, there is a way.

The secret to successful organizations: Let HR drive hiring

Why do organizations make bad hires?  Because they rely too much on supervisors to do the hiring.

Imagine this scenario.  You take your car into the shop because it's making a strange sound.  The mechanic fixes it, you pay, and you're about to leave.  But on your way out, the mechanic stops you.  She* notices that you're frustrated with your iPad because it's being sluggish.  She offers to fix it for you, reasoning that cars and iPads can't be all that different.

Would you take the mechanic up on her offer?

Of course not.  But handing over important decisions--arguably the most important decisions--to the wrong people is something organizations do every day.

Auto mechanics are trained to deal with a specific situation, and to do it successfully: fix cars.  Their education and experience prepares them to do so.

Similarly, supervisors are trained to do primarily one thing: supervise the day-to-day work.  They're usually promoted because they excelled at the line level (i.e., they understand the work) and they show aptitude for leadership (hopefully).  They are not, generally speaking, schooled or experienced with the professional side of personnel selection.

Am I suggesting supervisors NOT be involved?  Absolutely not.  Am I suggesting that all HR shops are staffed with experts in personnel assessment and measurement?  Nope.

What I am suggesting is organizations get serious about this issue and stop treating supervisors as if they are people measurement professionals.

In the modern workplace with information overload and time at a premium, it's tough enough to ensure that hiring and promotion decisions get made in a thoughtful fashion.  The problem is exacerbated by the fact that most supervisors overestimate their hiring ability.  After all, how hard can it be to interview someone?

Not hard, if you don't care about getting it right.  But getting it right requires careful thought and planning; hiring right is not something done at the last minute without regard to competencies that drive success in the specific position.  It's not something that looks exactly the same time.  It's tailored to the position, the culture, and the particular needs at the time.

Great hires happen systematically for one reason: a tight partnership between line supervisors and talented HR consultants.  The supervisor knows the job. They often know best what competencies are needed to perform the job successfully (although a good internal HR consultant will have a pretty good idea).  HR professionals are trained in the professional side of selection, recognizing the pitfalls and identifying those measures most likely to predict success.

"But our HR shop doesn't know what it's doing.  They don't have the expertise!" some might say.

Perhaps.  If so, you've just identified your second strategic problem.  And it should be fixed.  That's like having people in your budget office that are bad at math.

But consider this: you may be surprised to learn that many hiring supervisors welcome being removed as the primary driver of hiring.  Good supervisors recognize that this isn't their greatest strength, and they will be happy to have HR assist them in identifying the most qualified applicants.  Particularly in this age of online questionnaires and massive candidate lists for many jobs.

So who is responsible for ensuring this supervisor-HR partnership happens systematically and is built into the organizational culture?  The leaders at the top.  Director.  CEOs.  They're the ones minding the store, and they're ultimately responsible for organizational success.  They should be instituting policies, procedures, and cultural norms that emphasize how important and critical hiring is--so critical that it requires a team approach to get it right.

Anything else is simply not taking the success of the enterprise seriously.


* Did I throw you with a female mechanic?  Now might be a good time to take the implicit association test for gender and careers.

Mega research update

I hope you like research, because there's a lot of it coming your way...and many are free as of this posting!

Without further ado...


Let's start with the Journal of Applied Psychology, January 2015 issue:

- We see a lot of research involving large candidate groups, but much less for individuals.  In this meta-analysis of individual assessments, the authors found support for their usefulness, but it varied significantly across studies.  Highest validity was found for managerial jobs and assessments that included a cognitive ability test.

- Being in the wrong job can be frustrating for both the employee and the employers.  In this study, the authors show a relationship between poor vocational fit and counterproductive work behaviors (CWBs).

- Speaking of CWB, there may be more of them going on than you would think based on the assessment literature...

- And even more on CWB!  These authors found support for both self- and acquaintance-reported personality ratings, specifically conscientiousness and agreeableness, in predicting "workplace deviance".

- Unfortunately, gender bias still exists in selection.  In this meta-analysis, the authors found this to be particularly the case in male-dominated jobs.  On a positive note, they do suggest ways of mitigating this: provide clear evidence of the competence of applicants, encourage careful decision making, and use experienced raters.

- The over-/under-prediction of cognitive ability tests debate for different ethnic groups continues.  In this study, the authors find support of overprediction for African Americans, suggesting the tests are not predictively biased.



Next, the March issue of J.A.P.:

- More support for the predictive validity of emotional intelligence, but more importantly, how the concept overlaps with other constructs such as the Big 5 and self-efficacy.

- All situational judgment tests (SJTs) are not equal, and according to these authors in a large number of instances the context that is presumably important?  Not so much.

- Speaking of SJTs, these researchers suggest that putting the "situational" back in SJTs--i.e., assessing how the situation is analyzed rather than the response options--is a useful method.

- A fascinating update of effect size benchmarks that can be used for a variety of purposes.

- Trying to predict safety-related behavior?  This research suggests that personality traits, particularly agreeableness, can usefully predict this behavior.


Moving on to the March issue of IJSA (free right now!):

- Some guidelines on preparing norms for personality inventories.

- Evidence that different cultures have different procedural justice perceptions of different selection mechanisms

- Some important findings on the equivalence and stability of job performance ratings over time

- Development of a new measure of subjective career success

- More evidence that both technical knowledge and prosocial knowledge are important factors in predicting medical student clinical performance

- This study found that CWBs are under-reported and organizational commitment increases the likelihood that peers will report them

- Evidence that forced-choice and Likert-type scales used in personality inventories have similar measurement properties


On to the Spring issue of Personnel Psych (also free right now!):

- This meta-analysis on narcissism showed that it is related to leadership emergence (through extraversion) and leadership effectiveness in a curvilinear fashion.

- More evidence of the importance of political skill--particularly the aspects of networking ability, interpersonal influence, and apparent sincerity--in predicting a range of important outcomes, including task performance beyond GMA and the Big 5.  It would be interesting to see how this is related to emotional intelligence (yes this is a foreshadowing).


Turning to the March issue of Psych Bulletin:

- More on narcissism: this time, researchers found that men consistently report higher levels of narcissism compared to women, which is interesting when taken in combination with the study above.


In the December issue of Industrial and Organizational Psychology:

- The first focal article calls out researchers for using incorrect assumptions about criterion reliabilities, thus impacting criterion validity values.  They make suggestions for how to improve meta-analyses moving forward.

- The second makes the important argument that utility analyses should consider measures of well-being when determining the effectiveness of interventions (such as an employment test).


Finally, in the January issue of JOB (also free right now):

- a proposal for improving the calculation and reporting of Cronbach's alpha

- a fascinating study showing that high conscientiousness may hinder performance during stressful situations

- in support of EI, this study found a link between emotion recognition ability and income (interestingly through political skill and interpersonal facilitation...remember the earlier study on political skill?).


That's all for now!

"Power posing" impacts interview performance

Many people have watched Amy Cuddy's riveting TED talk on how body language impacts thought processes.  Specifically, over 23.5 million people as of today have watched that particular video, millions more through YouTube and other outlets.

In the presentation she specifically talks about how "power posing"--standing in a way that denotes confidence (hands on hips)--impacts testosterone and cortisol levels and ultimately behavior.

In an upcoming issue of the Journal of Applied Psychology, Cuddy and her colleagues describe the results of an experiment where university students were told to adopt either a high-power (shown on left) or low-power (shown on right) posture:

They were instructed to stand in this pose for 5-6 minutes while preparing for a job interview speech.  They then delivered said speech, which was videotaped, to two evaluators/interviewers.  These evaluators subsequently rated the participants on (a) performance, and (b) hirability.  They also judged the participants on their verbal and nonverbal behavior.

Results (N=61)?  Those who prepared for their speech while in a high-power pose were rated significantly higher (p<.01) on both factors than those who prepared in a low-power pose.  And the results could not be explained by nonverbal behavior in the interview itself.  Per the authors, "Compared to low-power posers, high-power posers appeared to better maintain their composure, project more confidence, and present more captivating and enthusiastic speeches, which led to higher overall performance evaluations."

This suggest, in the words of the authors, "By nonverbally manipulating their own sense of power, the high-power posers were effectively imbued with the psychological and physiological advantages typically associated with high power, despite their low-power position relative to the evaluators."

So what does this mean?  I believe it has several important implications.  First, it indicates a potential source of "error", akin to test anxiety, that may impact assessment performance.  Second, it suggests a potential avenue that those plagued by test anxiety may pursue to increase their chances of success.  This includes those who experience stereotype threat.  Third, it may help explain why even a "perfect" measurement of job-related KSAs does not yield a perfect correlation with performance (I'm thinking of similar confounding effects like mood or physical appearance).

You can read what appears to be the submitted version here.  I recommend reading it as well as the excellent TED talk if this subject interests you.

Federal civil service reform is HR reform

Texas.  Georgia.  Washington.  Florida.  California.

What do these states have in common?  They are some of the states that have undertaken civil service reform in an attempt to modernize and streamline all those laws and rules associated with human resources.

The U.S. federal government is no stranger to reform efforts either.  In the latest issue of Government Executive magazine, there is an interesting article about the challenges associated with the current state of affairs in the U.S. federal civil service--the largest employer in the world.  Things like excessively long time-to-hire, outdated hiring rules, and onerous discipline systems.  There are efforts afoot (again) to fix this.

They could very easily have been talking about state or local civil service systems.  Or--in many places--HR in general.

Here are some selected quotes.  See if any of them sound familiar...

"...'if you don’t view your HR specialist as a consultant prior to posting the vacancy announcements, then you are going to get people who are not qualified for your job.'"

"USAJobs, the government’s online warehouse of job vacancies, is still difficult to navigate and lacks sophisticated search capabilities to help applicants find positions that meet their interests and qualifications."

“'What we’ve seen across managers,...they are almost sourcing a unicorn...They could be looking at too technical or specialized of a skill set, or they are looking for too many competencies or experiences creating this kind of applicant that we can’t actually find.'”

"When HR and program supervisors work closely throughout the hiring process, communicating at every stage, it increases the likelihood that the system works the way it should: fairly, as expeditiously as possible, and yielding the most qualified candidates for the job."

It's worth a read, and I bet many of you will find much that resonates.  None of the principles behind reform are particularly revolutionary, but for various reasons many organizations have difficulty getting it right.  There are no shortcuts to outstanding HR systems, and only those organizations that recognize the strategic and competitive value inherent in a talented, engaged workforce will put the required resources into ensuring that it's done right.

Next time: research update

2014 Research of the Year (+ research update)

Happy New Year!  As I've done in previous years, I present below the research articles I ran across in this area that I think were the most impactful and/or important of 2014.  But first, let's catch up on two issues:

First, the Winter issue of Personnel Psychology:

- Situational judgment tests have been shown to be useful for measuring interpersonal skills, but beware: levels of "angry hostility" moderate that relationship.  (Is there a happy hostility?)

- When hiring leaders, should you look for those that have a busy home life, or be wary of them?  In this fascinating study, the authors found that leader family-to-work conflict negatively impacts followers in that it can increase their burnout. However, family-to-work enrichment increased follower engagement through leader engagement.  So the answer is, as usual, not simple: home/family life can be a good thing for followers if it makes the leader more engaged; but if the home/family life is increasing burnout, the leader may pass that along to others.  So it would seem it all depends on how the individual is handling their life outside of work!

Let's look at the November issue of the Journal of Applied Psychology:

- Are men or women perceived as better leaders?  According to this meta-analysis, it depends on how you ask the question.  If you limit the question to other-ratings, women are rated significantly higher.  But if you look at self-ratings, men rate themselves significantly higher.  Which leads to the next question: is it a biological perception or a gender perception, and if the latter, what traits are the most important?

- An intriguing study of how applicant confidence interacts with and can be altered by the recruitment experience, in this case among recruits for the U.S. military.

- Next, a study of employment and job search efficacy.  Not surprisingly, within-person frequency of job search behavior correlated with job offers; interestingly, the relationship between perceived job search progress and efficacy beliefs were moderated by beliefs of internal attribution.

- Last but not least, more evidence of the importance of defining the criteria when predicting job performance.  In this meta-analysis, the researchers found more support for personality traits out-predicting cognitive ability in predicting counterproductive work behavior, that the two predictors are approximately equal in predicting organizational citizenship behaviors, and that cognitive ability outperforms personality when predicting task and overall performance.  So do you want high task performance, OCBs, or do you want to avoid CWB?  :)  (of course the situation is even more complicated depending on whether you're looking at individual, team, leader performance, over what period of time, etc.)


Okay, on to the awards!  Without further ado, here are my nominations for Research of the Year for 2014:

1) Important advancements in our understanding of weight-based discrimination at work: Vanhove & Gordon.

2) A study of applicants posting faux pas on their social networking sites: Roulin.

3) Two important looks at assessments delivered remotely via mobile devices: Arthur, Doverspike, Munoz, Taylor, & Carr, and Morelli, Mahan, & Illingworth.

4) Two fascinating looks at personality at work: Judge, Simon, Hurst, & Kelley; and Wille & De Fruyt

5) An excellent study of how effective staffing and training practices impact firm-level flexibility and adaptability: Kim & Ployhart.

6) An important study of the movement of impactful I/O researchers to business schools: Aguinis, Bradley, & Brodersen.

7) The relationship between conscientiousness and job performance is more accurately described as curvilinear: Carter, Dalal, Boyce, O'Connell, Kung & Delgado

Finally, honorable mention to two great developments in 2014:  the change of some publishers to making access to articles more affordable, and the announcement of an additional journal, the Journal of Personnel Assessment and Decisions.


I'm continually amazed at the quality of thought and research in our area and the passion and practicality you exhibit.  Here's to an amazing 2014 and more in 2015!