Mobile assessment comes of age + research update

The idea of administering employment tests on mobile devices is not new.  But serious research into it is in its infancy.  This is to be expected for at least two reasons: (1) historically it has taken a while with new technologies to have enough data to analyze (although this is changing), and (2) it takes a while for researchers to get through the arcaneness of publishing (this, to my knowledge, isn't changing, but please prove me wrong).

Readers interested in the topic have benefited from articles elsewhere, but we're finally at a point where good research is being published on this topic.  Case in point: the June issue of the International Journal of Selection and Assessment.

The first article on this topic in this issue, by Arthur, Doverspike, Munoz, Taylor, & Carr, studied data from over 3.5 million applicants who completed unproctored internet-based tests (UIT) over a 14-month period.  And while the percentage that completed them on mobile devices was small (2%), it still yielded data on nearly 70,000 applicants.

Results?  Some in line with research you may have seen before, but some may surprise you:

- Mobile devices were (slightly) more likely to be used by women, African-Americans and Hispanics, and younger applicants.  (Think about that for a minute!)

- Scores on a personality inventory were similar across platforms.

- Scores on a cognitive ability test were lower for those using mobile devices.  Without access to the entire article, I can only speculate on proffered reasons, but it's interesting to think about whether this is a reflection of the applicants or the platform.

- Tests of measurement invariance found equivalence across platforms (which basically means the same thing(s) appeared to be measured).

So overall, in terms of using UITs, I think this is promising in terms of including a mobile component.


The next article, by Morelli, Mahan, and Illingworth, also looked at measurement variance of mobile versus non-mobile (i.e., PC-delivered) internet-based tests, with respect to four types of assessment: cognitive ability, biodata, a multimedia work simulation, and a text-based situational judgment test.  Data was gathered from nearly 600,000 test-takers in the hospitality industry who were applying for maintenance and customer-facing jobs in 2011 and 2012 (note the different job types).  Nearly 25,000 of these applicants took the assessment on mobile devices.

Results?  The two types of administrations appeared be equivalent in terms of what they were measuring.  However, interestingly, mobile test-takers did worse on the SJT portion.  The authors reasonably hypothesize this may be due to the nature of the SJT and the amount of attention it may have required compared to the other test types.  (btw this article appears to be based on Morelli's dissertation, which can be found here--it's a treasure trove of information on the topic)

Again, overall these are promising results for establishing the measurement equivalence of mobile assessments.  What does this all mean?  It suggests that unproctored tests delivered using mobile devices are measuring the same things as tests delivered using more traditional internet-based methods.  It also looks like fakability or inflation may be a non-issue (compared to traditional UIT).  This preliminary research means researchers and practitioners should be more confident that mobile assessments can be used meaningfully.

I agree with others that this is only the beginning.  In our mobile and app-reliant world, we're only scratching the surface not only in terms of research but in terms of what can be done to measure competencies in new--and frankly more interesting--ways.  Not to mention all the interesting (and important) associated research questions:

- Do natively developed apps differ in measurement properties--and potential--compared to more traditional assessments simply delivered over mobile?

- How does assessment delivery model interact with job type?  (e.g., may be more appropriate for some, may be better than traditional methods for others)

- What competencies should test developers be looking for when hiring?  (e.g., should they be hiring game developers?)

- What do popular apps, such as Facebook (usage) and Candy Crush (score), measure--if anything?

- Oh, and how about: does mobile assessment impact criterion-related validity?


Lest you think I've forgotten the rest of this excellent issue...

- Maclver, et al. introduce the concept of user validity, which uses test-taker perceptions to focus on ways we can improve assessments, score interpretation, and the provision of test feedback.

- Bing, et al. provide more evidence that contextualizing personality inventory items (i.e., wording the items so they more closely match the purpose/situation) improves the prediction of job performance--beyond noncontexual measures of the same traits.

- On the other hand, Holtrop, et al. take things a step further and look at different methods of contextualization.  Interestingly, this study of 139 pharmacy assistants found a decrease in validity compared to a "generic" personality inventory!

- This study by Ioannis Nikolaou in Greece of social networking websites (SNWs) that found job seekers still using job boards more than SNWs, that SNWs may be particularly effective for passive candidates (!), and that HR professionals found LinkedIn to be more effective than Facebook.

- An important study of applicant withdrawal behavior by Brock Baskin, et al., that found withdrawal tied primarily to obstructions (e.g., distance to test facility) rather than minority differences in perception.

- A study of Black-White differences on a measure of emotional intelligence by Whitman, et al., that found (N=334) Blacks had higher face validity perceptions of the measure, but Whites performed significantly better.

- Last, a study by Vecchione that compared the fakability of implicit personality measures to explicit personality measures.  Implicit measures are somewhat "hidden" in that they measure attitudes or characteristics using perceptual speed or other tools to discover your typical thought patterns; you may be familiar with project implicit, which has gotten some media coverage.  Explicit measures are, as the name implies, more obvious items--in this case, about personality aspects.  In this study of a relatively small number of security guards and semiskilled workers, the researchers found the implicit measure to be superior in terms of fakability resistance.  (I wonder how the test-takers felt?)


That's it for this excellent issue of IJSA, but in the last few months we also got some more great research care of the March issue of the Journal of Applied Psychology:

- An important (but small N) within-subjects study by Judge, et al. of the stability of personality at work.  They found that while traits exhibited stability across time, there were also deviations that were explained by work experiences such as interpersonal conflict, which has interesting implications for work behavior as well as measurement.  In addition, the authors found that individuals high in neuroticism exhibited more variation in traits over time compared to those who were more emotionally stable.  You can find an in press version here; it's worth a read, particularly the section beginning on page 47 on practical implications.

- Smith-Crowe, et al. present a set of guidelines for researchers and practitioners looking to draw conclusions from tests of interrater agreement that may assume conditions that are rarely true.

- Another interesting one: Wille & De Fruyt investigate the reciprocal relationship between personality and work.  The researchers found that while personality shapes occupational experiences, the relationship works in both directions and work can become an important source of identity.

- Here's one for you assessment center fans: this study by Speer, et al. adds to the picture through findings that ratings taken from exercises with dissimilar demands actually had higher criterion-related validity than ratings taken from similar exercises!

- Last but not least, presenting research findings in a way that is understandable to non-researchers poses an ongoing--and important--challenge.  Brooks et al. present results of their study that found non-traditional effect size indicators (e.g., a common language effect size indicator) were perceived as more understandable and useful when communicating results of an intervention.  Those of you that have trained or consulted for any length of time know how important it is to turn correlations into dollars or time (or both)!

That's it for now!

March '14 IJSA

In my last research update just a couple days ago, I mentioned that the new issue of IJSA should be coming out soon.

I think they heard me because it came out literally the next day.

So let's take a look:

- This study adds to our (relatively little) knowledge of sensitivity reviews of test items and finds much room for improvement

- More evidence that the utility of UIT isn't eliminated by cheating, this time with a speeded ability test

- Applicant motivation may be impacted by the intended scoring mechanism (e.g., objective vs. ratings).

- The validity of work experience in predicting performance is much debated*, but this study found support for it among salespersons, with personality also playing a moderating role.

- A study of the moderating effect of "good impression" responding on personality inventories

- This review provides a great addition to our knowledge of in-baskets (a related presentation can be found through IPAC)

- Another excellent addition, this time a study of faux pas on social networking websites in the context of employer assessment

- According to this study, assessors may adjust their decision strategy for immigrants (non-native language speakers)

- Letters of recommendation, in this study of nonmedical medical school graduate students, provided helpful information in predicting degree attainment

- Interactive multimedia simulations are here to stay, and this study adds to our confidence that these types of assessments can work well

Until next time!

* Don't forget to check out the U.S. MSPB's latest research study on T&Es!

Will robots replace assessment professionals?

Technology and assessment have had a close relationship for years.  From the earliest days of computers, we were using them to calculate statistics, store items, and put applicants into spreadsheets.

Over time as computers advanced, we used them for more advanced tasks, such as multiple regression, applicant tracking, and computer-based testing.

With the advent of the Internet, a whole new area of opportunity opened for us: web-based recruitment and testing.  People began "showing off for the world" by creating personal webpages, commenting on articles, writing blogs, and living their lives through online social networks.  We developed Internet testing, allowing applicants to examine more conveniently.  And new forms of assessment opened up, such as advanced simulations.

We now find ourselves evolving yet again to take advantage of another significant technology advance: the social web.  As millions and billions of people began living their lives publicly on the web, they began developing a web identity and leaving footprints all over the place.  It was only a matter of time before recruiters (historically some of the first in HR to embrace technology) figured out how to harvest this information.  

One of the hottest trends now in HR technology is scouring the web to seek out digital footprints and making this information readily available to recruiters.  It's the latest iteration of Big Data applied to HR, and it's a creative way to make Internet recruiting more efficient.  Companies like Identified, TalentBin, Gild, and Entelo offer solutions that purport to lay qualified applicants at your doorstop, without all the hassle of spending hours manually searching the web.  They claim an additional benefit of targeting passive job seekers, who are obviously more challenging to attract.

But just exactly how big of an evolutionary step is this?  How big of a solution will this be?  Will this next evolutionary step result in us working ourselves out of a job?

I don't think so.  And let me explain why.

Fundamentally, assessment is about measuring--in a valid, reliable way--competencies key for successful performance in a field, job and/or organization.  Assessment can be performed using a number of different methods, the biggest ones being:

- Ability testing.  Measuring things like critical thinking, reading comprehension, and physical agility.  These tests seek the boundaries of individuals, the maximum they are capable of demonstrating related to a variety of constructs.  When properly developed and used, these tests have been shown to be highly predictive of performance, although some can result in adverse impact.

- Interviews.  One of the oldest forms of assessment and still probably the most popular.  Like ability tests, interview questions can seek "maximum" performance (i.e., knowledge-based), but they can also be used to probe creativity (i.e., situational) as well as gain a better understanding of someone's background and accomplishments (i.e., behavioral).  Interviews have also been shown to be valid predictors of performance, although they rely heavily on potentially unrelated competencies such as memory and verbal skills.

- Knowledge testing.  SAT or GRE anyone?  Multiple-choice tests have been around a long time, and with newer technologies like computer adaptive testing, don't show any signs of going away any time soon.  While these used to be quite common in employment testing, they have fallen out of favor in many places, which is odd given that they too have been shown to be successful predictors of performance (I suspect it is due to their "unsexy" nature and the fact that they require a significant amount of time to prepare)

- Personality inventories.  While these haven't been used nearly as much as the others above, there is an enormous interest in measuring personality characteristics related to job performance.  While they sometimes suffer from a lack of face validity (although contextualizing them seems to help), they have been shown to be useful, and typically demonstrate low adverse impact.

- Applications.  Also extremely popular, and the most relevant for this topic.  The assumption here is that qualifications and (like behavioral questions) past accomplishments predict future performance.  There is potential truth here, but as we know relying on applications (and resumes) is fraught with risks, from irrelevant content to outright lies.


An important thing that all of these assessment types have in common is that they are employer-generated.  One of the fundamental changes society has seen in the last ten years is an enormous shift to user-generated content at the grass roots level.  Anyone can have a blog, regardless of qualifications, and many of questionable veracity are read more than those written by people who actually know what they're talking about.  Content has become, if it wasn't already, king/queen.

But therein lies the fundamental challenge for aggregating digital footprints/content for use in assessment.  Relying on user-generated content, whether from social networks, blogs, comments, or other sources, is predicated on the assumption that qualified candidates are leaving digital versions of themselves.  In places that you have access to.  And that it is accurate.  And predicts performance.  This may work decently in certain industries, like IT, where it may be nearly universal--and expected--that professionals live their lives publicly on the web.  But for many people in many different professions, they may have neither the time nor the inclination to reveal their qualifications online.  In contrast, you can always test someone's ability, and a significant advantage of ability testing is it gives candidates an opportunity to demonstrate what they can do even if they haven't had the chance to do it yet.

I should note that using this information for recruitment is a different--but related--animal.  In this context, concerns about replacing tried-and-true assessment methods are moot.  However, we should carry the same concerns about content generation, both frequency and veracity.

As I've said before, taking technology to its logical endpoint would result in a massive database of everyone on the planet and their competency levels.  This database would empower users to generate and control their content, but allow organizations the widest possible field of qualified candidates.  At this point I'm aware of only one thing that comes close, and honestly I don't see anything approaching this scope anytime soon, particularly with more and more concerns over digital privacy.

Which leaves us...where exactly?  Will robots replace assessment professionals?  Not anytime soon.  At least not if we want hiring to work.  But we should be active observers of these trends, looking both for opportunities as well as pitfalls.  We shouldn't fear technology, but rather the way it's used.  Any important endeavor that requires human analysis should use technology as an assistive tool, not a sexy replacement.

I also want to give props to these companies for taking advantage of user-generated content.  It's a much more efficient way of assessing (i.e., it doesn't require applicants to in some sense double their efforts by completing a separate assessment).  And it's not surprising that these companies have sprouted up, given the trend in HR to automate user-initiated activities that lend themselves to automation, such as leave requests, benefit changes, and training.  But importantly, the science of whether digital footprints predict real-world job performance is in its infancy.  With something as important--operationally as well as legally--as hiring, we have to be careful that our addiction to technology doesn't outstrip our evidence that it works.

May mega research update

It's time once again for the monthly research round-up.  So let's dive right in:

The June International Journal of Selection and Assessment doesn't disappoint; let's take a look:

- More evidence of the link between personality variables and CWBs; this time with concurrent data in China

- Dovetailing nicely with a post I've been working on regarding promotional testing, this research indicates some interesting characteristics of internal test takers

- Why are supervisors open to behavioral interviews but shun discussion of "structure"?  Looks like how we communicate about them plays a big role.

- More research on self-efficacy, this time teasing apart the concept a bit.

- Always a popular topic: applicant reactions to selection mechanisms.  This time with a sample from Saudi Arabia.

- Speaking of applicant reactions...how about another study?  This one comparing U.S. and Vietnamese college students.  By the way, not surprisingly work samples came out a winner in both of these studies.

- Next, a fascinating study of a hidden bonus to UIT: despite the cheating element, it likely increases your candidate pool and eventually performance outcomes

- Speaking of response distortion, here's another study, this time of military cadet selection using personality inventories

- Okay, one more on inflation.  This time a study of Chinese applicants--no difference compared to American samples.

- Back in March I wrote about a study Jeremy Bernerth published in J.A.P.  that got a lot of attention.  This time, Bernerth studied ethnic differences and found minority status was negatively related to credit scores.


Moving on to the summer issue of Personnel Psychology:

-  The "file drawer problem" is the theory that nonsignificant results are less likely to get published.  According to this study, that appears unlikely.  But IMHO looking at all correlations is different than looking at the correlations key to one's hypothesis(es)...

- Back to faking (that may be this post's theme!), can response elaboration reduce faking on biodata items?  This study suggests so.  Although I'm left wondering...what was the impact on validity?

- Speaking of biodata, there are various ways of keying these items.  This research suggests the best method depends on your sample size, although rational keying performed the worst.


How about the May issue of Journal of Applied Psychology?

-  Well this is interesting...Chad Van Iddekinge and his colleagues have provided an updated meta-analysis on the criterion-related validity of integrity tests.  What did they find? Well, the results appear to be less promising than those published previously (e.g., corrected r=.18 for job performance).  Much like SIOP's research journal, this time J.A.P. published several commentaries in response to the study that...well, let's just say a debate ensued about the analysis...

- The Dark Triad.  It sounds like something in a Dan Brown novel.  But in this meta-analysis the authors show that personality characteristics that make up this triad (Machiavellianism, narcissism, and psychopathy) explain some variance in CWBs.

- Why are some people more proactive in seeking career goals than others?  It's an important and under-researched question.  In this study the authors show that part of the explanation lies in "future work selves", or how people's hopes and aspirations as they relate to work.

- Think self-reports of CWBs are biased?  Perhaps not, according to this new study.

- Interested in what causes proactive customer service behavior?  According to this multi-national study, self-efficacy is a key (along with service climate).

- Why do some leaders engage in more self-interested behavior than others?  Perhaps not surprisingly, it appears due in part to the strength of their moral identity.


The May issue of the Journal of Applied Social Psychology has a couple gems...

- Hey, look, turns out being sensitive to your subordinates pays off.  Talk about a lesson that needs frequent repeating...

- And that's it.  Oh, wait, just this little study about using Facebook profiles to predict job performance...that I wrote about before....available in FULL right now...


Okay, getting to the end...The May/June issue of HRM:

- An interesting study of adverse impact in promotion decisions for managers in a Fortune 500 retailer.  The authors compared three methods (top-down assessment, assessment centers, and  multisource appraisal) and the results demonstrate how complex these analyses are!

- Speaking of complex.  Think that successful job postings on the web is just fancy graphics?  Think again--it still involves some classic factors like the labor market, firm reputation, and compensation incentives.  The more things change...

- Identifying future leaders.  There are few other issues that are as important for most organizations.  Yet how exactly to do it eludes many.  These authors propose a model that focuses on four main features: analytical ability, learning agility, drive, and emergent leadership.


Finally, a few from PARE:

- Does item order impact response anxiety?  Not according to this study.

- What's that?  How do we use a new jacknife procedure to eliminating items and improve structural equation modeling?  You're in luck.

- Looks like a lot of research rely on beta weights when interpreting and reporting multiple linear regression results.  But there's so much more...


Happy reading!

Facebook fallout continues

The fallout from earlier reports of employers asking applicants for their Facebook passwords continues. Obviously a nerve was struck.

Today, U.S. Senators Richard Blumenthal and Charles E. Schumer formally asked the Department of Justice (DOJ) and Equal Employment Opportunity Commission (EEOC) to launch an investigation into whether this practice violates federal laws.

From the press release:

"Blumenthal and Schumer argued that this disturbing practice represents a grave intrusion into personal privacy that could set a dangerous precedent for personal privacy and online privacy, make it more difficult for Americans to get jobs, and expose employers to discrimination claims"

"'With few exceptions, employers do not have the need or the right to demand access to applicants’ private, password-protected information.'”

"'In an age where more and more of our personal information – and our private social interactions – are online, it is vital that all individuals be allowed to determine for themselves what personal information they want to make public and protect personal information from their would-be employers. This is especially important during the job-seeking process, when all the power is on one side of the fence.'"

"In their letter to the Justice Department, Blumenthal and Schumer pointed out that two courts have found that when supervisors request employee login credentials, and access otherwise private information with those credentials, that those supervisors may be subject to civil liability. Although those two cases involved current employees, the courts’ reasoning does not clearly distinguish between employees and applicants."

"Blumenthal and Schumer also announced that they are currently drafting legislation that would seek to fill any gaps in federal law that allow employers to require personal login information from prospective employees to be considered for a job."


In related news, on today's Talk of the Nation show, they discussed this issue with a reporter from Wired magazine and an HR consultant.


There are a lot of issues here, ranging from online privacy to public reputation to discrimination, but one that I think deserves more attention is how employers can legitimately get the type of information they're seeking. Again, we're not talking about a background check for, say, a peace officer position, we're talking about your run-of-the-mill clerical job. Basically employers are hungry for any information like displays of poor judgment, a negative attitude about their employer, duplicity in their application, etc.

How might an employer get this type of information without resorting to asking for applicant passwords? It's pretty simple actually, we go back to the basics such as:

1) Reference checks; highly under-used and maligned, with many organizations unaware of technological advances made in this area that make it more likely they'll get the information they need.

2) Work sample/performance tests that simulate actual job tasks. These can be very effective in determining how an applicant will respond in an actual situation (i.e., where things like judgment are important).

3) Situational judgment tests: a lower fidelity version of a performance test that nonetheless can be very effective at assessing candidate's knowledge of, and propensity to engage in, appropriate behavior in various situations.

4) Personality inventories: made to measure things like conscientiousness, openness to experience, and extraversion, which may all be good or bad things depending on the needs of the position.

Bottom line: there are other--better--forms of assessment out there that have been around for a long time and when done well, do the trick. No need to ask for someone's online diary.


As a reminder, for those of you that are IPAC members, I gave a webinar about this topic about a year and a half ago where I gave an overview of the technology as well as a summary of many of the major challenges inherent in this practice. The recording is available in the Members Only area.

Epic fail: Employers continue asking applicants for Facebook account

The Associated Press put out an article this week about the ongoing trend of employers doing something stupid: asking applicants for their Facebook passwords.

Why stupid? Let me count the ways:

1) It makes the applicants feel like they've applied to a totalitarian regime. And they'll tell others, which goes to your reputation. And what do we know about employer reputation? It drives who applies for your jobs.

2) Employers are likely to see things they wish they hadn't. I don't just mean people passed out drunk at a party, I mean things like religious affiliation.

3) If you're trying to access their profile on your own, many are marked private and you won't see anything.

4) If you ask them to log into their account during the interview, it's like asking to see their personal diary.

5) The content on people's FB page is largely outside their control (e.g., comments, photos they're tagged in).

Oh, and let's not forget:

6) The content of someone's profile--aside from things like education and work history which you should have already--is likely to be totally unrelated to job performance, regardless of its potential usefulness, because frankly most employers aren't graduate students in psychology who have received training on interpreting Big 5 characteristics.

The only caveat I can think of is when this request is made as part of a full background check, in which case pretty much your life is an open book.

Facebook, notoriously unpredictable regarding its privacy policies, subsequently warned employers not to do this...but I don't anticipate that this will stop. Why? Because employers are obsessed (rightfully so) with getting as much--and as varied--information as they possibly can.

This just isn't the right way to do it.

February 2012 research update: Facebook; hope; 3-option items; cognitive ability and racism

So this month the big buzz is over a research article in JASP on using Facebook profiles to judge personality and predict job performance. So let's tackle that one first:

Kluemper, et al. had three trained university students judge Big Five personality factors (using the IPIP) based on over 200 public Facebook profiles of other students. A much smaller sample (56) was used to determine links between evaluator judgments and job performance as rated by supervisors.

So what did they find? Well, among other things:

(1) inter-rater reliability ranged from .48 to .72 which seems low to me but apparently is typical for other-ratings of personality;

(2) two of the other-ratings (emotional stability and agreeableness) significantly correlated with supervisory ratings (around .30), as did one of the self-ratings (extraversion), suggesting to me the two methods might vary in constructs being measured;

(3) the same two other-ratings added incremental validity beyond self-ratings, whereas the opposite was not true.

To their credit, the authors caution readers (those that actually go beyond the mainstream press articles) about using their results to support hiring decisions based on Facebook. In fact, I'd like to quote them:

"Our findings should not be used by organizations as unbridled support for using SNWs [Social Networking Websites] in employment selection. Without more evidence of criterion-related validity and comparability with established employment selection methods, the use of SNW information for hiring purposes is tenuous. In addition to the potential for employment discrimination, there are privacy rights and ethical issues associated with accessing personal information. Clearly, research investigations of such issues lag current informal HR practices."

So bravo to them for researching this issue, and double-bravo (that's a technical term) for cautioning those with Facebook fever.


Let's move to other research out there...

The March issue of IJSA is chalk-full of great research, so let's take a look:

- Much hay is made over the "type" of validity exhibited by cognitive ability tests (don't get me started). To the extent this distinction makes sense to you, you might enjoy Frank Schmidt's argument that ability tests in certain instances can demonstrate content validity. The article is followed by several commentaries and a response.

- Next up is O'Neill, et al. with a critical review of Stevens and Campion's Teamwork-Knowledge, Skills, and Ability Test.

- Reeder, et al. investigate individual differences as they relate to the perception of cognitive ability tests.

- Now here's somethin': Edwards, et al. argue that the three-option multiple choice item is underutilized. This should be read by everyone who has nightmares about writing distractors.

- Hoffman and Meade argue that score differences across assessment center exercises reflect true differences rather than measurement artifact.

- Got hope? Zysberg shows that, through problem-solving-oriented coping, hope is related to success in a selection process.


We already know that people generally aren't very good at accurately describing their skills and abilities, but in the March Psychological Bulletin, Freund and Kasten provide an illuminating meta-analysis indicating that while the relationship between self-estimated and psychometrically measured cognitive ability is modest (.33), it varies depending upon scales and dimensions.


For anyone measuring contextual performance, consider the relationship between role expectations and OCBs as described by Dierdorff, et al.

Speaking of OCBs, Nielsen et al.'s study suggests when measuring OCB expression in a group setting, consider the level of task interdependence.

On the other hand, if you are interested in job performance ratings, be aware that there may be gender bias, and it differs in direction between performance ratings and promotability ratings (with the latter favoring males), according to a recent meta-analysis by Roth, et al.


Last but definitely not least, research by Hodson and Busseri suggest that individuals lower in cognitive ability may be predisposed to exhibit more racism. This suggests that using ability tests may not only increase the validity of your selection process but lower your chances of discriminatory behavior.

****

On a side/editorial note, I find it fascinating and somewhat frustrating that the research energy still seems to be about teasing out major constructs such as cognitive ability and personality. As a practitioner, I gotta tell ya I'm happy when hiring supervisors use any sort of structured assessment beyond their standard interview. I'd love to see more energy behind increasing the validity of the entire selection process. I doubt I'm the only one that feels that way.

Is Google+ what we've been waiting for?

No doubt by now you've heard about Google+ ("Google plus"). It's essentially Google's latest stab at trying to topple Facebook as the global social networking leader. While Facebook holds a lot of promise in the way of recruiting, selection, and other core HR functions, its use has been sporadic due to a number of issues. Can Google+ give us the functionality we've always wanted--and stick around long enough to be a major player?

If you haven't heard of Google+ (or haven't read enough), you can get more information here or here or here or here. Or heck, just watch their intro video. By some estimates it already has around 10 million users, which is pretty amazing considering it only came out a few weeks ago, although it's not anywhere near Facebook which claims it has 750 million.

But let's not get ahead of ourselves. Let's first review why Facebook isn't the holy grail we once thought it could be.

At first blush, a public social networking site holds a heck of a lot of promise. It allows organizations to learn more about potential candidates, things beyond a test score. It allows people to network, potentially increasing the speed and efficiency of information sharing. And it allows applicants to learn more about organizations--faster and more informally than a career webpage.

This all sounds good in theory. And some organizations have made Facebook work for them, by using things like fan pages. But for many, the promise has never been fulfilled. Let's look at the main reasons why, and how Google+ may be the answer.

The main problem is that on Facebook, there's just one you. When you post something on Facebook, it goes to all your connections. Friends, family, co-workers, acquaintances--everyone, unless they've "hidden" you. (Yes, you can post to a smaller group but it's a pain and who actually does this?) This has several implications:

- You watch what you say. Do really want to say the same thing to your friends that would to your family? Do you really want to post that picture for everyone to see?

- It increases "lurkers", who simply read but never post, thus not being true participants in the online social interaction.

- This hyper-openness serves as a barrier to some users, who are simply uncomfortable sharing their life with people.

- You have friend request anxiety. You dread getting an email that your boss or someone you don't really know wants to be "friends" on Facebook. If you say no you feel like a jerk, if you say yes then you're openness goes down a notch. What fun is that?

Google+ deals with all this by allowing you to create something called "circles" where you add individuals to certain groups (e.g., friends, colleagues, family) which thereby allows you to post things to only those groups. It even allows you to view your own profile as if you were someone else (IMHO one of the most innovative features).

What are the implications of these circles? There are many:

1. People are more likely to create profiles. Along with the increased attention to security attracting new people, once it gets out that you can manage your social identity much better (and easier) with Google+, it's more likely that someone will create a profile to begin with.

2. Individuals will be more likely to accept invitations to connect from organizations and recruiters. Why? Because they'll be able to manage what information gets shared with those individuals.

3. The information employers have access to will be less--but of higher quality. Assuming people think before posting (yes I'm giving people the benefit of the doubt here), while there may be less information on a profile for an employer to view, it will be more relevant. Instead of posts about children or parties, it will be opinions or accomplishments--things that might actually be job related and much less likely to get employers into legal hot water.

4. It could help with referrals, as individuals will feel more comfortable sharing information about jobs--or their own interest in jobs--without fear of what their management might think.

5. It could give potential applicants a more realistic job preview. With the concern about tanking your job lessened, people will be more likely to be open about the good--and the bad--things about where they work.

You can envision how Google+ has broader applicability for organizations. It allows organizations to create employee-only groups. It allows employees to create informal social groups--or more formal interest groups. It adds another way for colleagues to share knowledge. And it helps create that intangible bond that connects co-workers in a way that meetings and off-sites never can.

So there is quite a bit of promise, but there are still many questions. Could Facebook add design elements to mimic aspects of Google+? Absolutely (and I strongly suspect they are in the process of doing just that). Could Google fail to attract enough followers to its new site to make it the killer app that Facebook is? Sure; in fact there's a good example of this in Orkut (although it is quite popular in Brazil and India). Are there examples of many social networking sites that have flared and fizzled? You bet (heard of MySpace?). And the juggernaut that is Facebook is not to be ignored.

But there are reasons to believe that this could be the real deal. Google spent a lot of time testing this thing out and appears to be listening intently to users on issues from design to privacy--something Facebook has been grilled about for as long as I can remember. And I didn't even touch on the other features of Google+, such as real-time group video chat.

The bottom line is when it comes to websites, most of us are followers. All it takes is your friends and colleagues to start posting somewhere else (heck, it's just another bookmark), and before you know it Facebook could start looking a lot like another casualty in the hyper-competitive web wars. Fortunately, organizations will be the better for it.

Mega research update

Time for a big research update. There's a lot to catch up on, so I'll go quickly and give you just enough for you to follow up on if you're interested.

- More support for the attraction-selection-attrition model

- A 25-year review of the study of leadership outcomes

- Faces with stereotype-relevant features are more open to prejudice

- Early socioeconomic experience plays big role in future risk-taking behavior

- Taking the perspective of another may be key to reducing racial bias

- Generational differences in Big 5 personality factors

- How to improve management research

- Pros and cons of using social networking sites to make HR decisions

- Research directions for talent management

- Organizational branding and "best employer" surveys

- Using social networking sites for hiring decisions (if we just had a single database...)

- How computer adaptive testing aids in delivering unproctored internet tests

- How broad dimension factors may improve assessment centers

- Practical intelligence predicts success as an entrepreneur

- Psychological capital (efficacy, hope, optimism, resilience) is related to job performance

- Expressions of personality factors varies with the situation

Last but not least, this gem, a meta-analysis of the efficacy of simulation games for instruction. Relevance for us? As active engagement went up, so did learning. Lessons? Consider interactive simulations for recruitment and selection, but make sure the viewer is truly involved and not just a spectator.

Unvarnished Unwrapped

A few months ago I mentioned a website called Unvarnished, which was getting a lot of mixed press. The basic concept is (as they describe it) Yelp mixed with LinkedIn. You provide anonymous reviews of people you've worked with. Call it a social resume, call it a web-based reference, I call it fascinating. And something that anyone interested in recruitment and assessment should pay attention to.

I had a chance recently to test out the site, and then I met with the co-founder, Peter Kazanjy. I'm still not sure which direction this will go, but I think you'll agree after reading what follows that the concept merits our attention.

The Test Drive
First, the test drive, which started with an invitation through Facebook from Peter. After spending some time on the site, I'm more optimistic about Unvarnished in some respects, more cautious in others.

Why optimism? The site hits it out of the ballpark on two accounts: it's simple and fast (at least someone learns from Google). Simple and fast is good, because one of the biggest challenges will be building a large community and making reviews easy helps immensely.

Even better, the ratings are relevant. This isn't a popularity contest, it's an honest attempt to provide a useful description of someone's performance. While it's unlikely the rating scales were developed after reading a Personnel Psychology meta-analysis, I was pleased to discover that they pass the smell test and some even have benchmarks.

The ratings consist of an overall performance rating (5-point, anchors described), what job you are rating the person in, four 10-point scales that are described but not anchored (skill, relationships, productivity, integrity), and an open-ended strengths/areas for improvement box. That's it. It's like a super-basic reference check form that takes all of about a minute. You can see what this process looks like below.

Your Unvarnished homepage is fed by your network and generates suggestions for your review (PMYK--People You Might Know). You can review people at any time (even people who haven't claimed a profile) or request reviews using your Facebook contacts. The open comments section is limited to 500 characters to encourage people to review and move on.

The site was developed to be heavily reliant on algorithms. A reviewer's reputation is based in part on the pattern of reviews they have generated as well as how their reviews have been rated. Recommendations for reviews are backed by similar math. A smaller (but important) feature is a profanity filter, which may allay some concerns regarding people looking to settle a score.

Speaking of Facebook, one of my major concerns is it relies HEAVILY on Facebook (not unlike Quora). At least in its current iteration, your identity is verified through having a Facebook account, you invite people through your Facebook contacts, and invitations are posted to Facebook. This is both a potentially good thing (e.g., cuts down on spammers) as well as a bad thing (e.g., not everyone wants to use their Facebook profile in the service of another website). It also begs the question of what would happen if Facebook went belly up. Read on to see what Peter had to say about this.

The interview

I happened to be in San Francisco (interestingly enough to go the Exploratorium) and had an opportunity to talk with Peter about the product and his company. After talking to him for about 10 minutes, one thing became abundantly clear: this guy has thought a great deal about online reputation management. I'd love to get him and Bob Hogan together.

Background
The impetus for the site came from a couple directions. One was a previous job where he was continually surprised that competitors (e.g., a certain company we'll call Bicrosoft) would recruit away the lesser talented employees. Why? His theory is they lacked important information--namely the reputation people had within the organization. (One could argue that they should have done better reference checks, but we all know how easy/productive those can be)

On the other end of the spectrum, he saw excellence not being properly recognized and questioned whether upper management really knew what their talent looked like (a point not lost, btw, on purveyors of performance management software with their 9-box grids and helicopter views).

His ah-ha moment (or one of them) came when he realized that if social ratings can work for things like books and software, couldn't they work for people? If he could develop a site that aggregated high quality information about people's performance, talent decisions would be higher quality as well as more fair (it's hard to argue with that goal).

Peter also believes there is an important employer segment not being served by existing background/reference checking processes. Employers that hire hourly workers rely largely on criminal/credit checks. Those hiring for executive-level positions often rely on high cost search firms. But for employers hiring large number of employees in the middle, there isn't really a good option.

Such a site would have three primary users: those being reviewed, those providing reviews, and those using the information (e.g., employers checking out candidates and vice-versa). And quite commonly a single individual could be in all three roles at various times. The site would have to accommodate all three perspectives.

Concerns/criticism
So what about my concerns? When it comes to the reliance on Facebook, Peter pointed out that it's a good bet Facebook will be around for a while, but the site is not being built to rely solely upon it. It has--or will have--the ability to use contact information from sites like Gmail and LinkedIn. I still have a concern about forcing people through Facebook, so it will be interesting to see whether this impacts the ability to generate reviews.

Concern from others has focused on the potential for abuse. But Peter made several important points. First, this isn't like an online newspaper comment space--those are anonymous with no repercussions for inaccuracy. In Unvarnished your reputation would suffer and your reviews become less valid (assuming your reviews are themselves reviewed). Second, this information is largely out there (or potentially at least) in the form of places like Twitter. But it's not in a central location that can be easily managed, and it's not objective. By guarding invites and relying on anonymity, the goal is to make legitimate reviewers feel safe in leaving honest feedback (whether it's an A+ or a D-) without worrying about the interpersonal implications.

What about stickiness--why would someone want to keep coming back? They're working on several (re)engagement initiatives. One idea is to provide people with periodic updates letting them know when people in their network have been updated (let's hope it doesn't turn into those updates from LinkedIn that are easy deleted). They're also working on the ability to follow an individual based on your "gestures" (e.g., you've reviewed them).

Future directions
The team is considering adding several features. One is the ability for reviewers to better define their relationship--e.g., describing not only the organization they worked together in but their relationship. This would factor into behind-the-scenes algorithms but would not be published.

They're also discussing allowing people to identify themselves, but this raises all of the associated issues such as accuracy, thoroughness, and feeling a need for reciprocity.

Another is allowing access to premium features (e.g., at a cost) such as making trusted reviews more obvious, something that "super users" such as recruiters would likely be willing to pay for.

In terms of opening up access, they're in no big rush to expand access beyond Facebook invites. While this may hinder their growth, it helps keep the data quality high, and they're willing (smartly, I think) to make this trade off.

Conclusion
Currently the company is focused on acquiring the talent it needs to succeed (check out the way they recently advertised for new engineers). One of Peter's primary concerns is that the community evolve in the right direction. Right now it's somewhat of a "love fest" with lots of positive reviews. The site will gain in usefulness when reviews are a combination of pros and cons.

Given what we know about performance ratings, it will be interesting to see if the existing invitation and rating process is sufficient to generate that depth. It also remains to be seen whether some type of incentive will need to be given to generate reviewers.

My overriding concern continues to be the size and diversity of the user group (right now primarily filled with Silicon Valley IT folk). Accuracy, something other writers have been obsessed with, is less of a concern for me after kicking the tires and talking with Peter. But we'll see how the promise and concerns ebb and flow with the user base as well as changes to the service.

At the very least, I hope you'll agree with me that this website is a fascinating development and one we should watch. After all, you might want to use Unvarnished to provide feedback on someone. Or research a potential boss. Or research an applicant. Or...you may be the applicant.

June 2010 IJSA

The summer journal season continues with the June 2010 issue of the International Journal of Selection and Assessment. Take a deep breath, there's a lot of stuff packed into this issue:

- Roth et al. provide evidence that women outperformed men on work sample exams that involved social skills, writing skills, or a broad array of KSAs. To the extent that an employer is trying to avoid discriminating against female applicants, this provides support for work sample usage.

- In a study of managers in Taiwan, Tsai et al. show that the most effective way an applicant can make up for a slip in an interview is to apologize (vs. attempting to justify or use an excuse).

- Jackson et al. strive to add some clarity on task-based assessment centers

- Blickle & Schnitzler provide evidence of the construct and criterion-related validity of the political skill inventory

- Colarelli et al. studied how racial prototypicality and affirmative action policies impact hiring decisions. Results of a resume review indicated more jobs were awarded to black candidates as racial prototypicality and affirmative action policy strength increased, but stronger AA policies decreased the percentage of minority hires attributed to higher qualifications.

- In my personal favorite article of the issue, Karl et al. found in a study of U.S. and German students that those low on conscientiousness (especially), agreeableness, and emotional stability were more likely to post "Facebook Faux Pas". This provides some support for employers who screen out applicants based on inappropriate social networking posts. I'll talk more about this in my upcoming webinar.

- Denis, et al. provide support for the NEO PI-R's ability to predict job performance in two French-Canadian samples.

- Bilgiç and Acarlar report results of a study of Turkish students and perceptions of various selection instruments. Interviews were rated most highly and there were some differences in terms of privacy perceptions depending on the goal orientation of the student.

- Trying to figure out how to hire better direct support professionals (e.g., those providing long-term residential care or care to those with disabilities)? Robson, et al. describe the development of a composite predictor composed of various measures (e.g., agreeableness, numerical ability) that predicted performance, satisfaction, and turnover.

- Ahmetoglu et al. provide support for using the Fundamental Interpersonal Relationship Orientations-Behaviour (FIRO-B) to predict leadership capability.

- Ispas et al. describe results of a study that showed support for a nonverbal cognitive ability measure (the GAMA) in predicting job performance in two samples.

- Last but not least, in another win for context-specific assessments, Pace & Brannick show how a measure of openness to experience tailored to specific work outpredicted the comparable general NEO PI-R scale. IMHO this is how personality measures will eventually become more prominent and accepted as pre-hire assessments.

Is recruiting using SNS discriminatory?

I keep reading/hearing about how recruiting using social networking sites (SNS) opens employers up to discrimination lawsuits because of who uses the sites. For the most part, this just plain isn't true.

A recent Pew study is the latest to show that when it comes to using SNS like Facebook, MySpace, and LinkedIn, you really should have one primary demographic concern when it comes to ensuring a diverse candidate pool: age.

Not gender, at least not in traditional sense. While four years ago SNS users tilted slightly male (55%), the balance has essentially flipped today (54% female).

Not race, there simply do not appear to be generalizable differences in racial groups when it comes to these sites (in fact I've seen some data that suggest the user base on these sites is more diverse)--but things change, and this may vary with particular sites, so keep an eye on this one.

But when it comes to age, SNS users are disproportionately younger than the overall Internet population. In the words of the Pew report, "[this] doesn't mean that more older adults aren't flocking to SNS--they are--but younger adults are ALSO flocking to the sites, so the overall representation of the age cohorts in the SNS user population has actually gotten younger."

One demographic difference I don't see a whole lot about: disability status. Are individuals with disabilities more/less likely to use SNS? I think that's an important question we need to address if we're truly trying to diversity our candidate pools.

SNWs for R&A professionals

In the August 2009 issue of IPAC's Assessment Council News I write about how recruitment and assessment professionals can take advantage of social networking websites (SNWs) such as LinkedIn, Facebook, and Ning, and offer some cautionary notes about using them.

The article covers what will be familiar ground to many of you, but I tried to also talk about how we can use these sites for professional development, not just for sourcing or selection.

The article ends with a very Web 1.0 idea--a solicitation of letters to the editor. I'll make the same request here: what do YOU think about these websites--flash in the pan or here to stay? Approach with kid gloves or jump right in?

Research update

Here are some research studies that you may have missed:

Predicting applicants' job pursuit behavior from their selection expectations: the mediating role of the theory of planned behavior

Complexity in the relationship among the subdimensions of extraversion and job performance in managerial occupations

The use of personality test norms in work settings: Effects of sample size and relevance

Networking as a job search behaviour: A social network perspective

Bonuses:

Avue recently published a white paper titled "The state of the federal workforce: A case for insourcing" that, while focusing on the federal government, has some good information and tips for all of us.

Can't get enough analysis of the Ricci decision? Check out this webinar on 8/14.

Reminder: Want more frequent updates? You can follow my shared items here.

Turn off qualified applicants in one easy step

Looking for a way to turn off qualified applicants in one easy step? The City of Bozeman, MT may have put its finger on it.

Turns out they have had--for several years--a requirement that all applicants seeking a position with the City must, after a conditional job offer that required a background check, turn over their ID and passwords for all social networks they're on, including Facebook and Twitter. After a firestorm of criticism, they decided to suspend the policy pending "a more comprehensive evaluation."

With all due respect to city officials...what were they thinking?

Put aside for the moment the potential problems of violating the terms and conditions of the social networking sites (which generally prohibit sharing passwords), and the potential legal issues inherent in finding information you shouldn't, what high-potential applicant worth her/his salt is going to give over their password information? Its akin to asking someone for their diary--and about as valid and relevant to job performance.

According to the City Manager, "choosing not to disclose log-in information did not hurt candidates’ chances of getting the job." Somehow I find that hard to believe.

I can appreciate wanting to perform your due diligence as part of the hiring process, and gathering as much information as you can, but there are tried and true methods of doing this, including detailed reference checks for every hire.

Maybe the proximity to great fishing interfered with judgment making.

New SIOP networking groups

The Society for Industrial and Organizational Psychology (SIOP), bless its heart, is getting into the 21st century.

In their April newsletter they mention that two SIOP members, John Golden and Daniel Russell, have formed a LinkedIn group called I-O Practitioners Network. To join, click here. (If you don't have a LinkedIn account, and shame on you, you'll have a chance to create one)

But wait, that's not all, John Golden has also created a Ning site for I-O practitioners, which comes with joining the LinkedIn group. ("What the heck is Ning?" you say? Think Wikipedia meets MySpace)

So bravo, John! And SIOP, welcome to the party! (now will someone please tell them about RSS feeds?)

Mini book review: Groundswell

Did you know that by reading this blog post you've been put into a category? Yep, at the very least you're a "spectator." In fact, you might be a "joiner", a "collector", a "critic" or even (as I am) a "creator." Where am I getting these labels? They all come from Charlene Li and Josh Bernoff's new book, Groundswell: Winning in a world transformed by social technologies.

This is by no means a new subject. There have been quite a few books in recent years that cover social technologies and how they can be used productively. But these books have tended to have either a more narrow focus (e.g., by focusing on particular technologies or organizational functions such as marketing) or an extremely broad focus. In Groundswell, Li and Bernoff, both VPs and analysts at Forrester Research, describe the current slate of social technologies and provide organizations with a road map of how they should (or shouldn't) be used.

Those of you familiar with Charlene's blog will recognize much of the content of the book--in fact to be honest there aren't a lot of new concepts in the book, which is a potential drawback. The book is, to a large extent, a collection of the various concepts that the authors have already published. But there's no denying that having it all in one place is mighty handy, and the in-depth case studies serve to flush out the details. And those of you that aren't avid readers of the blog will find much in here to digest.

So what is "the groundswell"? According to the authors it's "A social trend in which people use technologies to get the things they need from each other, rather than from traditional institutions like corporations."

These technologies include all the usual suspects, including blogs, wikis, social networks, forums, review sites, tags, RSS, widgets--you get the point. They also touch on newer technologies such as Twitter (which isn't so new anymore).

The biggest strength of the book lies in its description of two concepts:

- the social technographic ladder, a graphical description of various categories of technology participation. These are the categories I mentioned at the beginning of this post and it's how the authors suggest organizations stratify their customers to figure out what social technology will work best.

- POST, the authors' recommendation for how to plan to engage the groundswell. This deliberate, logical approach to using social technologies is the biggest takeaway from the book. It recognizes that not all technologies are appropriate for all organizations and provides users with a rational way of planning the strategic implementation of them.

On balance, the book is an easy read and provides a great mix of big picture considerations with real-world examples. Unless you're completely new to the subject it's not likely to rock your world, but if you're interested in using social technologies but aren't sure how, this is not a bad book to have.

What would improve the book? More specifics--details--on how exactly to use the technologies. Best practices for setting up a Facebook page, for example. The different blogging platforms and their pros and cons. It's not enough to understand the concept of the technology--you need to understand the technology itself.

So what does all this have to do with recruitment and assessment? Quite a bit actually, but mostly with the former (after all, check out where people spend their time). Tapping into social technologies is a great way to spread and monitor your employment brand--i..e, why should people want to work for you? It's also a great way to identify potential candidates and spread the word about opportunities. In terms of testing, we're not there yet (and won't be until we have a secure database of individual test scores). But maybe that's okay--after all you want a job-person match, not a person-test score match.

So why did I call this a mini-review? Because I haven't read the whole book (yet). I've read most of the beginning and latter chapters, but haven't made my way through the middle, which is comprised mostly of case studies. I don't claim to have read it cover-to-cover, so take this review with that in mind.

Web 2.0 Presentation

Last week I had the pleasure of presenting at WRIPAC's April meeting in beautiful Napa, CA.

The topic was "Recruiting in the Age of Web 2.0" and I covered topics such as wikis, social networking, and blogs (which, coincidentally, I'm sure they're talking about right now just over the hills in San Francisco at the Web 2.0 Expo).

For those of you that weren't able to attend but are interested, you can view the slideshow below or here.

Links a go-go

Good reading for December 14, 2007:

Why your company needs to be on Facebook

Target Corp. to pay $510K for race discrimination in application process

Check out the U.S. Office of Personnel Management's virtual conference

Monster launches website targeting Hispanic applicants

SIOP's December, 2007 newsletter

Is your organization green? Candidates want to know.

Using social networking sites to reach out to entry-level hires

The importance of background checks (white paper)

LinkedIn's JobsInsider

Yes, I'm a bit late to the game on this one (okay, a LOT late to the game). But I'm guessing a significant number of my readers aren't familiar with a really useful piece of technology offered by LinkedIn, one of the most popular business social networking sites (11+ million members), called JobsInsider.

What is it? It's a free browser plug-in/toolbar (for IE or Firefox) that serves several purposes. One is simply as a quicker way to access LinkedIn content. But the much cooler feature is that when you're looking at jobs on Monster, CareerBuilder, HotJobs, Craigslist, SimplyHired, Dice, or Vault, a separate window comes up that notifies you if anyone in your LinkedIn network works for the organization and allows you to contact them to help with making the right connections.

Let's look at an example. I went to SimplyHired and looked up jobs working for Apple in Sacramento, CA. When I click on any of the jobs that come up the JobsInsider window pops up and tells me 207 people in my LinkedIn network work for Apple, and two are friends of my connections. I can click on the link and it takes me to a description of those people. Click on any one person and it tells you how you're linked to that person. Here's what it looks like:









Not only that, but (at least with SimplyHired), when job search results come up, you can click on "who do I know?" for each position to have LinkedIn search your network.

Pretty nifty, huh? So why do we care, other than it being a nifty little piece of technology?

For one, it's another reason to be a member of LinkedIn--at least if you're interested in being contacted by applicants. Given the choice between pursuing a job somewhere where I don't know anyone and a place where I can make a contact, I'll take the latter.

Second, it's a good way to double- (or triple-) check credentials of applicants. Most of these networking sites strongly encourage you to put in your educational background and job experience. If what's listed here doesn't match the resume or application they submitted to you, that's something to follow up on. Could be a simple explanation, could not be.

Finally, another reason to care about this is it's likely a sign of things to come. With meta-people search sites like ZoomInfo out there, and ones like Spock coming on board, we need to be very comfortable with our on-line identities and understand how they link to other people.

One last cool feature of the LinkedIn toolbar. When you open an e-mail in Gmail, Yahoo! Mail, MSN Hotmail, or AOL, you automatically have the option to get someone's LinkedIn information or invite them to your network. In fact there's even a tool that will do the same thing for your Outlook mail.

Happy Memorial Day!