September 2010 IJSA (those considering SHRM certification, read on)

The September issue of the International Journal of Selection and Assessment (IJSA) is out with a boatload of content. Let's check out some of the highlights:

First up, a piece by Gentry, et al. that has implications for self-rating instruments. The authors studied self-observer ratings among managers in Southern Asia and Confucian Asia and found an important difference: the discrepancy between the ratings was greater in Southern Asia. Specifically, the difference appears in self-ratings rather than observer ratings, indicating differences in how managers in the different areas perceived themselves. Implication? Differences in self ratings may be due to cultural differences in addition to things like personality and instrument type.

The second article is a fascinating one by Saul Fine in which the author analyzed differences in integrity test scores across 27 countries. Fine found two important things: first, there are significant differences in test scores across countries. Second, test results were significantly correlated (r= -.48) with country-level measures of corruption as well as several aspects of Hofstede's cultural dimensions.

Next, an article by De Corte, et al. that describes a method for creating Pareto-optimal selection systems that balance validity, adverse impact, and predictor constraints. This article continues the quest for balancing utility and subgroup differences. A link to the article is here but it wasn't functional at the time I wrote this; hopefully it will be soon.

Next, in an article that SHRM will probably place on their homepage if they haven't already, Lester et al. studied alumni from three U.S. universities to analyze the relationship between attainment of the Professional in Human Resources (PHR) certification offered by SHRM and early career success. Results? Those with a PHR were significantly more likely to obtain a job in HR (versus another field) BUT possession was not associated with starting salary or early career promotions. I'll let you decide if you think it's worth the time (and expense).

If you need another reason to focus on work samples and structured interviews, here ya go. Anderson, et al. provide us with the results of a meta-analysis of applicant reactions to selection instruments. Drawing from data from 17 countries, the authors found results similar to what we've seen in the past: work samples and interviews were most preferred, while honesty testing, personal contacts, and graphology were the least preferred. In the middle (favorably evaluated) were resumes, cognitive tests, references, biodata, and personality inventories.

Fans of biodata and personality testing may find the article by Sisco & Reilly reassuring. Using results from over 700 participants, the authors found that the factor structures of a personality inventory and biodata measure were not significantly impacted by social desirability at the item level. Implication? The measures seemed to hold together and retain at least an aspect of their construct validity even in the face of items that beg inflation.

Speaking of personality tests, Whetzel et al. investigated the linearity of the relationship between the OPQ and job performance. Results? Very little departure from linearity and where present the departure was small. This suggests that utility gains may be obtained across the spectrum of personality test results.

Are you overloading your assessment center raters? Melchers et al. present the results of a study that strongly suggests that if you are using group discussions as an assessment tool, you need to be sensitive to the number of participants that raters are simultaneously observing.

There are other articles in here you may be interested in, including ones on organizational attractiveness, range shrinkage in cognitive ability test scores, and staffing services related to innovation.

The personality echo

Psychologists have known for a while about something called perceiver effects, which refer to general tendencies to judge others in a particular way. For example you may tend to see people as generally self-serving or selfless, open-minded or closed minded, etc.

It turns out that these perceiver effects say something about you. For example, one of the ways Machiavellianism is measured is by asking whether you generally see a lack of sincerity or integrity in others. In a sense, the judgments you make about others echo back and, when interpreted properly, can say something about your personality.

In the July 2010 issue of the Journal of Personality and Social Psychology, Wood, et al. describe the results of several studies of this phenomenon. Most of the previous studies have used an "assumed similarity" paradigm, where the researchers have attempted to confirm that how one views themselves is assumed to transfer to how one views others.

This study, on the other hand, made no such assumptions. The researchers were interested in what impact self-ratings of personality had on perceptions, regardless of whether it was the same trait. The primary relationship they looked at was the correlation between how one scored on a personality inventory and how they tended to rate others.

In three different studies of college students, the strongest trend was related to agreeableness: those that rated others high in agreeableness tended to rate themselves high in the same trait (r's of .19 to .29 depending on the sample).

The second highest was conscientiousness, and in the same direction, but for different traits: those that rated others high in conscientiousness tended to rate themselves high in agreeableness (r's varied from .10 to 25). Rating others high in openness was also associated with higher self-rating scores of agreeableness (r's from .12 to .27).

Interestingly, in the third study the authors also identified moderate positive correlations between perceiving others in a positive light and several individual characteristics, including self-rated agreeableness, fit with peers, and organizational goals, and negative correlations with need for power, social dominance orientation, and depression.

So what does this mean? Essentially this study suggests that if someone (say, a job applicant) tends to describe others in a positive light--specifically, as agreeable, conscientious, and open--there is a significant chance that they themselves will rate highly on agreeableness. Anyone that's ever interviewed someone who makes negative stray marks about previous co-workers likely has intuited this.

There are several important caveats:

1) The effect sizes (correlations) were modest

2) The sample was restricted to university students

3) There are likely important differences between jobs, impacting both the value of agreeableness as well as how these attitudes are formed.

By the way, an in press version is available here.

Generational differences in work values: Fact or fiction?

There's been a lot written over the years about so-called "generational differences" among different age groups--e.g., Baby Boomers, Gen X, and GenMe (i.e., Gen Y, Millennials, Net Generation). Various authors have claimed important differences, such as GenMe valuing altruistic employers and social experiences much more than, say, Boomers. Take a look at the business section of your local bookstore and you're sure to find examples.

The problem isn't with the issue--if there truly are differences, they would have important implications for attracting and retaining different segments of the workforce. The problem is with the data.

Turns out most of the previous research is either qualitative (anecdotes) or based on cross-sectional studies done at a single point of time. The problem with these studies is they make it impossible to separate generational differences from career stage differences. In other words, younger applicants/employees may indeed differ from older ones at any point in time--but that could be purely due to factors that impact one's age, not factors related to being born or experiencing a particular point in time.

Luckily for us, in the September issue of the Journal of Management, Twenge, et al. report the results of a longitudinal study that allows us to answer these generational questions with some authority.

The authors used data collected from a nationally representative sample of over 16,000 U.S. high school seniors taken in 1976, 1991, and 2006 (from the Monitoring The Future project).

The results may surprise you. Let's look at each of the work values studied by the authors:

Leisure (e.g., vacation, work-life balance): This became progressively more important over the generations, with GenMe valuing it the most. The difference between GenMe and Boomers was the largest reported in the study (d>.50).

Intrinsic (e.g., interesting and challenging work): While GenY did not differ significantly from Boomers, GenMe were significantly less likely to value this compared to either GenY or Boomers.

Altruistic (e.g., ability to help others and society): While it's commonly reported that GenMe values this more highly than previous generations, results did not support this. No significant differences were found between the three groups.

Social (e.g., job gives feeling of belonging and being connected): This is another area where some have suggested that with the skyrocketing success of sites like Facebook, the younger generations more highly value--indeed, insist on--a workplace that allows social interaction. The results? Not so much. In fact, GenMe placed less value on this compared to both GenY and Boomers.

Extrinsic (e.g., job pays highly or is prestigious). This is an interesting example of non-linearity. Turns out this value peaked with GenY. While GenMe valued this more than Boomers, the difference was more pronounced between Boomers and GenY.

Among the items with the biggest differences were:

- GenMe valuing having 2+ weeks of vacation compared to Boomers
- Boomers valuing a job that allows you to make friends compared to GenMe
- GenY valuing having a job with prestige/status compared to Boomers
- GenMe reporting that work is just a way to make a living compared to Boomers
- GenX valuing being able to participate in decision making compared to Boomers

Overall, intrinsic reward items had the highest means across the generations, with a job that is "interesting" having the highest item mean. Altruistic and social values were also valued more highly, with extrinsic rewards having lower mean values and leisure rewards having the lowest.

The authors summarize the results by saying the data suggest "small to moderate generational differences." If you aren't surprised, kudos to your observational skills. At the very least this is important data to consider when evaluating your recruiting and retention efforts. And it certainly calls into question some of the conclusions being drawn in the popular press.

By the way, a full version of the article is available (at least it was at the time I published this) here.

Ignore Facebook

Somewhere there is a team--or an individual--that has been tasked with figuring out "how to use this Facebook thing" to assist with recruitment and assessment. Management doesn't know what they want--they may not even know what Facebook is--but they're convinced the organization needs to use it.

For most organizations this is a complete waste of time.

Yeah, yeah, I know a study just came out from Nielsen showing that Americans spend nearly a quarter of their online time on social networking or blogs (an odd combination if you ask me), up 43% from a year ago, and I know Facebook recently announced they surprised 500 million users worldwide.

But just because something's popular doesn't mean it has direct application in all walks of life. Just because the Kindle is hot doesn't mean you should use it as a cutting board.

This is a classic mis-direction. It's like focusing on re-painting the guest room because brown is the new blue, while meanwhile the rest of your house falls apart.

Don't get me wrong, I'm all about innovation in our field, and longtime readers know I'm no hater of online developments. But we should be spending our most valuable commodity (time) where it counts.

What we should be doing is learning from this shift in online behavior and applying those lessons to our existing efforts. What the heck does that mean? Well, let's look at what people are using sites like Facebook for:

1. Sharing their life. Perhaps surprisingly, a lot of people like sharing their lives publicly, and for many, status updates are the 2010 version of the family letter. Think about the extent to which you allow applicants to share themselves. Does your application process consist only of a resume upload, or do you allow them to upload cool things they've done? Do candidates have the ability to customize their job search based on their background, not your jobs?

2. Connecting with people. There at least three direct applications here: (1) think about the extent to which your career portal allows applicants to contact a real human being; (2) think about the possibility of starting an internal network (e.g., Yammer) to capture all that knowledge sharing, bench strength, and informal connection goodness; and (3) consider publishing employee profiles online so potential candidates know who they'd be working with.

3. Playing games. I've written several times over the years about the benefits of interactive and entertaining recruitment and assessment applications. Video and online games aren't just for nerds anymore (see Farmville, the Wii). Think about how you can make your career portal not only attractive and useful--but FUN. Or at the very least engaging.

Maybe the biggest lesson here: your reputation is out there more than ever. If you don't think your employees are updating/blogging/tweeting about their jobs, you're mistaken. And you may have the best sourcers and recruiters in the world, but if your reputation as an employer stinks, it's all over the web and you've just made their job 10x harder. Know how your employees feel. Work hard on improving their work life. Reward good performance and deal with lack thereof.

Don't let the hype about Facebook get to you. Take your time and identify what you really need to be focusing on to increase your talent pipeline.

Disagree? Think we should be spending more, not less, time with these technologies? Bring it. (ya know, with the comments feature)

Human Performance, v23 (3)

Three interesting articles in the most recent issue of Human Performance:

Kanar, et al. studied the impact of organizational reputation on applicant attraction with a sample of college-level job seekers and found that negative information had a greater impact than positive information on attraction to the organization as well as recall, and this effect persisted for one week. Implication? Recognize that negative information about your organization may be processed differently than positive information, and thus not easily balanced. Also reinforces keeping an eye out for negative remarks and addressing them.

Second, a fascinating study by Kell, et al. on how Big 5 personality factors differentially predict various aspects of performance within a single job. Specifically, the authors found (as judged by raters) that emotionally stable and conscientious actions were more effective in task situations and open and agreeable actions were more effective in interpersonal situations. Implications? Not only does it seem that different assessment types predict different general aspects of performance (i.e., personality measures usually are better at predicting contextual performance), there appears to be prediction differences within the same job such that there is value in considering each facet of the Big 5 and its relationship to different aspects of performance.

Lastly, if you're looking for a way to predict performance in jobs that require "multi-tasking", you'll be interested in Poposki & Oswald's description of the development of the Multitasking Preference Inventory. In addition to the development, the authors describe a study of its convergent and discriminant validity as well as initial criterion-related validity. I found it interesting that this is not an ability test but rather a preference inventory, which actually makes sense since the brain doesn't actually focus on multiple things very well!

Learning from orchestra hiring

Lessons come from all sorts of places. Recently an article in the New York Times described the surprising number of vacancies nationwide in major symphony orchestras, including the N.Y. Philharmonic, which will have an unprecedented 12 vacancies next season (12% of its "workforce").

Here are some things that stood out from the article:

1) The customers (audience) may not notice. Top seats do not go vacant because assistant principals step up or substitutes are hired.

2) Economic problems have impacted an incredibly wide breadth of industries. Some positions are left vacant because it is cheaper to do so.

3) Leaders sometimes defer the decision. Some conductors have passed on filling positions to give their successor an opportunity to make the pick.

4) The selection process is resource intensive and difficult to coordinate. This is primarily due to the difficulty in getting the conductor and a committee of players (let's call them SMEs) together to do the judging.

5) Sometimes no decision is made. Even after the months of advertising and auditioning a suitable person isn't found. Kudos to those organizations with the wisdom to pass, assuming a valid selection process.

6) Which raises the next point--the selection process is suspect at best. Why? Well, aside from the historical gender discrimination, the process used to select musicians relies upon a single hour-long work sample test. What if the person is having a bad day? Perhaps more importantly, the most important criteria--symphony performance--relies upon all the players working together.

7) On a related point, applicants aren't being judged solely on technical proficiency. Committee members also judge them on fit, or as one described it, they look for "thinking, thoughtful musicians who are the whole package". What does that mean? It's hard to say, since, as stated in the article, "orchestra officials and musicians are loath to discuss the auditioning process in detail." Here's a question: why? Afraid that the "correct answers" will get out there--or afraid of the critical eye that may be focused on them?

Hat tip.

July 2010 J.A.P.

A new round of journals is out, so let's start with the June issue of the Journal of Applied Psychology.

First up, Schleicher et al. looked at whether there were demographic differences in how much candidate scores improved upon retesting. Turns out there were several. Whites showed larger improvements than Blacks or Hispanics on several assessments, particularly on written tests. Women and applicants under 40 showed greater improvements than men and applicants 40+. Implications? In some situations allowing applicants to retest may exacerbate adverse impact.

Next, an important piece by Aguinis et al. (that you can read here) about test bias. This follows on the heels of the June IOP articles on the same topic and seems to represent a resurgence of interest in a topic that seemed dormant. In this article the authors report the results of a very large Monte Carlo simulation (billions and billions of data points) where they found that if bias is measured using slope-based techniques, it's likely to go undetected, and intercept-based bias favoring minority group members is likely to be found when in fact it does not exist. This study, combined with points made in the IOP article suggest that some of the "established" conclusions regarding test bias may not be as solid as we thought.

Third, for those of you interested in differential functioning (of items or scales), you should check out the piece by Adam Meade where he presents a taxonomy of potential differential functioning effect sizes and also describes a software program created for computing the indices and graphing differential functioning.

Next, a piece by Wang et al. on locus of control. Importantly, they found that when locus of control (LOC) is specific to work-related issues, there are stronger correlations between LOC and work-related criteria such as job satisfaction and commitment. Similarly, when LOC is defined more broadly to include non-work issues, there are some stronger correlations with non-work criteria such as life satisfaction. Implications? Much like research on personality items, specifying a work-related context would seem to increase the predictive power of LOC measures.

Last but not least, an important article on counterproductive work behavior (CWB) and organizational citizenship behavior (OCB) by Spector, et al. CWB and OCB seem like they should be opposites of each other--one demonstrated by disengaged, unhappy workers, the other by engaged, happy ones--right? Not so fast. The authors report the results of an experiment that suggest that the concepts are unrelated and do not necessarily have opposite relationships with other variables. The authors also recommend that when measuring these behaviors, frequency of performance be used rather than level of agreement.