In my last research update just a couple days ago, I mentioned that the new issue of IJSA should be coming out soon.
I think they heard me because it came out literally the next day.
So let's take a look:
- This study adds to our (relatively little) knowledge of sensitivity reviews of test items and finds much room for improvement
- More evidence that the utility of UIT isn't eliminated by cheating, this time with a speeded ability test
- Applicant motivation may be impacted by the intended scoring mechanism (e.g., objective vs. ratings).
- The validity of work experience in predicting performance is much debated*, but this study found support for it among salespersons, with personality also playing a moderating role.
- A study of the moderating effect of "good impression" responding on personality inventories
- This review provides a great addition to our knowledge of in-baskets (a related presentation can be found through IPAC)
- Another excellent addition, this time a study of faux pas on social networking websites in the context of employer assessment
- According to this study, assessors may adjust their decision strategy for immigrants (non-native language speakers)
- Letters of recommendation, in this study of nonmedical medical school graduate students, provided helpful information in predicting degree attainment
- Interactive multimedia simulations are here to stay, and this study adds to our confidence that these types of assessments can work well
Until next time!
* Don't forget to check out the U.S. MSPB's latest research study on T&Es!
Showing posts with label Experience. Show all posts
Showing posts with label Experience. Show all posts
Thursday, February 20, 2014
Tuesday, March 04, 2008
Hiring a president
In my last post I wrote about how looking at pure length of experience probably isn't the best way to pick someone for a top leadership position--like President of the United States. This comes from decades of research on the relationship between work experience and job performance (interestingly Time magazine just published an article that comes to a similar conclusion using research on expertise).
So how would we go about making the most informed decision if we treated this like a hiring decision rather than an election? We've already heard from the recruitosphere on this issue. Now it's time for an assessment perspective.
We know right off the bat we need some tests to differentiate between the best candidates. And like all hiring decisions, we'd choose tests by starting out with good job analysis data. But unfortunately we don't have any.
"Waddya mean?" you say. "There's lots of experts and articles out there that have documented what makes a good president!" Ahh, yes, but that's not how we would conduct a job analysis for hiring someone. We don't just conduct a literature review, we follow the requirements of the Uniform Guidelines by, among other things, creating detailed statements describing the work to be performed and the knowledge, skills, abilities, and other characteristics (KSAOs) necessary to do it. We also have subject matter experts rate these statements on things like critical, frequency, and necessity at entry to the job.
Can you imagine getting the current incumbent in the same room with previous presidents and conducting a job analysis session?! Count me in on that meeting!
But since that's not going to happen, how can we use assessment research to inform the job of hiring a president? What do years of research tell us about hiring someone who is likely to succeed at this type of job?
Here are the tests I would consider:
- Cognitive ability test. For a complex job like president, high cognitive ability is an absolute must, and research shows ability is the #1 predictor for complex jobs. Unfortunately, we (usually) have a field of very smart applicants, so giving them an ability test might not narrow the field.
- A work sample/performance test. Each candidate is given a live scenario that's pretty close to what they'd face as president. A discussion with a world leader, acting quickly in an emergency, a press conference, or serving as mediator between two disagreeing parties. Sit back and rate the performance using pre-established rating scales.
- A structured interview. This is no softball interview with questions about favorite memories. Each candidate gets the same challenging job-related questions and we have a pre-determined rating scale with benchmarks for judging good answers.
- A job knowledge test. A comprehensive written test covering all of the topics that a president would be expected to know. If you think about it, it's rather scary to think that we hire a president without gauging their full knowledge.
- What about a personality test? This is probably the trickiest (but potentially most interesting) of all the tests. If the job analysis showed that a certain trait, measurable by a reputable instrument, related to success (and some attempts at this have been made at this) we could go forward. Research has indicated that particularly when informed by job analysis, personality tests can have useful levels of performance prediction.
- The best: All of these! Imagine an assessment center-like format where the candidates go through a day-long battery of all these tests. Police officer candidates often have to do it--why not arguably the most powerful position in the world?
What don't you see here? "Debates" that consist mostly of canned phrases, speeches to supporters, and policies that may or may not have been written by the candidate. In other words, most of what we have now. This is similar to hiring someone based purely on a resume they created.
Imagine having all the data that these tests would provide. Talk about an informed hire!
So how would we go about making the most informed decision if we treated this like a hiring decision rather than an election? We've already heard from the recruitosphere on this issue. Now it's time for an assessment perspective.
We know right off the bat we need some tests to differentiate between the best candidates. And like all hiring decisions, we'd choose tests by starting out with good job analysis data. But unfortunately we don't have any.
"Waddya mean?" you say. "There's lots of experts and articles out there that have documented what makes a good president!" Ahh, yes, but that's not how we would conduct a job analysis for hiring someone. We don't just conduct a literature review, we follow the requirements of the Uniform Guidelines by, among other things, creating detailed statements describing the work to be performed and the knowledge, skills, abilities, and other characteristics (KSAOs) necessary to do it. We also have subject matter experts rate these statements on things like critical, frequency, and necessity at entry to the job.
Can you imagine getting the current incumbent in the same room with previous presidents and conducting a job analysis session?! Count me in on that meeting!
But since that's not going to happen, how can we use assessment research to inform the job of hiring a president? What do years of research tell us about hiring someone who is likely to succeed at this type of job?
Here are the tests I would consider:
- Cognitive ability test. For a complex job like president, high cognitive ability is an absolute must, and research shows ability is the #1 predictor for complex jobs. Unfortunately, we (usually) have a field of very smart applicants, so giving them an ability test might not narrow the field.
- A work sample/performance test. Each candidate is given a live scenario that's pretty close to what they'd face as president. A discussion with a world leader, acting quickly in an emergency, a press conference, or serving as mediator between two disagreeing parties. Sit back and rate the performance using pre-established rating scales.
- A structured interview. This is no softball interview with questions about favorite memories. Each candidate gets the same challenging job-related questions and we have a pre-determined rating scale with benchmarks for judging good answers.
- A job knowledge test. A comprehensive written test covering all of the topics that a president would be expected to know. If you think about it, it's rather scary to think that we hire a president without gauging their full knowledge.
- What about a personality test? This is probably the trickiest (but potentially most interesting) of all the tests. If the job analysis showed that a certain trait, measurable by a reputable instrument, related to success (and some attempts at this have been made at this) we could go forward. Research has indicated that particularly when informed by job analysis, personality tests can have useful levels of performance prediction.
- The best: All of these! Imagine an assessment center-like format where the candidates go through a day-long battery of all these tests. Police officer candidates often have to do it--why not arguably the most powerful position in the world?
What don't you see here? "Debates" that consist mostly of canned phrases, speeches to supporters, and policies that may or may not have been written by the candidate. In other words, most of what we have now. This is similar to hiring someone based purely on a resume they created.
Imagine having all the data that these tests would provide. Talk about an informed hire!
Saturday, March 01, 2008
Does experience matter?
There's been a lot of talk this election season about whether experience matters when it comes to the job of U.S. president.
There's been a lot of back and forth, but I haven't heard a lot of discussion about whether there's any research related to the question. A lot of folks might be surprised to learn there's actually quite a bit of research directly related to this point. And we can use it to inform decisions like selecting a president--or any other leader for that matter.
So what does the research say? The best sources of research and analysis on this topic (e.g., this one and this one among others) have reached some general conclusions:
1. The most important conclusion is that the answer depends on how you define both experience (e.g., amount, time, type) and job performance (e.g., task, contextual) and the particular job you're looking at. There is no single answer.
2. Experience does predict job performance, but not as well as, say, cognitive ability--this is particularly true for high-complexity jobs.
3. Length of experience best predicts job performance when incumbents have relatively low amounts of experience (e.g., entry-level jobs).
4. Length of experience best predicts job performance when the job is low-complexity. At high levels of complexity it does significantly worse at predicting performance. After, say, about 5 years of experience, more doesn't seem to add anything to predicting performance.
5. Prediction is increased when we look at amount of experience performing particular tasks rather than length of experience. This makes sense--just because someone's held a job for 20 years doesn't mean they've performed the tasks you're interested in (and done them well).
So what does all this mean for, say, choosing a president? I'm afraid the answer is not simple, which is as it should be. Pure amount of experience doesn't appear to be all that important after a few years (although this is difficult to analyze since there's only one incumbent at a time!). Ultimately the question is what type of experience is important--and THAT question hasn't been answered.
For a highly complex job like president, simply looking at experience does not seem the best way to measure and predict performance. So from a personnel assessment standpoint, how would we hypothetically select a president? I'll cover that in my next post: Hiring a President.
There's been a lot of back and forth, but I haven't heard a lot of discussion about whether there's any research related to the question. A lot of folks might be surprised to learn there's actually quite a bit of research directly related to this point. And we can use it to inform decisions like selecting a president--or any other leader for that matter.
So what does the research say? The best sources of research and analysis on this topic (e.g., this one and this one among others) have reached some general conclusions:
1. The most important conclusion is that the answer depends on how you define both experience (e.g., amount, time, type) and job performance (e.g., task, contextual) and the particular job you're looking at. There is no single answer.
2. Experience does predict job performance, but not as well as, say, cognitive ability--this is particularly true for high-complexity jobs.
3. Length of experience best predicts job performance when incumbents have relatively low amounts of experience (e.g., entry-level jobs).
4. Length of experience best predicts job performance when the job is low-complexity. At high levels of complexity it does significantly worse at predicting performance. After, say, about 5 years of experience, more doesn't seem to add anything to predicting performance.
5. Prediction is increased when we look at amount of experience performing particular tasks rather than length of experience. This makes sense--just because someone's held a job for 20 years doesn't mean they've performed the tasks you're interested in (and done them well).
So what does all this mean for, say, choosing a president? I'm afraid the answer is not simple, which is as it should be. Pure amount of experience doesn't appear to be all that important after a few years (although this is difficult to analyze since there's only one incumbent at a time!). Ultimately the question is what type of experience is important--and THAT question hasn't been answered.
For a highly complex job like president, simply looking at experience does not seem the best way to measure and predict performance. So from a personnel assessment standpoint, how would we hypothetically select a president? I'll cover that in my next post: Hiring a President.
Subscribe to:
Posts (Atom)