One large, diverse candidate pool
One cognitive ability OR physical agility test
One protective services job (police or fire if in season, otherwise corrections)
Optional: large, aggressive employee union
Optional: history of litigation
1. Begin by deciding what type of exam--cognitive ability or physical agility--you feel like giving; don't worry about performing a job analysis first as these are time consuming and boring. If you must, select a small sample of current employees (preferably the poor performers) and provide minimal instruction. Don't worry about whether they are "true" job experts, and whatever you do, don't link tasks to KSAs--everyone hates doing it.
2. Select what you will be measuring. Base your decision on what you feel like, or whatever's easiest. Usually this is just doing what you did last time.
3. Have untrained analysts prepare the exams. Because anyone can do hiring, select whoever has time on their hands. Optional: search the Internet for a test that catches your eye. Rule of thumb is one question per content area (if you have more than one question, you're wasting applicant time).
4. Make sure the reading level of the exam is graduate school-level. After all, isn't reading an important part of any job? And don't you want the best?
5. Next, choose weighting of your exam components either randomly or based on gut feeling. When in doubt, place the largest weight on the test that is most related to cognitive ability.
6. Select a pass point. It should either be: (a) 70 percent; (b) based on administrative convenience; or (c) chosen at random.
7. Administer the exam, preferably with limited advertising. If you must advertise, give applicants a very short amount of time to prepare--after all, this isn't grade school. Do not pre-test the exam--if you've followed these instructions, it should be fine.
8. Score the exam--if you can, avoid "right/wrong" questions and go with ones where you can personally judge the quality of the answer. Don't worry about a boring "benchmark"--you know a good response when you see it.
9. Keep all scoring results and details regarding the process to yourself. Candidates don't need to know (and won't understand).
10. Make final selection decisions. Do not administer yet another test before making your selection, or if you must because of boring rules, make it an unstructured interview. Ask lots of questions like, "If you were a book, what would your title be?" If you have any women or minorities, ALWAYS ask questions about their ability to perform the job.
11. Do not document any of this process. Everyone involved will be with the organization for a long time, and people have really good memories.
Above all: have fun! After all, it's just people's livelihood.
A good example of how this type of thing plays out (albeit in a less horrific manner) is the recent decision in Easterling v. Connecticut Dept. of Corrections.
And for those of you that want to read more about how the law applies to selection, look for an upcoming IPAC monograph written by yours truly!