OA Part 6. Homemade Assessments

​This past week, I was asked to review an assessment currently in use by a relatively large company. One of their senior people, who had some experience with quality assessments, had some doubts about the effectiveness of the one in use by their HR department. It had five separate parts delivered in a slick online presentation. There were drag-and-drop tiles that automatically rearranged themselves. There were cool sliding scales of various types and a few ordinary radio button choices. The terminology used the industry’s key issues and buzzwords. The reports had lovely colored graphs and charts, seven different ones, complete with special names for certain combinations of characteristics. On the surface, it was a most impressive presentation. The only problem was that it was inaccurate to the point of being misleading. It was like a beautifully colored parachute the only problem is that it just wouldn’t open.

There are thousands of such products in the marketplace. The Internet has made it easy to create pseudo-assessments. The fact is that developing serious psychometric instruments is extremely difficult. It requires specialized knowledge and experience that is far beyond the scope of the average psychology professor or industrial psychologist. It is a unique specialty within a specialized world. 

Tests look deceptively simple. It’s just a set of questions and answers, isn’t it? Well, it is easy to make a test that looks like a test…to the untrained eye. Here’s the catch, and it’s a nasty one. The quality of a test, its accuracy, and reliability is determined by its psychometric and scaling properties, not what the test questions look like. It is not the same as homemade cookies, where any can still be eaten. Homemade tests lead companies to hire the wrong people. They encourage companies to spend large sums of money on training programs that return less than expected. They lead individuals into jobs in which they cannot excel. They affect the lives of real people. They are often just good enough to be dangerously wrong.

Certain tasks are best left to professionals. Gathering wild mushrooms, parachute packing and assessment development are all examples of this because the possible consequences of amateur efforts can be disastrous.

More information on the qualitative differences in assessments is available at www.aboutassessments.com. Part 7 will discuss why measuring cognitive abilities is important.