| You're completely right, the testing method is likely very linked to why this is an ineffective way of deciding someone's qualification. I think multiple choice exams can be a good way – a physics exam I took was multiple choice, and so because the examiners were "giving you the answers", they took it as an opportunity to ask really out-there questions. They would frequently take two wildly different aspects of the physics course and mash them up in a new way. I thought this was very effective at testing if the student understood the areas and could therefore combine understanding with intuitive leaps, or whether they had memorised the formulas necessary for the exam and could only repeat those on command in the exact form they knew. This suggests to me that it's less multiple-choice, and more that this is a purely fact-based style of testing. For a computer security course I took at university one of the exam questions was "Describe Stuxnet – 20 marks" (half the exam's marks). we had had a lecture dissecting the whole Stuxnet incident. For those simply memorising facts this question would be quite hard, but for those who had understood why certain things matter and could write an in-depth explanation of the security failings, it was great. The problem is that marking this sort of question requires a significant amount of manual work, and that doesn't scale. Another example would be Phd vivas, which I've heard are generally a well respected way of determining ability, but which again take a significant amount of expert manual input. I don't think we'll get good certifications for these sorts of things until we find better ways to examine like this. |
That precludes having studied "enough". "Enough" is measured relative to your peers, not to how hard the exam is. As long as someone else is studying, "enough" is a moving target. On a certification, I can check out how hard the exam is and study just enough to pass it. If I know for a fact that the exam isn't going to test comprehension, just recall, and that I'm not going to fail because someone else studied enough to comprehend, it makes sense to just memorize the recall portions.
Certifications would do better if they went the same way. For each group of people taking the test, curve the test so that some minimum, fixed percentage of them fail. Let's say it's 20%; so for every group of 100 people taking the test, at least 20 of them will fail (although it could be more if you set a floor on the lowest score you will allow to pass). Even if the exam doesn't test comprehension, I'm willing to bet that most people would pick it up just through studying.