| I personally dislike tests that are multiple choice. It loads the mind with (sometimes obviously) false data that must be rejected and the ability to pattern match on word usage or even guessing between answers that haven't been eliminated means that the instructor never really knows if the person actually knew the answer and neither does the student. When I used to participate in math contests I'd go out of my way not to read the answers before attempting to solve the problem because it's much more gratifying that to reverse engineer the problem from the input. In the end, I'd resort to that or guessing if I couldn't crack the problem on my own, but then at least I knew which problems I really solved and which ones I didn't. And when I was invited to do more complex math contests than the ones that Waterloo was putting out I was astonished at how much harder just having to write in an exact answer was, since you missed out on the "not one of the options" feedback if you flubbed a number somewhere. With these certification tests it's impossible to really do the same thing. It doesn't even feel like real thinking compared to writing software or prose. Also the pointless memorization of things like "what classes of MITM attacks are there?"[0] don't really get a person to learn the stuff. It's like memorizing the atomic weight of elements. That stuff never stuck for me until I was doing real chemistry and the basic act of looking up a weight each time slowly got me to memorize the elemental weight of the chemicals I was working on. Same goes for all those French words I was forced to learn in Canada. Once I started wanted to speak Russian I was amazed at my ability to actually be able to remember words. I'm not saying these things should never be tested. I wouldn't trust a structural engineer that didn't know what lateral torsional buckling is[1] but the manner in which we test it should be as close the reality we would expect, and this is what we do in engineering school most of the time. [0] Answer: Surveil, Fabricate, Modify, and Deny. [1] Essentially the concept that a tall but thin beam can swing to the side, or "torsionally buckle" when loaded from the top (thus requiring bracing) unlike a beam rotated 90 degrees, which does not require bracing for this reason. |
I think multiple choice exams can be a good way – a physics exam I took was multiple choice, and so because the examiners were "giving you the answers", they took it as an opportunity to ask really out-there questions. They would frequently take two wildly different aspects of the physics course and mash them up in a new way. I thought this was very effective at testing if the student understood the areas and could therefore combine understanding with intuitive leaps, or whether they had memorised the formulas necessary for the exam and could only repeat those on command in the exact form they knew.
This suggests to me that it's less multiple-choice, and more that this is a purely fact-based style of testing.
For a computer security course I took at university one of the exam questions was "Describe Stuxnet – 20 marks" (half the exam's marks). we had had a lecture dissecting the whole Stuxnet incident. For those simply memorising facts this question would be quite hard, but for those who had understood why certain things matter and could write an in-depth explanation of the security failings, it was great.
The problem is that marking this sort of question requires a significant amount of manual work, and that doesn't scale. Another example would be Phd vivas, which I've heard are generally a well respected way of determining ability, but which again take a significant amount of expert manual input.
I don't think we'll get good certifications for these sorts of things until we find better ways to examine like this.